music_library/backlog/tasks/ml-172 - Research-improve-record-similarity-embedding-text-for-better-musical-similarity-results.md

id, title, status, assignee, created_date, updated_date, labels, dependencies, modified_files, priority

id	title	status	assignee	created_date	updated_date	labels	dependencies	modified_files	priority
ML-172	Research: improve record similarity embedding text for better musical similarity results	To Do		2026-05-09 05:49	2026-05-11 06:47	research		lib/music_library/records/similarity.ex lib/music_library/records/record_embedding.ex	medium

Description

Problem

The current record similarity approach (MusicLibrary.Records.Similarity.text_representation/1) builds embedding text from artist biographical info (Wikipedia, Discogs, MusicBrainz), genres, Last.fm tags, and similar artists — then uses OpenAI embeddings + cosine distance to find similar records.

This yields poor results. Example: Gotthard's "Made in Switzerland" returns zero similar records at the current 0.45 threshold, despite obvious musical similarity to Mr. Big and other hard rock acts.

Root cause confirmed: The actual embedding text for this record is dominated by repetitive Wikipedia biographical prose. Here's the real output:

Album: Made in Switzerland
Artists: Gotthard
Genres: hard rock, rock, heavy metal, blues rock, alternative rock
Released: 2006
Type: Live

Gotthard (Switzerland):
Swiss hard rock band. Gotthard is a Swiss hard rock band founded in Lugano
by Steve Lee and Leo Leoni. Their last sixteen albums have all reached number
one in the Swiss album charts... [~200 more chars of biography]
Gotthard is a Swiss hard rock band founded in Lugano by Steve Lee and Leo Leoni.
Tags: hard rock, rock, melodic rock, heavy metal, swiss, switzerland, classic rock

Issues visible:

• Wikipedia description + summary are both included and redundant (both say "Swiss hard rock band founded by Steve Lee and Leo Leoni")
• Biographical text is ~300+ chars, while genres+tags are ~100 chars — the musical similarity signal is proportionally tiny
• No Last.fm similar artists appear in this particular embedding text (may need regeneration after similar artists were fetched)
• The embedding model likely weights "Swiss band from Lugano" and chart statistics more than "hard rock / melodic rock / heavy metal"

Meanwhile, the Last.fm similar artists data (Magnum, Thunder, Tesla, etc.) is correctly stored in ArtistInfo.lastfm_data and visible on the artist page — so the signal exists but is severely underrepresented.

Research goals

Investigate and propose alternative approaches:

1. Text weighting / restructuring — Give more prominence to genres, Last.fm tags, and similar artists; reduce or restructure biographical text so it doesn't dominate. E.g., deduplicate Wikipedia description+summary, cap bio at 100-150 chars, put genres/tags/similar artists first.
2. Different embedding strategies — e.g., separate artist-level and record-level embeddings, or generate embedding text focused purely on musical descriptors rather than biography
3. Hybrid approaches — Combine embedding similarity with explicit genre/tag overlap scoring, or use Last.fm similar artists as a direct influence on ranking
4. Model / parameter tuning — Explore whether a different OpenAI model, different chunking strategy, or threshold adjustment alone could fix this
5. External signals — Could additional data sources (e.g., MusicBrainz genre tags, AllMusic style descriptors) provide better similarity signals?

Success criteria

• Identify 2-3 promising approaches ranked by expected impact and implementation effort
• For top approach(es), rough out what the new text_representation would look like
• Document findings so an implementation task can follow