ML-172: add research task

backlog/tasks/ml-172 - Research-improve-record-similarity-embedding-text-for-better-musical-similarity-results.md

@@ -0,0 +1,73 @@
+---
+id: ML-172
+title: >-
+  Research: improve record similarity embedding text for better musical
+  similarity results
+status: To Do
+assignee: []
+created_date: "2026-05-09 05:49"
+updated_date: "2026-05-09 05:50"
+labels:
+  - research
+  - similarity
+  - embeddings
+  - records
+dependencies: []
+modified_files:
+  - lib/music_library/records/similarity.ex
+  - lib/music_library/records/record_embedding.ex
+priority: medium
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+
+## Problem
+
+The current record similarity approach (`MusicLibrary.Records.Similarity.text_representation/1`) builds embedding text from artist biographical info (Wikipedia, Discogs, MusicBrainz), genres, Last.fm tags, and similar artists — then uses OpenAI embeddings + cosine distance to find similar records.
+
+This yields poor results. Example: Gotthard's "Made in Switzerland" returns zero similar records at the current 0.45 threshold, despite obvious musical similarity to Mr. Big and other hard rock acts.
+
+**Root cause confirmed:** The actual embedding text for this record is dominated by repetitive Wikipedia biographical prose. Here's the real output:
+
+```
+Album: Made in Switzerland
+Artists: Gotthard
+Genres: hard rock, rock, heavy metal, blues rock, alternative rock
+Released: 2006
+Type: Live
+
+Gotthard (Switzerland):
+Swiss hard rock band. Gotthard is a Swiss hard rock band founded in Lugano
+by Steve Lee and Leo Leoni. Their last sixteen albums have all reached number
+one in the Swiss album charts... [~200 more chars of biography]
+Gotthard is a Swiss hard rock band founded in Lugano by Steve Lee and Leo Leoni.
+Tags: hard rock, rock, melodic rock, heavy metal, swiss, switzerland, classic rock
+```
+
+Issues visible:
+
+- Wikipedia description + summary are both included and redundant (both say "Swiss hard rock band founded by Steve Lee and Leo Leoni")
+- Biographical text is ~300+ chars, while genres+tags are ~100 chars — the musical similarity signal is proportionally tiny
+- No Last.fm similar artists appear in this particular embedding text (may need regeneration after similar artists were fetched)
+- The embedding model likely weights "Swiss band from Lugano" and chart statistics more than "hard rock / melodic rock / heavy metal"
+
+Meanwhile, the Last.fm similar artists data (Magnum, Thunder, Tesla, etc.) is correctly stored in `ArtistInfo.lastfm_data` and visible on the artist page — so the signal exists but is severely underrepresented.
+
+## Research goals
+
+Investigate and propose alternative approaches:
+
+1. **Text weighting / restructuring** — Give more prominence to genres, Last.fm tags, and similar artists; reduce or restructure biographical text so it doesn't dominate. E.g., deduplicate Wikipedia description+summary, cap bio at 100-150 chars, put genres/tags/similar artists first.
+2. **Different embedding strategies** — e.g., separate artist-level and record-level embeddings, or generate embedding text focused purely on musical descriptors rather than biography
+3. **Hybrid approaches** — Combine embedding similarity with explicit genre/tag overlap scoring, or use Last.fm similar artists as a direct influence on ranking
+4. **Model / parameter tuning** — Explore whether a different OpenAI model, different chunking strategy, or threshold adjustment alone could fix this
+5. **External signals** — Could additional data sources (e.g., MusicBrainz genre tags, AllMusic style descriptors) provide better similarity signals?
+
+## Success criteria
+
+- Identify 2-3 promising approaches ranked by expected impact and implementation effort
+- For top approach(es), rough out what the new `text_representation` would look like
+- Document findings so an implementation task can follow
+<!-- SECTION:DESCRIPTION:END -->