ML-170: add enrichment fields to collection API
Add MusicLibrary.Collection.Enrichment module with batch scrobble, artist country, and selected release enrichment from existing tables. Wire enrichment into all four collection API endpoints (index, latest, random, on_this_day). New JSON fields: scrobble_count, last_listened_at, artist_country, selected_release. Uses 3 fixed-count batch queries regardless of page size — no N+1 risk. No schema changes or migrations required.
This commit is contained in:
Claudio Ortolina
+36
-35
@@ -97,7 +97,7 @@ Last.fm schemas (separate, not Ecto-persisted to main DB):
|
||||
| Context | Schemas | Responsibility |
|
||||
| ---------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Records` | Record, RecordEmbedding, SearchIndex | CRUD, search, import from MusicBrainz, cover/genre/color management, PubSub notifications |
|
||||
| `Collection` | Record (via SearchIndex) | Querying collected records (purchased_at != nil), stats, collected artist IDs, collection summary for AI chat |
|
||||
| `Collection` | Record (via SearchIndex) | Querying collected records (purchased_at != nil), stats, collected artist IDs, collection summary for AI chat. Sub-module `Collection.Enrichment` batch-hydrates API results with scrobble stats, artist country, and selected release info. |
|
||||
| `Wishlist` | Record (via SearchIndex) | Querying wishlisted records (purchased_at is nil) |
|
||||
| `Artists` | ArtistInfo, ArtistRecord | Artist metadata from MusicBrainz/Discogs/Wikipedia/Last.fm, images, search |
|
||||
| `Assets` | Asset | Binary asset storage (covers, artist images), cache tracking, pruning unreferenced assets |
|
||||
@@ -118,40 +118,41 @@ Last.fm schemas (separate, not Ecto-persisted to main DB):
|
||||
|
||||
## Business Logic Modules
|
||||
|
||||
| Module | Purpose |
|
||||
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Records.SearchParser` | Parses search syntax: `artist:X`, `album:X`, `genre:"Y"`, `format:cd`, `type:album`, `purchase_year:2024`, `release_year:2024`, free text |
|
||||
| `ListeningStats.SearchParser` | Parses scrobbled tracks search syntax: `record:X`, `album_mbid:X`, `artist_mbid:X`, `artist:X`, `album:X`, `track:X`, free text |
|
||||
| `Records.Similarity` | Embedding generation and async enqueue (OpenAI, enriched with Last.fm tags, skips API call when text representation unchanged), artist-cascade regeneration when upstream metadata changes, cosine-distance search (sqlite-vec) |
|
||||
| `Records.TracklistPdf` | Generates 120mm×120mm PDF tracklist from record + release data (Typst) |
|
||||
| `Batch` | Generic batch runner: stream + transaction + error accumulation |
|
||||
| `Records.Batch` | Batch operations: refresh all MusicBrainz data, generate all embeddings (uses `Batch`) |
|
||||
| `Artists.Batch` | Batch refresh: MusicBrainz, Discogs, Wikipedia, Last.fm for all artists (uses `Batch`) |
|
||||
| `Req.RateLimiter` | ETS-backed Req request step enforcing per-API minimum intervals between requests |
|
||||
| `Req.RateLimiter.Clock` | Behaviour for time operations (allows test clock injection) |
|
||||
| `Req.RateLimiter.SystemClock` | Real clock implementation using System.monotonic_time |
|
||||
| `Assets.Cache` | ETS-based asset cache with TTL (7-day TTL, TTL-only invalidation since assets are content-addressable and immutable) |
|
||||
| `Assets.Image` / `Assets.Transform` | Image processing via Vix (libvips) |
|
||||
| `Colors.Extractor` | Behaviour for dominant color extraction (configurable, allows test stubbing) |
|
||||
| `Colors.KMeansExtractor` | Color extraction via K-Means clustering (dominant_colors library), implements `Colors.Extractor` |
|
||||
| `Chats.StreamProvider` | Behaviour for streaming AI chat (`stream_response/3` callback) |
|
||||
| `Chats.RecordChat` | Chat implementation for records (OpenAI streaming, web search enabled) |
|
||||
| `Chats.ArtistChat` | Chat implementation for artists (OpenAI streaming, uses Wikipedia/artist context) |
|
||||
| `Chats.CollectionChat` | Chat implementation for the collection (OpenAI streaming via gpt-5.1, uses collection summary as context) |
|
||||
| `Chats.Prompt` | Builds complete chat prompts by interpolating identity, content, and approach guidelines |
|
||||
| `Country` | Country code (alpha-2, alpha-3, subdivision, IETF) to flag emoji conversion |
|
||||
| `ErrorTracker.ErrorNotifier` | GenServer: attaches to ErrorTracker telemetry, skips muted errors, throttles repeated errors, dispatches email notifications |
|
||||
| `ErrorTracker.ErrorNotifier.Email` | Builds and sends Swoosh error notification emails with stack trace formatting |
|
||||
| `ErrorIgnorer` | ErrorTracker.Ignorer implementation: filters non-actionable errors (e.g., NoRouteError from bot scanners) |
|
||||
| `MusicLibrary.ErrorResponse` | Behaviour for structured API error responses (`retryable?/1`, `retry_delay_seconds/1`) — per-API `ErrorResponse` modules implement this |
|
||||
| `MusicLibrary.HttpError` | Default HTTP status → kind mapping (`:rate_limit`, `:server_error`, `:timeout`, `:auth_error`, `:not_found`, `:client_error`, `:unknown`) used as baseline by per-API `ErrorResponse` modules |
|
||||
| `MusicLibrary.RetryDelay` | Parses and clamps provider retry/reset headers into Oban snooze delays for structured API errors |
|
||||
| `MusicLibrary.Worker.ErrorHandler` | Translates per-API `ErrorResponse` structs into Oban tuples — `{:snooze, seconds}` for retryable, `{:cancel, reason}` for permanent |
|
||||
| `MusicLibraryWeb.RecordsOnThisDayEmail` | Builds and sends daily "records on this day" email with cover images, anniversary styling |
|
||||
| `MusicLibrary.Mailer` | Swoosh mailer (Mailgun in prod, local adapter in dev) |
|
||||
| `FormatNumber` | Number formatting utility |
|
||||
| `QueryReporter` | Dev-only Ecto telemetry reporter: captures executed SQL to a log file with interpolated params, source locations, and timing. Activated at runtime via `start/1` / `stop/0` |
|
||||
| `MusicLibrary.Logger.SingleLineFormatter` | Production-only Logger.Formatter safety net: replaces embedded newlines (`\n`) with escaped `\\n` in all log messages, ensuring every physical log line is exactly one log event |
|
||||
| Module | Purpose |
|
||||
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Records.SearchParser` | Parses search syntax: `artist:X`, `album:X`, `genre:"Y"`, `format:cd`, `type:album`, `purchase_year:2024`, `release_year:2024`, free text |
|
||||
| `ListeningStats.SearchParser` | Parses scrobbled tracks search syntax: `record:X`, `album_mbid:X`, `artist_mbid:X`, `artist:X`, `album:X`, `track:X`, free text |
|
||||
| `Records.Similarity` | Embedding generation and async enqueue (OpenAI, enriched with Last.fm tags, skips API call when text representation unchanged), artist-cascade regeneration when upstream metadata changes, cosine-distance search (sqlite-vec) |
|
||||
| `Records.TracklistPdf` | Generates 120mm×120mm PDF tracklist from record + release data (Typst) |
|
||||
| `Batch` | Generic batch runner: stream + transaction + error accumulation |
|
||||
| `Records.Batch` | Batch operations: refresh all MusicBrainz data, generate all embeddings (uses `Batch`) |
|
||||
| `Artists.Batch` | Batch refresh: MusicBrainz, Discogs, Wikipedia, Last.fm for all artists (uses `Batch`) |
|
||||
| `Collection.Enrichment` | Batch-hydrates collection API results with scrobble stats (from `scrobbled_tracks`), artist country (from `artist_infos`), and selected release details (from `records.musicbrainz_data`). Uses 3 fixed-count queries regardless of page size. |
|
||||
| `Req.RateLimiter` | ETS-backed Req request step enforcing per-API minimum intervals between requests |
|
||||
| `Req.RateLimiter.Clock` | Behaviour for time operations (allows test clock injection) |
|
||||
| `Req.RateLimiter.SystemClock` | Real clock implementation using System.monotonic_time |
|
||||
| `Assets.Cache` | ETS-based asset cache with TTL (7-day TTL, TTL-only invalidation since assets are content-addressable and immutable) |
|
||||
| `Assets.Image` / `Assets.Transform` | Image processing via Vix (libvips) |
|
||||
| `Colors.Extractor` | Behaviour for dominant color extraction (configurable, allows test stubbing) |
|
||||
| `Colors.KMeansExtractor` | Color extraction via K-Means clustering (dominant_colors library), implements `Colors.Extractor` |
|
||||
| `Chats.StreamProvider` | Behaviour for streaming AI chat (`stream_response/3` callback) |
|
||||
| `Chats.RecordChat` | Chat implementation for records (OpenAI streaming, web search enabled) |
|
||||
| `Chats.ArtistChat` | Chat implementation for artists (OpenAI streaming, uses Wikipedia/artist context) |
|
||||
| `Chats.CollectionChat` | Chat implementation for the collection (OpenAI streaming via gpt-5.1, uses collection summary as context) |
|
||||
| `Chats.Prompt` | Builds complete chat prompts by interpolating identity, content, and approach guidelines |
|
||||
| `Country` | Country code (alpha-2, alpha-3, subdivision, IETF) to flag emoji conversion |
|
||||
| `ErrorTracker.ErrorNotifier` | GenServer: attaches to ErrorTracker telemetry, skips muted errors, throttles repeated errors, dispatches email notifications |
|
||||
| `ErrorTracker.ErrorNotifier.Email` | Builds and sends Swoosh error notification emails with stack trace formatting |
|
||||
| `ErrorIgnorer` | ErrorTracker.Ignorer implementation: filters non-actionable errors (e.g., NoRouteError from bot scanners) |
|
||||
| `MusicLibrary.ErrorResponse` | Behaviour for structured API error responses (`retryable?/1`, `retry_delay_seconds/1`) — per-API `ErrorResponse` modules implement this |
|
||||
| `MusicLibrary.HttpError` | Default HTTP status → kind mapping (`:rate_limit`, `:server_error`, `:timeout`, `:auth_error`, `:not_found`, `:client_error`, `:unknown`) used as baseline by per-API `ErrorResponse` modules |
|
||||
| `MusicLibrary.RetryDelay` | Parses and clamps provider retry/reset headers into Oban snooze delays for structured API errors |
|
||||
| `MusicLibrary.Worker.ErrorHandler` | Translates per-API `ErrorResponse` structs into Oban tuples — `{:snooze, seconds}` for retryable, `{:cancel, reason}` for permanent |
|
||||
| `MusicLibraryWeb.RecordsOnThisDayEmail` | Builds and sends daily "records on this day" email with cover images, anniversary styling |
|
||||
| `MusicLibrary.Mailer` | Swoosh mailer (Mailgun in prod, local adapter in dev) |
|
||||
| `FormatNumber` | Number formatting utility |
|
||||
| `QueryReporter` | Dev-only Ecto telemetry reporter: captures executed SQL to a log file with interpolated params, source locations, and timing. Activated at runtime via `start/1` / `stop/0` |
|
||||
| `MusicLibrary.Logger.SingleLineFormatter` | Production-only Logger.Formatter safety net: replaces embedded newlines (`\n`) with escaped `\\n` in all log messages, ensuring every physical log line is exactly one log event |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user