cloud/music_library
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ML-170: research and plan

Browse Source
This commit is contained in:
Claudio Ortolina
2026-05-08 15:07:50 +01:00
parent 0a16b72411
commit 87310fd8ce
2 changed files with 484 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backlog/docs/doc-14 - Research-Exposing-additional-data-points-for-collection-API.md
+155
View File
@@ -0,0 +1,155 @@
---
id: doc-14
title: "Research: Exposing additional data points for collection API"
type: specification
created_date: "2026-05-08 13:03"
updated_date: "2026-05-08 14:01"
tags:
- api
- collection
- research
---
# Research: Exposing additional data points for collection API
## Problem
The `/api/v1/collection/*` endpoints currently return a flat subset of SearchIndex fields. We need to add four new data points per record:
| Data Point | Source | Current Access Pattern | Cost |
| ---------------------- | --------------------------------------------- | ------------------------------------------------ | ------------------------------------------- |
| Scrobble count | `scrobbled_tracks` | `ListeningStats.play_count/1` | 2 queries per record |
| Last listened at | `scrobbled_tracks` | `ListeningStats.get_last_listened_track/1` | 2 queries per record |
| Collected release info | `records.musicbrainz_data`, `record_releases` | `Record.releases/1`, `Record.selected_release/1` | Requires Record struct (not in SearchIndex) |
| Artist country | `artist_infos.musicbrainz_data` | `ArtistInfo.country/1` | 1 query per artist |
## Current architecture
- Collection API queries `records_search_index` (FTS5 virtual table), populated by triggers from `records`
- `SearchIndex` has a subset of Record columns; no `musicbrainz_data`, `artist_infos`, or scrobble data
- The JSON view (`CollectionJSON`) maps SearchIndex structs directly
## Implementation routes
### Route A: Post-query hydration layer (recommended)
Add a new enrichment module in `MusicLibrary.Collection` that batch-fetches additional data after the initial SearchIndex query.
**How it works:**
1. Execute existing SearchIndex query as before
2. Pass results through `Collection.enrich/1` which:
a. Collects all record IDs → batch-queries `records` table for `release_ids` and main artist MBIDs
b. Collects all release IDs → batch-queries `scrobbled_tracks` for play counts and most recent scrobble per release group
c. Collects unique artist MBIDs → batch-queries `artist_infos` for country data
d. Optionally queries `record_releases` for collected edition details
3. Merge enriched data into the JSON response map
**Changes required:**
- New functions: `Collection.enrich_scrobble_stats/1`, `Collection.enrich_artist_country/1`, `Collection.enrich_release_details/1`
- Update `CollectionJSON.record/1` to accept enriched data
- Update `CollectionController` actions to call enrichment
- New tests: enriched response shape, batch query correctness, nil handling for records without scrobbles/artist info
**Pros:**
- No migrations or SearchIndex trigger changes
- Clean separation of concerns — enrichment is opt-in and independently testable
- Can add `?include=scrobbles,country,releases` query params later for selective enrichment
- Existing API shape unchanged for clients not reading new fields
**Cons:**
- Additional queries (but batchable: ~3-5 queries total regardless of result count)
- Slightly slower response for the index endpoint (est. 5-20ms added for 20 records with SQLite)
**Performance profile:**
- Batch scrobble lookup: 1 query with `WHERE album_mbid IN (...release_ids...)` across all records in the page (constant query count, scales with page size not total table size)
- Batch artist_info lookup: 1 query with `WHERE id IN (...artist_mbids...)`
- Batch record_releases lookup: 1 query with `WHERE record_id IN (...record_ids...)`
- Total: ~3 additional queries for index endpoint (20 records), ~2 for single-record endpoints
- Memory: minimal — aggregated counts/dates, not loading full tables
### Route B: Add columns to SearchIndex + triggers
Extend the FTS5 virtual table and its sync triggers to include the new data points directly.
**How it works:**
1. Migration: add `scrobble_count INTEGER`, `last_listened_at INTEGER`, `artist_country TEXT`, `release_details TEXT` to SearchIndex
2. Update triggers to compute or join these values when records change
3. Scrobble data would need separate triggers on `scrobbled_tracks` to update SearchIndex
**Changes required:**
- Migration with `ALTER VIRTUAL TABLE` (FTS5 has limitations)
- New triggers on `records` for artist country, release details
- New triggers on `scrobbled_tracks` for scrobble counts and last listened at
- Update `essential_fields()` to include new columns
- Update `CollectionJSON` to map new fields
**Pros:**
- Zero additional queries per API request
- Consistent with existing SearchIndex-first architecture
**Cons:**
- FTS5 virtual tables are complex to alter; may require full rebuild
- Triggers on `scrobbled_tracks` fire on every scrobble import (high write volume), keeping SearchIndex in sync adds write overhead
- Data staleness: scrobble counts change frequently; SearchIndex would lag behind (or be constantly rebuilt)
- Tight coupling: SearchIndex purpose is search, not analytics
- Artist country requires joining `artist_records` → `artist_infos` in a trigger, which is fragile
### Route C: Separate details endpoint
Keep index endpoints lightweight and add a new `GET /api/v1/collection/:id` endpoint for enriched single-record views.
**How it works:**
1. Index endpoints remain unchanged
2. New show endpoint fetches a record + all enrichment data
3. Can use `ListeningStats.play_count/1` and `get_last_listened_track/1` directly since it's a single record
**Changes required:**
- New route and controller action
- New JSON view for enriched record
- Tests for new endpoint
**Pros:**
- Zero impact on existing index endpoint performance
- Per-record functions (`play_count/1`, `get_last_listened_track/1`) work for single records without batching
- Clean API design: lightweight list, heavy detail
**Cons:**
- Clients need two API calls to get enriched data for multiple records
- Doesn't satisfy the requirement if the user wants enriched data in index responses
## Recommendation
**Route A (post-query hydration)** is the recommended approach because:
1. It satisfies the requirement of adding data to existing endpoints without architectural changes
2. Performance impact is minimal (~3 batch queries regardless of page size)
3. No migration or trigger complexity
4. Can be implemented incrementally (scrobble stats first, then country, then releases)
5. Easily extended to support `?include=` query parameters for selective enrichment in the future
Routes B and C have merit but introduce either significant complexity (B) or don't fully satisfy the requirement (C).
## Open questions for decision
1. **"Collected release information" — what specifically?** Options include:
- Just the `selected_release_id` (already in SearchIndex — trivial)
- List of `release_ids` with their formats/types from `record_releases`
- Full MusicBrainz release data (tracks, dates, countries) from `records.musicbrainz_data`
- Need clarification from stakeholder
2. **Always include vs. opt-in?** Should these fields always appear in the response, or should they be gated behind a query parameter like `?include=scrobbles,country`?
3. **Artist country — which artist?** A record can have multiple artists (collaborations). Should we show country for the main artist only, or all artists? Main artist is simpler and more predictable.
backlog/tasks/ml-170 - Expose-additional-data-points-for-api-v1-collection-records.md
+329
View File
@@ -0,0 +1,329 @@
---
id: ML-170
title: Expose additional data points for api/v1/collection/* records
status: To Do
assignee: []
created_date: "2026-05-08 13:02"
updated_date: "2026-05-08 14:03"
labels:
- api
- enhancement
dependencies: []
documentation:
- doc-14 - Research-Exposing-additional-data-points-for-collection-API.md
priority: medium
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
The current `/api/v1/collection/*` API endpoints return a subset of record fields from the SearchIndex virtual table. The following additional data points should be exposed to API consumers:
1. **Scrobble count** (if available) — number of times tracks from this record have been listened to (from Last.fm scrobble data in the `scrobbled_tracks` table)
2. **Last listened at date** — timestamp of the most recent scrobble for this record
3. **Collected release information** — details about specific releases (editions) that are part of the collection for each record/release group
4. **Artist country** — the country/area of the main artist, available via `MusicLibrary.Artists.ArtistInfo`
Currently, `CollectionJSON` maps SearchIndex fields into a flat JSON record with: id, type, format, musicbrainz_id, genres, release_date, purchased_at, artists (names only), title, and cover_url variants. None of the above data points are exposed.
The API endpoints affected are:
- `GET /api/v1/collection` (index, paginated)
- `GET /api/v1/collection/latest`
- `GET /api/v1/collection/random`
- `GET /api/v1/collection/on_this_day`
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [ ] #1 `GET /api/v1/collection` (index) responses include `scrobble_count`, `last_listened_at`, `artist_country`, and `selected_release` for each record
- [ ] #2 `GET /api/v1/collection/latest` response includes all four new fields
- [ ] #3 `GET /api/v1/collection/random` response includes all four new fields
- [ ] #4 `GET /api/v1/collection/on_this_day` responses include all four new fields for each record
- [ ] #5 Records without scrobble data return `scrobble_count: 0` and `last_listened_at: null`
- [ ] #6 Records without artist info return `artist_country: null`
- [ ] #7 Records without a selected release return `selected_release: null`
- [ ] #8 All existing tests pass (backward compatible — no existing field removed or renamed)
<!-- AC:END -->
## Implementation Plan
<!-- SECTION:PLAN:BEGIN -->
## Implementation Plan
### Overview
Add a `MusicLibrary.Collection.Enrichment` module that batch-hydrates SearchIndex (and Record struct) results with three new data dimensions (scrobble stats, artist country, selected release info) using fixed-count batch SQL queries. The enrichment layer runs in the controller after the initial query, before JSON rendering.
### Architecture impact analysis
| Touchpoint | Impact |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `lib/music_library/collection.ex` | New sub-module `MusicLibrary.Collection.Enrichment` |
| `lib/music_library_web/controllers/collection_json.ex` | Extended `record/1` to include new fields with nil-safe `Map.get/3` |
| `lib/music_library_web/controllers/collection_controller.ex` | Wire enrichment call between query and render in all 4 actions |
| Schemas (`SearchIndex`, `Record`) | **No changes** — enrichment works on maps/structs with `id`, `release_ids`, `selected_release_id`, and `artists` embedded list |
| Database | **No migrations** — reads existing `scrobbled_tracks`, `records`, `artist_infos`, `record_releases` |
| Routes | No changes |
| PubSub | No changes |
| Supervision tree | No changes |
| External APIs | No changes |
**Data source compatibility:** The 4 controller actions use different query paths. Verify each path produces results with the fields enrichment needs:
| Action | Query path | Result type | Has `release_ids`? | Has `artists`? | Has `selected_release_id`? |
| ------------- | --------------------------------- | -------------------- | ------------------------- | ----------------- | -------------------------- |
| `index` | `SearchIndex` via FTS5 | `SearchIndex` struct | ✅ yes (array of strings) | ✅ yes (embedded) | ✅ yes (string) |
| `latest` | `Record` via `essential_fields()` | Record struct / map | ✅ yes | ✅ yes | ✅ yes |
| `random` | same as `latest` | same | ✅ yes | ✅ yes | ✅ yes |
| `on_this_day` | `Record` via `essential_fields()` | Record struct / map | ✅ yes | ✅ yes | ✅ yes |
All paths produce results with the three required fields. Enrichment functions accept any map with those keys.
### Scope clarification
"Collected release information" means the **single selected release** per record (`selected_release_id` → details from `musicbrainz_data`), not the full list of all releases. The field is named `selected_release` in the API response.
---
### Step 1: Create `MusicLibrary.Collection.Enrichment` module
Create `lib/music_library/collection/enrichment.ex` with three public functions and a composition function:
```elixir
@spec enrich([map()]) :: [map()]
def enrich(records)
@spec enrich_scrobbles([map()]) :: [map()]
def enrich_scrobbles(records)
@spec enrich_artist_country([map()]) :: [map()]
def enrich_artist_country(records)
@spec enrich_selected_release([map()]) :: [map()]
def enrich_selected_release(records)
```
**Data flow:**
```
maps[] → enrich_scrobbles() → enrich_artist_country() → enrich_selected_release() → maps[]
```
Each function is independently callable and testable. `enrich/1` composes all three via pipeline.
**Verification:** Unit tests for each enrichment function verify correct field additions.
---
### Step 2: Implement `enrich_scrobbles/1`
**Algorithm:**
1. Collect all `release_ids` from all records into a flat, unique, non-empty list.
2. **Guard:** If the collected list is empty (all records have `release_ids: []` or `nil`), return records unchanged with `scrobble_count: 0, last_listened_at: nil` on each.
3. Single batch query against `scrobbled_tracks`:
```sql
SELECT json_extract(album, '$.musicbrainz_id') AS release_id,
COUNT(DISTINCT scrobbled_at_uts) AS scrobble_count,
MAX(scrobbled_at_uts) AS last_listened_at
FROM scrobbled_tracks
WHERE json_extract(album, '$.musicbrainz_id') IN (?)
GROUP BY 1
```
4. Build a lookup map `%{release_id => %{scrobble_count, last_listened_at}}`.
5. For each record, sum scrobble counts across its `release_ids` (a release group may have multiple releases, each with scrobbles) and take the `MAX` of `last_listened_at`.
6. For records with no matching release IDs in the lookup, `scrobble_count: 0, last_listened_at: nil`.
**No title+artist fallback.** The batch release_id lookup is sufficient. Records without scrobbles correctly show zero/null. If a stakeholder later requires title+artist matching, that is a separate follow-up with its own performance analysis — it would add N individual queries and break the fixed-query-count guarantee.
**Uses existing index:** `scrobbled_tracks_album_musicbrainz_id_index` on `json_extract(album, '$.musicbrainz_id')`.
**Converts `last_listened_at` to ISO8601 string** after the query (UNIX timestamp → `DateTime.from_unix!` → `DateTime.to_iso8601`).
**Edge case — shared release_ids across records:** Two records may share the same release group (different formats). This is correctly handled: the lookup map is per release_id, and each record independently sums across its own `release_ids`.
**Verification:**
- Records with scrobble matches → `scrobble_count > 0`, `last_listened_at` is a non-nil ISO8601 string
- Records with no scrobbles → `scrobble_count: 0`, `last_listened_at: nil`
- Records with empty/nil `release_ids` → `scrobble_count: 0`, `last_listened_at: nil`
- Two records sharing a release group → both correctly attribute scrobbles
- Verify `last_listened_at` is the most recent timestamp across all releases in the group
---
### Step 3: Implement `enrich_artist_country/1`
**Algorithm:**
1. Extract main artist `musicbrainz_id` from each record's `artists` embedded list (first element only — `hd(record.artists)`).
2. Collect unique non-nil artist IDs.
3. **Guard:** If no artist IDs, return records unchanged with `artist_country: nil`.
4. Batch query `artist_infos` (primary key lookup — uses existing PK index):
```sql
SELECT id, musicbrainz_data
FROM artist_infos
WHERE id IN (?)
```
5. For each row, extract country using `MusicLibrary.Artists.ArtistInfo.country/1` (parses `musicbrainz_data.area.iso-3166-1-codes`, falls back to `musicbrainz_data.country`, returns `%{name: "World", code: "XW"}` if no area data).
6. Build lookup map `%{artist_mbid => %{name: country_name, code: country_code}}`.
7. Add `artist_country` (map, or nil if no `artist_infos` row exists for that artist).
**Performance note:** `artist_infos.musicbrainz_data` is a full MusicBrainz JSON blob. Loading it for up to 20 unique artists per page is acceptable. The JSON extraction (`ArtistInfo.country/1`) is in Elixir, not SQL — correct, as the extraction logic (iso-3166-1-codes vs iso-3166-2-codes vs country fallback) is non-trivial.
**Verification:**
- Artist with country data → `artist_country: %{name: "...", code: "..."}`
- Artist with no `artist_infos` row → `artist_country: nil`
- Record with empty artists list → `artist_country: nil`
- Multiple records with same main artist → reuse single lookup entry (dedup works)
---
### Step 4: Implement `enrich_selected_release/1`
**Algorithm:**
1. Collect record IDs where `selected_release_id` is not nil and not empty string.
2. **Guard:** If no record IDs qualify, return records unchanged with `selected_release: nil`.
3. Batch query `records` (primary key lookup — uses existing PK index):
```sql
SELECT id, musicbrainz_data, selected_release_id
FROM records
WHERE id IN (?)
```
4. For each record, extract the selected release from `musicbrainz_data` using `Record.releases/1` + `Record.find_release/2`.
5. Fields to expose from the selected release:
- `format` — parsed from media formats (e.g., "cd", "vinyl", "multi")
- `date` — release date string
- `country` — country code (e.g., "US", "GB", "XW")
- `catalog_number` — catalog number string
- `packaging` — packaging description
- `disambiguation` — disambiguation comment
6. Add `selected_release` (map or nil) to each record.
**Performance note:** `records.musicbrainz_data` is a large JSON blob (full release group with releases, tracks, media). Loading it for up to 20 records per page is acceptable. The plan does not attempt to extract the selected release in SQL — this is correct, as `Record.releases/1` and `Record.find_release/2` contain release ordering logic (by date, country) that would be impractical to replicate in SQL.
**Verification:**
- Record with valid `selected_release_id` that exists in `musicbrainz_data` → all 6 fields populated
- Record with `selected_release_id` that doesn't match any release in `musicbrainz_data` → `selected_release: nil`
- Record with nil/empty `selected_release_id` → `selected_release: nil`
- Record not in the batch query (not in DB) → `selected_release: nil`
---
### Step 5: Update `CollectionJSON`
Extend the `record/1` function to include new fields using `Map.get/3` with safe defaults:
```elixir
defp record(record) do
%{
# ... existing fields unchanged ...
scrobble_count: Map.get(record, :scrobble_count, 0),
last_listened_at: Map.get(record, :last_listened_at),
artist_country: Map.get(record, :artist_country),
selected_release: Map.get(record, :selected_release)
}
end
```
Using `Map.get/3` ensures backward compatibility: the function works with both raw SearchIndex structs (no enrichment) and enriched maps. No existing fields are removed or renamed.
**Verification:** Controller tests validate the new JSON shape.
---
### Step 6: Update `CollectionController`
Wire enrichment into all four actions. The pattern is:
```elixir
records = Collection.search_records(...)
enriched = Collection.Enrichment.enrich(records)
render(conn, :index, records: enriched, total: total, ...)
```
**Affected actions:**
- `index` — enrich paginated results before render
- `latest` — enrich single record before render (wrap in list, enrich, unwrap)
- `random` — enrich single record before render (same pattern)
- `on_this_day` — enrich results list before render
**Verification:** Integration tests for each endpoint confirm new fields in response.
---
### Step 7: Write tests
**Unit tests** (`test/music_library/collection/enrichment_test.exs`):
- `enrich_scrobbles/1` — with matches, without matches, empty release_ids, shared release_ids
- `enrich_artist_country/1` — with country, without artist_info, without artists
- `enrich_selected_release/1` — with selected release, without selected_release_id, with non-matching release_id
- `enrich/1` — composition of all three, verifying fields accumulate correctly
**Controller tests** (update `test/music_library_web/controllers/collection_controller_test.exs`):
- Update `expected_record_json/1` to include new fields with expected shapes
- Test nil/null values for records without enrichment data
- Test all 4 endpoints (`index`, `latest`, `random`, `on_this_day`) include the new fields
**End-to-end test** — verify the exact API response JSON shape for each of the 4 endpoints with records that have enrichment data (requires fixture setup with scrobbled_tracks, artist_infos, and records with musicbrainz_data).
**Verification:** `mix test test/music_library/collection/enrichment_test.exs test/music_library_web/controllers/collection_controller_test.exs`
---
### Performance profile
| Operation | Query count | Complexity |
| ------------------------- | ----------------------------------------- | -------------------------------------------------------------- |
| `enrich_scrobbles` | 1 batch query | O(R + T) where R=records per page, T=matching scrobbled_tracks |
| `enrich_artist_country` | 1 batch query | O(A) where A=unique artists per page |
| `enrich_selected_release` | 1 batch query + in-memory JSON extraction | O(R) |
| **Total per request** | **exactly 3 queries** | All fixed-cost regardless of total table size |
No N+1 risk — all batch queries scale with page size (≤20 records), not table size. No title+artist fallback queries.
**Memory:** All enrichment data fits in maps proportional to page size. `musicbrainz_data` JSON blobs are loaded for up to 20 records and 20 artists, then discarded after field extraction. At typical MusicBrainz payload sizes (~50-200KB per release group), worst-case memory is under 10MB per request.
**Indexes used:**
- `scrobbled_tracks_album_musicbrainz_id_index` for scrobble batch query (index seek, not table scan)
- `artist_infos` primary key for artist country lookup
- `records` primary key for selected release lookup
No new indexes needed.
---
### Production Changes
None required. This is a read-only change that adds fields to existing API responses. No migrations, no environment variables, no service provisioning.
**Rollback:** Remove the enrichment call from controllers (one line per action). The `CollectionJSON.record/1` function gracefully handles missing enrichment fields via `Map.get/3` defaults.
### Documentation updates
- **`docs/architecture.md`**: Add `MusicLibrary.Collection.Enrichment` to the Contexts + Modules section under `MusicLibrary.Collection`. Note that collection API responses include scrobble stats, artist country, and selected release data.
- **API response documentation**: If any README or doc page describes the `/api/v1/collection` response shape, update it to document the four new fields:
- `scrobble_count` (integer, always present, defaults to 0)
- `last_listened_at` (ISO8601 string or null)
- `artist_country` (`{name: String, code: String}` or null — country of the main artist)
- `selected_release` (`{format, date, country, catalog_number, packaging, disambiguation}` or null)
- **No changes** to `docs/project-conventions.md`, `docs/production-infrastructure.md`, or README needed.
<!-- SECTION:PLAN:END -->