Claudio Ortolina
d8c84e787b
Three independent refactors of lib/music_library/listening_stats.ex, measured against the dev DB (104k tracks) via bench/listening_stats.exs: | Query | Baseline | After | Speedup | |---------------------------------|-----------|-----------|---------| | recent_activity(tz, 100) | 38.98 ms | 2.87 ms | 13.6x | | list_tracks(page 1, 200) | 47.30 ms | 4.81 ms | 9.8x | | get_top_artists_by_days(7) | 66.51 ms | 1.02 ms | 65x | | get_top_artists_by_days(30) | 67.41 ms | 2.84 ms | 23.7x | | get_top_artists_by_days(365) | 76.03 ms | 25.57 ms | 3.0x | | get_top_albums_by_days(7) | 92.63 ms | 1.61 ms | 57.5x | | get_top_albums_by_days(30) | 94.52 ms | 3.69 ms | 25.6x | | get_top_albums_by_days(365) | 105.23 ms | 32.81 ms | 3.2x | | get_top_artists(limit: 10) | 123.96 ms | 101.68 ms | 1.22x | | get_top_albums(limit: 10) | 209.64 ms | 108.23 ms | 1.94x | tracks_with_record_info_query/0 now uses correlated scalar subqueries against record_releases and artist_records instead of materializing helper subqueries on every call. The cost scales with the outer LIMIT, not with the size of record_releases. top_albums_base_query/0 and top_artists_base_query/0 are replaced by aggregate_query + attach_metadata pairs. The pattern is aggregate first, attach metadata second: GROUP BY runs against the raw track scan, then a tiny outer SELECT attaches record_releases / artist_infos lookups for the <= 10 result rows via correlated subqueries. tracks_since_query/1 wraps the date-filtered inner scan with limit: -1. SQLite cannot flatten a subquery that has a LIMIT, so it materializes the date-bounded subset and the optimizer uses the timestamp index for the range scan instead of the album/artist composite index. This trick replaces SQLite's WITH ... AS MATERIALIZED (which ecto_sqlite3 doesn't expose). All json_extract(?, '\$.path') fragments use the canonical form rather than the equivalent ? ->> '\$.path' shorthand. SQLite's index matcher requires the GROUP BY expression to match the index expression textually to use the composite index for natural ordering. collected_releases_query/0 and wishlisted_releases_query/0 are removed from Collection and Wishlist — they had a single internal caller that no longer exists after the refactor. New regression tests lock the semantics that the optimized queries must preserve: - count(DISTINCT scrobbled_at_uts) — 579 duplicate timestamps exist in the dev DB from rapid Last.fm scrobbles, so replacing with count(*) would silently change results - :artist_id key in list_tracks result maps — ScrobbledTracksLive.Index destructures it even though the template body never references it Closes #148
Music Library
- Music Library
Features
- Add records from MusicBrainz, with optional override of specific pieces of data
- Manage a collection and a wishlist of records, with ways to quickly search and filter based on records' metadata
- Browse record releases and select a collected release
- AI-powered chat for records and artists, with web search
- Artist details with biography, discography, and similar artists
- Wishlist with links to online stores for purchasing
- Curate record sets (e.g. "best live albums")
- Universal search across collection, wishlist, artists, and record sets
- Integration with Last.fm:
- display latest scrobbles, and where possible connect them with records in the collection or wishlist
- scrobble a record
- store a local copy of the complete scrobble history, and setup rules to fix its data as needed
- audit scrobble data quality and identify tracks with missing MusicBrainz IDs
- Similarity search via OpenAI embeddings
- Barcode scanning for quick imports
- Stats dashboard with collection overview, top artists, top albums, and records on this day
- All data stored in SQLite databases for portability and ease of backup/restore
Screenshots
Stats dashboard
Collection
Searching for a record to add
Record details
Record releases
Record chat
Artist details
Wishlist record details
Record sets
Scrobble rules
New scrobble rule
Universal search
Setup
The project is managed and configured via mise-en-place:
mise installwill pull the correct Erlang, Elixir and Node.js versionsmise run dev:setupwill setup dependencies and database structure
Important
The project uses Fluxon UI, so it requires a valid set of credentials. See the
envsection inmise.tomlfor the required environment variables.
It's recommended to use the git hooks included in the project. Install with:
mise generate git-pre-commit --write --task=dev:precommit
Environment configuration
Required environment variables for development are listed in mise.toml.
You can create a mise.local.toml with the required variables (sample values
are included at the top of mise.toml).
For production, please see compose.yaml for a list of required variables.
Running the application
Start the Phoenix endpoint with mise run console (along with an attached IEx session).
Auditing Scrobble Data Quality
The application includes a Mix task to audit scrobbled tracks and identify data quality issues such as missing MusicBrainz IDs for artists and albums.
Running the Audit
# Audit all tracks
mix scrobble.audit
# Audit with detailed output including sample tracks
mix scrobble.audit --verbose
# Audit only artist issues
mix scrobble.audit --type artist
# Audit only album issues
mix scrobble.audit --type album
# Output as JSON for processing
mix scrobble.audit --format json
Understanding the Audit Report
The audit report shows:
- Total number of scrobbled tracks
- Artists with missing MusicBrainz IDs (grouped by artist name)
- Albums with missing MusicBrainz IDs (grouped by album title and artist)
- Track counts for each issue
Fixing Data Quality Issues
After identifying issues, you can:
-
Create Scrobble Rules: Navigate to the Scrobble Rules page in the web interface and add rules to map artist or album names to their correct MusicBrainz IDs.
-
Apply Rules: Use the "Apply Rules" button in the Scrobble Rules page to update existing tracks, or run in IEx:
MusicLibrary.ScrobbleRules.apply_all_rules() -
Re-audit: Run the audit again to verify the fixes worked.
The application also provides helper functions in the MusicLibrary.ScrobbleActivity context:
count_tracks_missing_artist_musicbrainz_id/0count_tracks_missing_album_musicbrainz_id/0get_artists_missing_musicbrainz_id/1get_albums_missing_musicbrainz_id/1
Deployment
The application is deployed via Coolify, using a Docker Compose strategy.
CI
See the .github folder.
Architecture
See the docs folder.
Favicons
This favicon was generated using the following graphics from Twitter Twemoji:
- Graphics Title: 1f4bd.svg
- Graphics Author: Copyright 2020 Twitter, Inc and other contributors (https://github.com/twitter/twemoji)
- Graphics Source: https://github.com/twitter/twemoji/blob/master/assets/svg/1f4bd.svg
- Graphics License: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
