From d5d3cae064d8660964848b574d47a04e72db0730 Mon Sep 17 00:00:00 2001 From: Claudio Ortolina Date: Tue, 10 Feb 2026 08:06:33 +0000 Subject: [PATCH] Add project architecture document --- AGENTS.md | 4 + ARCHITECTURE.md | 341 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 345 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/AGENTS.md b/AGENTS.md index 2aa91980..d46a97c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3 +1,7 @@ +# Project architecture + +[Project architecture](architecture.md) + # Usage Rules diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..edbd1f17 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,341 @@ +# Architecture Summary + +> **Maintenance rule**: Update this file whenever you add, remove, or restructure modules, +> schemas, contexts, workers, routes, or external integrations. Keep descriptions factual +> and concise. This file exists to accelerate future iteration — treat it as a living map. + +## Overview + +Phoenix LiveView application for managing a personal music collection and wishlist. +Uses SQLite (via `ecto_sqlite3`) with two databases: one for the app, one for background +jobs (Oban). All schemas use `binary_id` primary keys. + +Key capabilities: +- Browse/search collected and wishlisted records +- Import metadata from MusicBrainz, enrich with Discogs/Wikipedia/Last.fm +- Scrobble tracks to Last.fm, import listening history +- Similarity search via OpenAI embeddings + sqlite-vec +- Barcode scanning for quick imports +- Encrypted secret storage (Cloak) + +**Elixir ~> 1.14, Phoenix ~> 1.8, LiveView ~> 1.1, SQLite3** + +--- + +## Supervision Tree + +``` +MusicLibrary.Application (one_for_one) +├── MusicLibrary.Vault # Cloak encryption vault +├── MusicLibrary.Repo # Main SQLite repo +├── MusicLibrary.BackgroundRepo # Oban SQLite repo (separate DB) +├── MusicLibraryWeb.Telemetry # Telemetry supervisor +│ ├── Telemetry.Storage # Metrics storage (circular buffer) +│ └── :telemetry_poller # 30s periodic measurements +├── Oban # Background job engine +├── Ecto.Migrator # Auto-migration on boot +├── Phoenix.PubSub (:music_library) +├── LastFm.Supervisor (one_for_one) +│ ├── Phoenix.PubSub (:last_fm) # Last.fm-specific PubSub +│ └── LastFm.Refresh # GenServer, periodic scrobble fetch +└── MusicLibraryWeb.Endpoint +``` + +--- + +## Database & Repos + +| Repo | DB file (dev) | Purpose | +|------|---------------|---------| +| `MusicLibrary.Repo` | `data/music_library_dev.db` | All application data | +| `MusicLibrary.BackgroundRepo` | `data/music_library_background_dev.db` | Oban job queue | + +SQLite extensions loaded at runtime: `unicode`, `vec0` (vector search). + +FTS5 virtual table `records_search_index` auto-synced via database triggers — do not +write to it directly; insert/update the `records` table instead. + +--- + +## Schemas + +| Schema | Table | PK | Key Fields | +|--------|-------|----|------------| +| `Records.Record` | `records` | `id` (binary_id) | title, type, format, cover_url, cover_hash, musicbrainz_id, genres[], release_date, purchased_at, dominant_colors[], embeds_many :artists | +| `Records.RecordEmbedding` | `record_embeddings` | `id` | embedding (Float32 vector), text_representation, belongs_to :record | +| `Records.SearchIndex` | `records_search_index` | `id` | FTS5 mirror of records (virtual, trigger-synced) | +| `Records.ArtistRecord` | `artist_records` | composite | musicbrainz_id, record_id — DB view joining artists to records | +| `Artists.Artist` | — | — | Embedded schema (name, sort_name, musicbrainz_id, joinphrase) | +| `Artists.ArtistInfo` | `artist_infos` | `id` | musicbrainz_data, discogs_data, wikipedia_data, image_data_hash | +| `Assets.Asset` | `assets` | `hash` (SHA256) | content (binary), format, properties (map) | +| `Notes.Note` | `notes` | `id` | entity (:record/:artist), content, musicbrainz_id | +| `RecordSets.RecordSet` | `record_sets` | `id` | name, description, has_many :items | +| `RecordSets.RecordSetItem` | `record_set_items` | `id` | position, belongs_to :record_set, belongs_to :record | +| `ScrobbleRules.ScrobbleRule` | `scrobble_rules` | `id` (integer) | type (:album/:artist), match_value, target_musicbrainz_id, enabled | +| `OnlineStoreTemplates.OnlineStoreTemplate` | `online_store_templates` | `id` | name, url_template, enabled | +| `Secrets.Secret` | `secrets` | `name` (string) | value (encrypted binary) | + +Last.fm schemas (separate, not Ecto-persisted to main DB): +- `LastFm.Track` — scrobbled tracks stored in Last.fm's own tables via `LastFm.Feed` +- `LastFm.Album`, `LastFm.Artist` — parsed API responses + +--- + +## Contexts (lib/music_library/) + +| Context | Schemas | Responsibility | +|---------|---------|---------------| +| `Records` | Record, RecordEmbedding, SearchIndex | CRUD, search, import from MusicBrainz, cover/genre/embedding management, PubSub notifications | +| `Collection` | Record (via SearchIndex) | Querying collected records (purchased_at != nil), stats | +| `Wishlist` | Record (via SearchIndex) | Querying wishlisted records (purchased_at is nil) | +| `Artists` | ArtistInfo, ArtistRecord | Artist metadata from MusicBrainz/Discogs/Wikipedia, images | +| `Assets` | Asset | Binary asset storage (covers, artist images), cache tracking | +| `Notes` | Note | Free-text notes for records and artists | +| `RecordSets` | RecordSet, RecordSetItem | User-curated record groupings with ordering | +| `ScrobbleRules` | ScrobbleRule | Rules to remap Last.fm scrobble data to correct MusicBrainz IDs | +| `ScrobbleActivity` | (LastFm.Track, ArtistRecord) | Scrobbling releases, listening stats, top albums/artists | +| `OnlineStoreTemplates` | OnlineStoreTemplate | URL templates for buying records online | +| `Search` | (cross-context) | Universal search across collection, wishlist, artists, record sets | +| `Secrets` | Secret | Encrypted key-value storage | +| `BarcodeScan` | (Result struct) | Barcode → MusicBrainz lookup workflow | +| `Colors` | — | Dominant color extraction from cover images | + +--- + +## Business Logic Modules + +| Module | Purpose | +|--------|---------| +| `Records.SearchParser` | Parses search syntax: `artist:X`, `album:X`, `genre:"Y"`, `format:cd`, `type:album`, `purchase_year:2024`, free text | +| `Records.Similarity` | Embedding generation (OpenAI), cosine-distance search (sqlite-vec) | +| `Records.Batch` | Batch operations: refresh all MusicBrainz data, generate all embeddings | +| `Artists.Batch` | Batch refresh: MusicBrainz, Discogs, Wikipedia for all artists | +| `Assets.Cache` | ETS-based asset cache with TTL | +| `Assets.Image` / `Assets.Transform` | Image processing via Vix (libvips) | +| `FormatNumber` | Number formatting utility | + +--- + +## External API Integrations + +| Module | API | Purpose | +|--------|-----|---------| +| `MusicBrainz` / `MusicBrainz.API` | musicbrainz.org | Release/artist metadata, search | +| `LastFm` / `LastFm.API` | last.fm | Scrobbling, listening history, artist info | +| `Discogs` / `Discogs.API` | discogs.com | Artist profiles, images | +| `Wikipedia` / `Wikipedia.API` | wikipedia.org | Artist biographies | +| `BraveSearch` / `BraveSearch.API` | search.brave.com | Cover art image search | +| `OpenAI` / `OpenAI.API` | api.openai.com | Text embeddings for similarity | + +Each has a `Config` module reading from application env. In tests, all HTTP calls are +stubbed via `Req.Test` (configured in `config/test.exs`). + +--- + +## Oban Workers (lib/music_library/worker/) + +### Queues + +| Queue | Concurrency | Purpose | +|-------|-------------|---------| +| `default` | 10 | General async tasks | +| `heavy_writes` | 1 | DB-intensive or serialized operations | +| `music_brainz` | 1 | Rate-limited MusicBrainz calls (500ms delay) | +| `discogs` | 1 | Rate-limited Discogs calls (1s delay) | +| `wikipedia` | 1 | Rate-limited Wikipedia calls | + +### On-Demand Workers + +| Worker | Queue | Trigger | +|--------|-------|---------| +| `FetchArtistInfo` | default | Artist page visit / import | +| `FetchArtistImage` | heavy_writes | Artist info fetched | +| `RefreshCover` | heavy_writes | Manual action / import | +| `ExtractColors` | heavy_writes | Manual action / import | +| `PopulateGenres` | heavy_writes | Manual action (chains → GenerateRecordEmbedding) | +| `GenerateRecordEmbedding` | heavy_writes | Manual / after genre population | +| `RecordRefreshMusicBrainzData` | music_brainz | Manual / batch | +| `ArtistRefreshMusicBrainzData` | music_brainz | Manual / batch | +| `ArtistRefreshDiscogsData` | discogs | Manual / batch | +| `ArtistRefreshWikipediaData` | wikipedia | Manual / batch | +| `PruneArtistInfo` | default | Record deleted (cleanup orphaned artist data) | +| `BackfillScrobbledTracks` | heavy_writes | Manual (self-chaining batch import) | + +### Cron Workers + +| Schedule | Worker | Queue | +|----------|--------|-------| +| Every 12h | `ApplyScrobbleRules` | heavy_writes | +| Every 12h | `PruneAssetCache` | default | +| Daily 2 AM | `PruneAssets` | default | +| Daily 3 AM | `RepoVacuum` | heavy_writes | +| Daily 4 AM | `RepoOptimize` | heavy_writes | + +--- + +## PubSub Topics + +| PubSub | Topic Pattern | Message | Used By | +|--------|---------------|---------|---------| +| `:music_library` | `"records:#{id}"` | `{:update, record}` | CollectionLive.Show, WishlistLive.Show — real-time record updates | +| `:last_fm` | `"feed:update"` | `%{track_count: n}` | StatsLive.Index, ScrobbledTracksLive.Index — new scrobbles arrived | + +--- + +## Web Layer (lib/music_library_web/) + +### Router Structure + +All authenticated routes live inside a single `live_session` with three `on_mount` hooks: +- `StaticAssets` — detects app updates, shows toast +- `GetTimezone` — reads timezone from connect params +- `ShowToast` — enables `put_toast!/2` in LiveViews + +### LiveViews + +| LiveView | Route | Purpose | +|----------|-------|---------| +| `StatsLive.Index` | `/` | Dashboard: counts, recent activity, records on this day | +| `CollectionLive.Index` | `/collection` | Browse/search collected records (grid/list, paginated) | +| `CollectionLive.Show` | `/collection/:id` | Record detail: metadata, scrobbles, similar, colors | +| `WishlistLive.Index` | `/wishlist` | Browse/search wishlisted records | +| `WishlistLive.Show` | `/wishlist/:id` | Wishlist record detail with store links | +| `ArtistLive.Show` | `/artists/:musicbrainz_id` | Artist bio, discography, similar artists | +| `RecordSetLive.Index` | `/record-sets` | Browse/manage curated record sets | +| `RecordSetLive.Show` | `/record-sets/:id` | Set detail with reorderable items | +| `ScrobbleLive.Index` | `/scrobble` | Search MusicBrainz releases to scrobble | +| `ScrobbleLive.Show` | `/scrobble/:release_id` | Select tracks and scrobble | +| `ScrobbledTracksLive.Index` | `/scrobbled-tracks` | Browse/search Last.fm history | +| `ScrobbleRulesLive.Index` | `/scrobble-rules` | Manage scrobble remapping rules | +| `OnlineStoreTemplateLive.Index` | `/online-store-templates` | Manage store URL templates | +| `MaintenanceLive.Index` | `/dev/maintenance` | Admin: batch jobs, DB maintenance | + +### LiveComponents + +| Component | Used In | Purpose | +|-----------|---------|---------| +| `RecordForm` | Collection/Wishlist (edit) | Record editing: cover search, genre autocomplete, color picker, file upload | +| `ArtistLive.Form` | ArtistLive.Show | Edit artist image/bio | +| `RecordSetLive.Form` | RecordSetLive.Index | Create/edit record set | +| `RecordSetLive.RecordPicker` | RecordSetLive.Show | Search and add records to set | +| `ScrobbledTracksLive.Form` | ScrobbledTracksLive.Index | Edit scrobbled track | +| `ScrobbleRulesLive.Form` | ScrobbleRulesLive.Index | Create/edit scrobble rule | +| `OnlineStoreTemplateLive.Form` | OnlineStoreTemplateLive.Index | Create/edit store template | +| `StatsLive.TopAlbums` | StatsLive.Index | Top albums by period (assign_async) | +| `StatsLive.TopArtists` | StatsLive.Index | Top artists by period (assign_async) | +| `UniversalSearchLive.Index` | Layout (global) | Cmd+K search modal | + +### Shared Component Modules (lib/music_library_web/components/) + +| Module | Purpose | +|--------|---------| +| `CoreComponents` | Forms, buttons, icons, tables, flash messages | +| `RecordComponents` | Record cards, cover images, labels, grids | +| `ArtistComponents` | Artist images, links | +| `ChartComponents` | SVG charts for stats | +| `StatsComponents` | Stats dashboard widgets | +| `ScrobbleComponents` | Scrobble activity displays | +| `SearchComponents` | Search result rendering | +| `Pagination` | Pagination UI and logic | +| `AddRecord` | MusicBrainz import interface | +| `BarcodeScanner` | Barcode scanning UI (uses barcode-detector JS) | +| `Release` | MusicBrainz release display | +| `Notes` | Markdown note rendering | + +### Controllers + +| Controller | Routes | Purpose | +|------------|--------|---------| +| `SessionController` | `/login`, `/sessions/create` | Login/logout | +| `HealthController` | `/health` | Health check | +| `LastFmController` | `/auth/last_fm/callback` | Last.fm OAuth | +| `ArchiveController` | `/backup` | Database backup download | +| `AssetController` | `/assets/:transform_payload` | Serve images with transforms | +| `CollectionController` | `/api/collection/*` | JSON API for collection queries | + +--- + +## Frontend (assets/) + +- **Bundler**: esbuild +- **CSS**: Tailwind CSS + Fluxon UI component library +- **JS entry**: `assets/js/app.js` + +### JS Hooks + +| Hook | Type | Purpose | +|------|------|---------| +| `FormatNumber` | External (`assets/js/hooks/`) | Client-side number formatting | +| `UniversalSearchNavigation` | External | Keyboard navigation in search modal | +| `RecordPickerNavigation` | External | Keyboard navigation in record picker | +| Various `.ColocatedHooks` | Colocated (in .heex) | Inline hooks prefixed with `.` | + +### JS Event Listeners (app.js) + +| Event | Action | +|-------|--------| +| `clipcopy` | Copy text to clipboard | +| `scroll_top` | Scroll window to top | +| `confetti` | Trigger canvas-confetti animation | + +### NPM Dependencies + +- `barcode-detector` — Barcode scanning API +- `canvas-confetti` — Confetti animation +- `live_toast` — Toast notifications (local dep) + +--- + +## Testing Patterns + +### Test Support + +| Module | Purpose | +|--------|---------| +| `ConnCase` | HTTP test setup, auto-logged-in session | +| `DataCase` | Database test setup with Ecto sandbox | +| `LiveTestHelpers` | `escape/1` for HTML-escaped text assertions | + +### Fixture Modules (test/support/fixtures/) + +| Module | Creates | +|--------|---------| +| `MusicLibrary.RecordsFixtures` | Records with MusicBrainz data | +| `MusicLibrary.RecordSetsFixtures` | Record sets with items | +| `MusicLibrary.OnlineStoreTemplatesFixtures` | Store templates | +| `ScrobbleRulesFixtures` | Scrobble rules | +| `ScrobbledTracksFixtures` | Last.fm tracks | +| `Discogs.ArtistFixtures` | Discogs API responses | +| `LastFm.ArtistFixtures` | Last.fm API responses | +| `MusicBrainz.*Fixtures` | MusicBrainz API responses | +| `Wikipedia.Fixtures` | Wikipedia API responses | + +### Test Styles + +1. **PhoenixTest** (`visit`, `assert_has`, `click_button`, `click_link`) — used for page-level + LiveView tests (Collection, Wishlist, Stats, etc.) +2. **Phoenix.LiveViewTest** (`live/2`, `form/3`, `render_submit/1`, `element/2`, `render_click/1`) — + needed for LiveComponent interactions (`phx-target={@myself}`) +3. **Context tests** — standard ExUnit with DataCase + +### SQLite Test Gotchas + +- `utc_datetime` has second-level precision — rapid inserts get identical timestamps +- Use `Repo.update_all` to manually set timestamps for deterministic ordering tests +- External APIs stubbed via `Req.Test` (see `config/test.exs`) +- Oban runs in manual testing mode (jobs don't auto-execute) + +--- + +## Key Conventions + +- **Streams everywhere**: Collection/Wishlist lists, recent activity, record sets, scrobbled tracks + all use LiveView streams (not assigns) for memory efficiency +- **Async enrichment**: Record/artist import triggers a cascade of Oban workers for metadata, + covers, colors, embeddings — all non-blocking +- **Search**: FTS5 for text search, custom `SearchParser` for structured queries, sqlite-vec + for similarity +- **Record lifecycle**: `purchased_at` distinguishes collection (set) from wishlist (nil) +- **`use MusicLibraryWeb, :live_view`** imports: Phoenix.LiveView, Fluxon, Gettext, LiveToast, + CoreComponents, Phoenix.HTML, verified routes, JS alias +- **`use MusicLibraryWeb, :live_component`** additionally imports `put_toast!/2`