From eeb0763466e8baa37ec01e2045dbe2266903531e Mon Sep 17 00:00:00 2001 From: Claudio Ortolina Date: Thu, 9 Apr 2026 15:19:32 +0100 Subject: [PATCH] Deduplicate and consolidate project docs Remove redundant sections from production-infrastructure.md that duplicated architecture.md (Oban queues/plugins/cron, external API integrations, SQLite extensions, scheduled maintenance). Absorb API rate limits into architecture.md's integration table. Replace tool versions table with pointer to mise.toml. Add two conventions from project history: scoped test setup and single-path shared components. --- docs/architecture.md | 21 ++++----- docs/production-infrastructure.md | 72 +------------------------------ docs/project-conventions.md | 2 + 3 files changed, 14 insertions(+), 81 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 85108d87..30448a0f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -142,17 +142,18 @@ Last.fm schemas (separate, not Ecto-persisted to main DB): ## External API Integrations -| Module | API | Purpose | -|--------|-----|---------| -| `MusicBrainz` / `MusicBrainz.API` | musicbrainz.org | Release/artist metadata, search | -| `LastFm` / `LastFm.API` | last.fm | Scrobbling, listening history, artist info (tags, similar artists), user profile/session validation | -| `Discogs` / `Discogs.API` | discogs.com | Artist profiles, images | -| `Wikipedia` / `Wikipedia.API` | wikipedia.org | Artist biographies | -| `BraveSearch` / `BraveSearch.API` | search.brave.com | Cover art and artist image search | -| `OpenAI` / `OpenAI.API` | api.openai.com | Text embeddings for similarity, streaming chat via Responses API (gpt-4.1 + web search), rate-limited | -| `MusicLibrary.Mailer` | Mailgun (via Swoosh) | Transactional email delivery (error notifications, daily digest) | +| Module | API | Rate limit | Purpose | +|--------|-----|-----------|---------| +| `MusicBrainz` / `MusicBrainz.API` | musicbrainz.org | 1000 ms | Release/artist metadata, search | +| `LastFm` / `LastFm.API` | last.fm | 500 ms | Scrobbling, listening history, artist info (tags, similar artists), user profile/session validation | +| `Discogs` / `Discogs.API` | discogs.com | 2000 ms | Artist profiles, images | +| `Wikipedia` / `Wikipedia.API` | wikipedia.org | 1000 ms | Artist biographies | +| `BraveSearch` / `BraveSearch.API` | search.brave.com | 1000 ms | Cover art and artist image search | +| `OpenAI` / `OpenAI.API` | api.openai.com | 250 ms | Text embeddings for similarity, streaming chat via Responses API (gpt-4.1 + web search) | +| `MusicLibrary.Mailer` | Mailgun (via Swoosh) | — | Transactional email delivery (error notifications, daily digest) | -Each has a `Config` module reading from application env. In tests, all HTTP calls are +Each has a `Config` module reading from application env. All HTTP clients use `Req` with +per-API rate limiting (`Req.RateLimiter`, ETS-backed). In tests, all HTTP calls are stubbed via `Req.Test` (configured in `config/test.exs`). --- diff --git a/docs/production-infrastructure.md b/docs/production-infrastructure.md index 9d452336..a8052771 100644 --- a/docs/production-infrastructure.md +++ b/docs/production-infrastructure.md @@ -48,15 +48,6 @@ Three separate SQLite databases, each managed by its own Ecto repo: All databases use incremental auto-vacuum. Paths are configured via environment variables (`DATABASE_PATH`, `BACKGROUND_DATABASE_PATH`, `TELEMETRY_DATABASE_PATH`). -SQLite extensions loaded at runtime: `unicode`, `vec0` (vector search). - -### Scheduled maintenance - -| Schedule | Worker | Purpose | -|----------|--------|---------| -| Daily 3 AM | `RepoVacuum` | Reclaim unused space | -| Daily 4 AM | `RepoOptimize` | Run `PRAGMA optimize` | - --- ## Backups @@ -238,61 +229,6 @@ Used for error notifications and the daily "records on this day" digest email. --- -## External API Integrations - -All HTTP clients use `Req` with per-API rate limiting (`Req.RateLimiter`, ETS-backed). - -| API | Rate limit | Purpose | -|-----|-----------|---------| -| MusicBrainz | 1000 ms cooldown | Release/artist metadata, search | -| Last.fm | 500 ms cooldown | Scrobbling, listening history, artist tags | -| Discogs | 2000 ms cooldown | Artist profiles, images | -| Wikipedia | 1000 ms cooldown | Artist biographies | -| Brave Search | 1000 ms cooldown | Cover art, artist image search | -| OpenAI | 250 ms cooldown | Text embeddings (similarity), streaming chat | - ---- - -## Background Jobs - -Oban with `Oban.Engines.Lite` (SQLite-backed, process-based). - -### Queues - -| Queue | Concurrency | Purpose | -|-------|-------------|---------| -| `default` | 10 | General async tasks | -| `heavy_writes` | 1 | Serialized DB-intensive operations | -| `music_brainz` | 1 | MusicBrainz API calls | -| `discogs` | 1 | Discogs API calls | -| `wikipedia` | 1 | Wikipedia API calls | -| `last_fm` | 1 | Last.fm API calls | - -### Plugins - -- **Pruner**: Removes completed/cancelled/discarded jobs older than 12 hours -- **Reindexer**: Weekly Oban table reindex -- **Cron**: Scheduled jobs (timezone: `Europe/London`) - -### Cron schedule - -| Schedule | Worker | -|----------|--------| -| Every 12h | `ApplyScrobbleRules` | -| Every 12h | `PruneAssetCache` | -| Daily 2 AM | `PruneAssets` | -| Daily 3 AM | `RepoVacuum` | -| Daily 4 AM | `RepoOptimize` | -| Daily 7 AM | `SendRecordsOnThisDayEmail` | -| Monthly 1st, 6 AM | `RecordRefreshAllMusicBrainzData` | -| Monthly 1st, 7 AM | `RecordGenerateAllEmbeddings` | -| Monthly 1st, 8 AM | `ArtistRefreshAllMusicBrainzData` | -| Monthly 1st, 9 AM | `ArtistRefreshAllDiscogsData` | -| Every 5 min | `RefreshScrobbles` | -| Monthly 1st, 10 AM | `ArtistRefreshAllWikipediaData` | - ---- - ## Encryption Cloak vault (`MusicLibrary.Vault`) with AES.GCM cipher for at-rest encryption of @@ -303,10 +239,4 @@ secrets stored in the `secrets` table. Key configured via `CLOAK_ENCRYPTION_KEY` ## Tool Versions -Defined in `mise.toml`: - -| Tool | Version | -|------|---------| -| Elixir | 1.20.0-rc.3-otp-28 | -| Erlang | 28.4.1 | -| Node.js | 24.13.0 | +Defined in `mise.toml`. diff --git a/docs/project-conventions.md b/docs/project-conventions.md index 48317999..88d4f8e6 100644 --- a/docs/project-conventions.md +++ b/docs/project-conventions.md @@ -29,6 +29,7 @@ Rules extracted from commit history that are specific to this project and not al - Extract when duplicated 3+ times. Identical template markup in 3+ places becomes a function component. - Delete thin wrapper modules with a single caller -- inline them instead. - Parameterize the differences when extracting shared logic. +- **When modifying an existing shared component**, refactor all callers to the unified new behavior rather than adding a mode/variant attr. Keep shared components to a single render path. - Private helpers go at module bottom, public functions first. ## Template / UI @@ -70,6 +71,7 @@ Rules extracted from commit history that are specific to this project and not al ## Testing +- **Feature-specific test setup stays in the test modules that need it**, not in shared case templates (DataCase, ConnCase). Only add to shared templates when every test using that template genuinely needs it. - **`@tag :logged_out`** for public endpoint tests. **`@tag :capture_log`** on tests with expected error log output. - **Fixture modules** use `System.unique_integer([:positive])` for unique names and call through context functions (not raw `Repo.insert`). - **Verify outcomes through context modules**, not just UI assertions. Delete tests assert both `refute has_element?` and `assert_raise Ecto.NoResultsError`.