From 4a6c80f7a9dae543eb6c895feb98cf0b3149ad68 Mon Sep 17 00:00:00 2001 From: Claudio Ortolina Date: Tue, 19 May 2026 16:45:19 +0100 Subject: [PATCH] ML-191: avoid VACUUM / scrobble refresh write collision Set busy_timeout: 5_000 on MusicLibrary.Repo so SQLite waits up to 5 seconds on lock contention instead of throwing SQLITE_BUSY immediately (default is 0). Move RepoVacuum from 0 3 to 3 3 (3:03 AM London) to avoid the deterministic collision with RefreshScrobbles which fires on the every-5-min cadence at the hour mark. --- ...r-in-ListeningStats.update-1-Error-4075.md | 61 +++++++++++++++++++ config/prod.exs | 5 +- config/runtime.exs | 1 + 3 files changed, 65 insertions(+), 2 deletions(-) create mode 100644 backlog/tasks/ml-191 - Fix-SQLite-database-busy-error-in-ListeningStats.update-1-Error-4075.md diff --git a/backlog/tasks/ml-191 - Fix-SQLite-database-busy-error-in-ListeningStats.update-1-Error-4075.md b/backlog/tasks/ml-191 - Fix-SQLite-database-busy-error-in-ListeningStats.update-1-Error-4075.md new file mode 100644 index 00000000..5e1ad9ff --- /dev/null +++ b/backlog/tasks/ml-191 - Fix-SQLite-database-busy-error-in-ListeningStats.update-1-Error-4075.md @@ -0,0 +1,61 @@ +--- +id: ML-191 +title: 'Fix SQLite "database busy" error in ListeningStats.update/1 (Error #4075)' +status: Done +assignee: [] +created_date: "2026-05-19 15:34" +updated_date: "2026-05-19 15:44" +labels: + - bug + - database + - production-error +dependencies: [] +references: + - "https://www.sqlite.org/pragma.html#pragma_busy_timeout" + - "https://hexdocs.pm/ecto_sqlite3/Ecto.Adapters.SQLite3.html" +modified_files: + - config/runtime.exs + - config/prod.exs +priority: medium +ordinal: 33000 +--- + +## Description + + + +Production error #4075: `Elixir.Exqlite.Error` "Database busy" when `ListeningStats.update/1` performs a 100-row batch `INSERT` into `scrobbled_tracks` during the daily `RefreshScrobbles` cron job at 02:00 UTC. + +**Root cause:** Two cron jobs collide on `MusicLibrary.Repo`: + +- `RepoVacuum` fires at `0 3 * * *` Europe/London (2 AM UTC during BST) and calls `MusicLibrary.Repo.vacuum()` — **acquires an exclusive lock on the entire DB file** +- `RefreshScrobbles` fires at `*/5 * * * *` every 5 minutes (including 2 AM UTC) and calls `ListeningStats.update/1` → `Repo.insert_all` on the same Repo + +When VACUUM holds the exclusive lock and `insert_all` tries to write, SQLite throws `SQLITE_BUSY` immediately because **no `busy_timeout` is configured** in production `config/runtime.exs` (only test has it). The default busy_timeout is 0 = fail immediately instead of waiting. + +The collision is deterministic: it happens every day at 2 AM UTC (3 AM London BST). During GMT (winter), the same collision would happen at 3 AM UTC. `RefreshScrobbles` runs every 5 minutes so it always collides with any scheduled maintenance job on the same Repo. + + + +## Acceptance Criteria + + + +- [x] #1 `busy_timeout: 5_000` is set for `MusicLibrary.Repo` in `config/runtime.exs` production config +- [x] #2 `RepoVacuum` cron schedule in `config/prod.exs` is set to `3 3 * * *` (3 minutes past the hour) to avoid collision with `RefreshScrobbles` every-5-min cadence +- [ ] #3 Error #4075 no longer occurs in production after deploy + + +## Final Summary + + + +**Fix:** SQLite "database busy" error (#4075) on `scrobbled_tracks` INSERT during daily `RefreshScrobbles` cron job. + +**Root cause:** `RepoVacuum` (3 AM London = 2 AM UTC BST) runs `VACUUM` which acquires an exclusive DB lock. `RefreshScrobbles` fires every 5 minutes (`*/5`) including at the hour mark. Without `busy_timeout`, SQLite throws `SQLITE_BUSY` immediately instead of waiting. + +**Changes:** + +- `config/runtime.exs`: Added `busy_timeout: 5_000` to `MusicLibrary.Repo` — SQLite now waits up to 5s on lock contention instead of failing immediately +- `config/prod.exs`: Moved `RepoVacuum` from `0 3 * * *` to `3 3 * * *` — VACUUM runs at :03 past the hour, 2 minutes before the next `RefreshScrobbles` at :05, avoiding the collision + diff --git a/config/prod.exs b/config/prod.exs index 98f80dd5..cf528281 100644 --- a/config/prod.exs +++ b/config/prod.exs @@ -49,8 +49,9 @@ config :music_library, Oban, {"0 */12 * * *", MusicLibrary.Worker.PruneAssetCache}, # every day at 2 am, {"0 2 * * *", MusicLibrary.Worker.PruneAssets}, - # every day at 3 am, - {"0 3 * * *", MusicLibrary.Worker.RepoVacuum}, + # every day at 3:03 am (staggered 3 min past the hour to avoid + # collision with RefreshScrobbles every-5-min cadence, fixes #4075) + {"3 3 * * *", MusicLibrary.Worker.RepoVacuum}, # every day at 4 am, {"0 4 * * *", MusicLibrary.Worker.RepoOptimize}, # every first day of the month, staggered hourly diff --git a/config/runtime.exs b/config/runtime.exs index 3f3e0c21..a23276a8 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -105,6 +105,7 @@ if config_env() == :prod do # 128MB * pool_size = base memory usage cache_size: -128_000, pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5"), + busy_timeout: 5_000, show_sensitive_data_on_connection_error: false background_database_path =