ML-191: avoid VACUUM / scrobble refresh write collision

Set busy_timeout: 5_000 on MusicLibrary.Repo so SQLite waits up to
5 seconds on lock contention instead of throwing SQLITE_BUSY
immediately (default is 0).

Move RepoVacuum from 0 3 to 3 3 (3:03 AM London) to avoid the
deterministic collision with RefreshScrobbles which fires on the
every-5-min cadence at the hour mark.
This commit is contained in:
Claudio Ortolina
2026-05-19 16:45:19 +01:00
parent d8b0dd2327
commit 4a6c80f7a9
3 changed files with 65 additions and 2 deletions
@@ -0,0 +1,61 @@
---
id: ML-191
title: 'Fix SQLite "database busy" error in ListeningStats.update/1 (Error #4075)'
status: Done
assignee: []
created_date: "2026-05-19 15:34"
updated_date: "2026-05-19 15:44"
labels:
- bug
- database
- production-error
dependencies: []
references:
- "https://www.sqlite.org/pragma.html#pragma_busy_timeout"
- "https://hexdocs.pm/ecto_sqlite3/Ecto.Adapters.SQLite3.html"
modified_files:
- config/runtime.exs
- config/prod.exs
priority: medium
ordinal: 33000
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Production error #4075: `Elixir.Exqlite.Error` "Database busy" when `ListeningStats.update/1` performs a 100-row batch `INSERT` into `scrobbled_tracks` during the daily `RefreshScrobbles` cron job at 02:00 UTC.
**Root cause:** Two cron jobs collide on `MusicLibrary.Repo`:
- `RepoVacuum` fires at `0 3 * * *` Europe/London (2 AM UTC during BST) and calls `MusicLibrary.Repo.vacuum()`**acquires an exclusive lock on the entire DB file**
- `RefreshScrobbles` fires at `*/5 * * * *` every 5 minutes (including 2 AM UTC) and calls `ListeningStats.update/1``Repo.insert_all` on the same Repo
When VACUUM holds the exclusive lock and `insert_all` tries to write, SQLite throws `SQLITE_BUSY` immediately because **no `busy_timeout` is configured** in production `config/runtime.exs` (only test has it). The default busy_timeout is 0 = fail immediately instead of waiting.
The collision is deterministic: it happens every day at 2 AM UTC (3 AM London BST). During GMT (winter), the same collision would happen at 3 AM UTC. `RefreshScrobbles` runs every 5 minutes so it always collides with any scheduled maintenance job on the same Repo.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 `busy_timeout: 5_000` is set for `MusicLibrary.Repo` in `config/runtime.exs` production config
- [x] #2 `RepoVacuum` cron schedule in `config/prod.exs` is set to `3 3 * * *` (3 minutes past the hour) to avoid collision with `RefreshScrobbles` every-5-min cadence
- [ ] #3 Error #4075 no longer occurs in production after deploy
<!-- AC:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
**Fix:** SQLite "database busy" error (#4075) on `scrobbled_tracks` INSERT during daily `RefreshScrobbles` cron job.
**Root cause:** `RepoVacuum` (3 AM London = 2 AM UTC BST) runs `VACUUM` which acquires an exclusive DB lock. `RefreshScrobbles` fires every 5 minutes (`*/5`) including at the hour mark. Without `busy_timeout`, SQLite throws `SQLITE_BUSY` immediately instead of waiting.
**Changes:**
- `config/runtime.exs`: Added `busy_timeout: 5_000` to `MusicLibrary.Repo` — SQLite now waits up to 5s on lock contention instead of failing immediately
- `config/prod.exs`: Moved `RepoVacuum` from `0 3 * * *` to `3 3 * * *` — VACUUM runs at :03 past the hour, 2 minutes before the next `RefreshScrobbles` at :05, avoiding the collision
<!-- SECTION:FINAL_SUMMARY:END -->
+3 -2
View File
@@ -49,8 +49,9 @@ config :music_library, Oban,
{"0 */12 * * *", MusicLibrary.Worker.PruneAssetCache}, {"0 */12 * * *", MusicLibrary.Worker.PruneAssetCache},
# every day at 2 am, # every day at 2 am,
{"0 2 * * *", MusicLibrary.Worker.PruneAssets}, {"0 2 * * *", MusicLibrary.Worker.PruneAssets},
# every day at 3 am, # every day at 3:03 am (staggered 3 min past the hour to avoid
{"0 3 * * *", MusicLibrary.Worker.RepoVacuum}, # collision with RefreshScrobbles every-5-min cadence, fixes #4075)
{"3 3 * * *", MusicLibrary.Worker.RepoVacuum},
# every day at 4 am, # every day at 4 am,
{"0 4 * * *", MusicLibrary.Worker.RepoOptimize}, {"0 4 * * *", MusicLibrary.Worker.RepoOptimize},
# every first day of the month, staggered hourly # every first day of the month, staggered hourly
+1
View File
@@ -105,6 +105,7 @@ if config_env() == :prod do
# 128MB * pool_size = base memory usage # 128MB * pool_size = base memory usage
cache_size: -128_000, cache_size: -128_000,
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5"), pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5"),
busy_timeout: 5_000,
show_sensitive_data_on_connection_error: false show_sensitive_data_on_connection_error: false
background_database_path = background_database_path =