ML-191: avoid VACUUM / scrobble refresh write collision
Set busy_timeout: 5_000 on MusicLibrary.Repo so SQLite waits up to 5 seconds on lock contention instead of throwing SQLITE_BUSY immediately (default is 0). Move RepoVacuum from 0 3 to 3 3 (3:03 AM London) to avoid the deterministic collision with RefreshScrobbles which fires on the every-5-min cadence at the hour mark.
This commit is contained in:
+61
@@ -0,0 +1,61 @@
|
|||||||
|
---
|
||||||
|
id: ML-191
|
||||||
|
title: 'Fix SQLite "database busy" error in ListeningStats.update/1 (Error #4075)'
|
||||||
|
status: Done
|
||||||
|
assignee: []
|
||||||
|
created_date: "2026-05-19 15:34"
|
||||||
|
updated_date: "2026-05-19 15:44"
|
||||||
|
labels:
|
||||||
|
- bug
|
||||||
|
- database
|
||||||
|
- production-error
|
||||||
|
dependencies: []
|
||||||
|
references:
|
||||||
|
- "https://www.sqlite.org/pragma.html#pragma_busy_timeout"
|
||||||
|
- "https://hexdocs.pm/ecto_sqlite3/Ecto.Adapters.SQLite3.html"
|
||||||
|
modified_files:
|
||||||
|
- config/runtime.exs
|
||||||
|
- config/prod.exs
|
||||||
|
priority: medium
|
||||||
|
ordinal: 33000
|
||||||
|
---
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||||
|
|
||||||
|
Production error #4075: `Elixir.Exqlite.Error` "Database busy" when `ListeningStats.update/1` performs a 100-row batch `INSERT` into `scrobbled_tracks` during the daily `RefreshScrobbles` cron job at 02:00 UTC.
|
||||||
|
|
||||||
|
**Root cause:** Two cron jobs collide on `MusicLibrary.Repo`:
|
||||||
|
|
||||||
|
- `RepoVacuum` fires at `0 3 * * *` Europe/London (2 AM UTC during BST) and calls `MusicLibrary.Repo.vacuum()` — **acquires an exclusive lock on the entire DB file**
|
||||||
|
- `RefreshScrobbles` fires at `*/5 * * * *` every 5 minutes (including 2 AM UTC) and calls `ListeningStats.update/1` → `Repo.insert_all` on the same Repo
|
||||||
|
|
||||||
|
When VACUUM holds the exclusive lock and `insert_all` tries to write, SQLite throws `SQLITE_BUSY` immediately because **no `busy_timeout` is configured** in production `config/runtime.exs` (only test has it). The default busy_timeout is 0 = fail immediately instead of waiting.
|
||||||
|
|
||||||
|
The collision is deterministic: it happens every day at 2 AM UTC (3 AM London BST). During GMT (winter), the same collision would happen at 3 AM UTC. `RefreshScrobbles` runs every 5 minutes so it always collides with any scheduled maintenance job on the same Repo.
|
||||||
|
|
||||||
|
<!-- SECTION:DESCRIPTION:END -->
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
<!-- AC:BEGIN -->
|
||||||
|
|
||||||
|
- [x] #1 `busy_timeout: 5_000` is set for `MusicLibrary.Repo` in `config/runtime.exs` production config
|
||||||
|
- [x] #2 `RepoVacuum` cron schedule in `config/prod.exs` is set to `3 3 * * *` (3 minutes past the hour) to avoid collision with `RefreshScrobbles` every-5-min cadence
|
||||||
|
- [ ] #3 Error #4075 no longer occurs in production after deploy
|
||||||
|
<!-- AC:END -->
|
||||||
|
|
||||||
|
## Final Summary
|
||||||
|
|
||||||
|
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
|
||||||
|
|
||||||
|
**Fix:** SQLite "database busy" error (#4075) on `scrobbled_tracks` INSERT during daily `RefreshScrobbles` cron job.
|
||||||
|
|
||||||
|
**Root cause:** `RepoVacuum` (3 AM London = 2 AM UTC BST) runs `VACUUM` which acquires an exclusive DB lock. `RefreshScrobbles` fires every 5 minutes (`*/5`) including at the hour mark. Without `busy_timeout`, SQLite throws `SQLITE_BUSY` immediately instead of waiting.
|
||||||
|
|
||||||
|
**Changes:**
|
||||||
|
|
||||||
|
- `config/runtime.exs`: Added `busy_timeout: 5_000` to `MusicLibrary.Repo` — SQLite now waits up to 5s on lock contention instead of failing immediately
|
||||||
|
- `config/prod.exs`: Moved `RepoVacuum` from `0 3 * * *` to `3 3 * * *` — VACUUM runs at :03 past the hour, 2 minutes before the next `RefreshScrobbles` at :05, avoiding the collision
|
||||||
|
<!-- SECTION:FINAL_SUMMARY:END -->
|
||||||
+3
-2
@@ -49,8 +49,9 @@ config :music_library, Oban,
|
|||||||
{"0 */12 * * *", MusicLibrary.Worker.PruneAssetCache},
|
{"0 */12 * * *", MusicLibrary.Worker.PruneAssetCache},
|
||||||
# every day at 2 am,
|
# every day at 2 am,
|
||||||
{"0 2 * * *", MusicLibrary.Worker.PruneAssets},
|
{"0 2 * * *", MusicLibrary.Worker.PruneAssets},
|
||||||
# every day at 3 am,
|
# every day at 3:03 am (staggered 3 min past the hour to avoid
|
||||||
{"0 3 * * *", MusicLibrary.Worker.RepoVacuum},
|
# collision with RefreshScrobbles every-5-min cadence, fixes #4075)
|
||||||
|
{"3 3 * * *", MusicLibrary.Worker.RepoVacuum},
|
||||||
# every day at 4 am,
|
# every day at 4 am,
|
||||||
{"0 4 * * *", MusicLibrary.Worker.RepoOptimize},
|
{"0 4 * * *", MusicLibrary.Worker.RepoOptimize},
|
||||||
# every first day of the month, staggered hourly
|
# every first day of the month, staggered hourly
|
||||||
|
|||||||
@@ -105,6 +105,7 @@ if config_env() == :prod do
|
|||||||
# 128MB * pool_size = base memory usage
|
# 128MB * pool_size = base memory usage
|
||||||
cache_size: -128_000,
|
cache_size: -128_000,
|
||||||
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5"),
|
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5"),
|
||||||
|
busy_timeout: 5_000,
|
||||||
show_sensitive_data_on_connection_error: false
|
show_sensitive_data_on_connection_error: false
|
||||||
|
|
||||||
background_database_path =
|
background_database_path =
|
||||||
|
|||||||
Reference in New Issue
Block a user