ML-200: daily scrobble count charts (plan)
This commit is contained in:
Claudio Ortolina
@@ -0,0 +1,114 @@
|
||||
---
|
||||
id: doc-30
|
||||
title: "ML-200 Research: Scrobble daily count widget routes"
|
||||
type: specification
|
||||
created_date: "2026-05-31 18:05"
|
||||
updated_date: "2026-05-31 18:15"
|
||||
tags:
|
||||
- research
|
||||
- stats
|
||||
- scrobbles
|
||||
- liveview
|
||||
---
|
||||
|
||||
# ML-200 Research: Scrobble daily count widget routes
|
||||
|
||||
## Problem
|
||||
|
||||
The stats dashboard should show a horizontal bar chart immediately before the existing Scrobble Activity section. The chart should cover the last 30 calendar days and display the count of scrobbled tracks per day.
|
||||
|
||||
## Current architecture touchpoints
|
||||
|
||||
- `MusicLibraryWeb.StatsLive.Index` renders the `/` stats page. It already imports `MusicLibraryWeb.ChartComponents`, subscribes to `ListeningStats` PubSub updates when connected, and refreshes scrobble-related assigns when new scrobbles arrive.
|
||||
- `MusicLibrary.ListeningStats` owns listening analytics queries. LiveViews should call this context rather than querying `LastFm.Track` directly.
|
||||
- `LastFm.Track` persists scrobbles in `scrobbled_tracks`; `scrobbled_at_uts` is an integer Unix timestamp in seconds.
|
||||
- The existing unique index `scrobbled_tracks_scrobbled_at_uts_title_index` supports range scans on `scrobbled_at_uts`. `EXPLAIN QUERY PLAN SELECT scrobbled_at_uts FROM scrobbled_tracks WHERE scrobbled_at_uts >= ? ORDER BY scrobbled_at_uts ASC` reports a covering index search using that index.
|
||||
- `MusicLibraryWeb.ChartComponents.horizontal_bar_chart/1` renders horizontal bars with CSS Grid. Project UI conventions prefer CSS Grid charts over SVG.
|
||||
- The stats page receives the user/browser timezone via `socket.assigns.timezone`; existing top-period stats use this timezone to compute day cutoffs.
|
||||
|
||||
Local development data observed during research: `scrobbled_tracks` contains 105,936 total rows and 1,002 rows in the most recent 30 days. This is useful as a realistic order-of-magnitude check for a personal library dashboard read.
|
||||
|
||||
## Route A — Context range scan, group by local day in Elixir, render with existing chart component
|
||||
|
||||
Add a `ListeningStats` function that accepts `timezone`, `days`, and an optional `current_time` for tests. It computes the local 30-day window, queries only `scrobbled_at_uts` values in that range, converts each timestamp to the user's timezone, groups by `Date`, fills missing days with zero counts, and returns a fixed 30-item list sorted by date. `StatsLive.Index` assigns the result, renders it before Scrobble Activity using `ChartComponents.horizontal_bar_chart/1` or a small stats-specific wrapper, and refreshes it from the existing `listening_stats:update` PubSub path.
|
||||
|
||||
Pros:
|
||||
|
||||
- Correct for the user's IANA timezone and DST boundaries because grouping happens through Elixir time zone conversion.
|
||||
- No migration, trigger, background job, cache, or external API required.
|
||||
- Keeps query ownership in `ListeningStats` and rendering in the stats web layer.
|
||||
- Reuses the existing CSS Grid chart approach.
|
||||
- Query is a narrow covering index range scan and returns only one integer column for the last 30 days.
|
||||
|
||||
Cons:
|
||||
|
||||
- Fetches one row per scrobble in the 30-day window before grouping. This is acceptable for the current app scale but is not constant-time.
|
||||
- Needs clear tests around timezone boundaries and zero-count days.
|
||||
|
||||
Performance profile:
|
||||
|
||||
- Runtime: O(n + d), where `n` is scrobbles in the 30-day window and `d = 30`.
|
||||
- Database: one indexed range scan over `scrobbled_at_uts`, no joins, no N+1 risk.
|
||||
- Memory: one integer per recent scrobble plus a 30-entry result map/list.
|
||||
- Expected local scale: approximately 1,000 rows for 30 days based on current dev data.
|
||||
|
||||
## Route B — SQL aggregate by UTC calendar day, fill gaps in Elixir
|
||||
|
||||
Use SQLite aggregation directly, for example grouping by `date(scrobbled_at_uts, 'unixepoch')`, and fill missing days in Elixir before rendering.
|
||||
|
||||
Pros:
|
||||
|
||||
- Returns at most 30 aggregate rows to Elixir.
|
||||
- Simple query shape and no schema change.
|
||||
- Still uses the `scrobbled_at_uts` range index for the date filter.
|
||||
|
||||
Cons:
|
||||
|
||||
- Groups by UTC days, not the user's browser timezone. This can move late-night/early-morning scrobbles to the wrong displayed day for the stats page.
|
||||
- SQLite does not understand IANA timezones or DST transitions, so accurate browser-timezone grouping cannot be expressed directly in SQLite without additional application logic.
|
||||
- `EXPLAIN QUERY PLAN` for the UTC aggregate still shows a temporary B-tree for `GROUP BY`.
|
||||
|
||||
Use this only if the product decision is that UTC days are acceptable for this widget. That would be inconsistent with the current stats page timezone handling.
|
||||
|
||||
## Route C — Materialized daily scrobble count table
|
||||
|
||||
Create a rollup table such as `daily_scrobble_counts` and maintain it via migration/backfill plus insert/update/delete handling when `scrobbled_tracks` changes. The stats page reads 30 rows from the rollup.
|
||||
|
||||
Pros:
|
||||
|
||||
- Dashboard read is constant and tiny.
|
||||
- Could support future long-range daily charts cheaply.
|
||||
|
||||
Cons:
|
||||
|
||||
- Adds migration, backfill, maintenance logic, and correctness concerns around deletes/updates to scrobble timestamps.
|
||||
- Still must choose a timezone for the stored day bucket. A UTC rollup has the same product issue as Route B; a local-time rollup would only be correct for one configured timezone and would not match arbitrary browser timezones.
|
||||
- Over-engineered for a 30-day personal dashboard widget at current data volume.
|
||||
|
||||
This route should be deferred unless dashboard reads become measurably slow or daily rollups become a broader analytics requirement.
|
||||
|
||||
## Route D — Reuse existing recent activity data or fetch Last.fm again
|
||||
|
||||
Either derive daily counts from the current `recent_activity/2` result or call Last.fm for a fresh 30-day activity window.
|
||||
|
||||
Pros:
|
||||
|
||||
- Reusing existing data sounds superficially simple.
|
||||
|
||||
Cons:
|
||||
|
||||
- `recent_activity/2` defaults to a 100-track limit, so it can miss older scrobbles within the last 30 days when listening volume exceeds 100 tracks.
|
||||
- Calling Last.fm duplicates data already persisted locally, adds network latency, consumes API quota/rate-limit budget, and introduces external failure modes to the dashboard.
|
||||
- This route is less reliable and less consistent than querying local `scrobbled_tracks`.
|
||||
|
||||
This route should be rejected.
|
||||
|
||||
## Recommended route
|
||||
|
||||
Route A is the simplest viable implementation that preserves correctness for the stats page timezone model. It avoids schema churn and external dependencies while using an existing indexed range scan and existing chart infrastructure. Route B is faster in bytes returned but trades away local-day correctness. Route C is operationally heavier than the current objective requires. Route D is incomplete or unnecessarily dependent on Last.fm.
|
||||
|
||||
## Open product choices before planning
|
||||
|
||||
- Confirm whether the 30-day window should include today plus the previous 29 local calendar days. This is the recommended interpretation.
|
||||
- Confirm chart ordering: oldest-to-newest reads naturally as a timeline, while newest-first keeps today's activity at the top. The recommended default is oldest-to-newest for a timeline chart.
|
||||
- Confirm whether the count should count scrobbled track rows. This matches the wording “scrobbled tracks per day” and the existing total scrobble counter semantics.
|
||||
@@ -0,0 +1,175 @@
|
||||
---
|
||||
id: ML-200
|
||||
title: scrobble count per day widget
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: "2026-05-31 18:02"
|
||||
updated_date: "2026-05-31 18:23"
|
||||
labels: []
|
||||
dependencies: []
|
||||
documentation:
|
||||
- doc-30 - ML-200-Research-Scrobble-daily-count-widget-routes.md
|
||||
ordinal: 33000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
In the stats page, right before Scrobble Activity, add a horizontal bar chart that shows the last 30 days of scrobble activity as the count of scrobbled tracks per day.
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [ ] #1 Stats page displays a new scrobble-count-per-day chart immediately before the existing Scrobble Activity section.
|
||||
- [ ] #2 The chart shows exactly 30 local calendar days: today plus the previous 29 days, ordered oldest to newest.
|
||||
- [ ] #3 Each day displays the count of scrobbled track rows for that local day, including zero-count days.
|
||||
- [ ] #4 Daily grouping respects the user's configured/browser timezone rather than UTC-only day boundaries.
|
||||
- [ ] #5 The chart refreshes when new scrobbles are imported and the existing listening-stats update notification is received.
|
||||
- [ ] #6 The implementation has context-level tests for daily count generation, zero-fill behavior, ordering, and timezone boundaries.
|
||||
- [ ] #7 The stats page has LiveView tests proving the chart is rendered in the correct location with expected labels/counts.
|
||||
- [ ] #8 All user-facing chart text is gettext-wrapped and gettext catalogs are updated.
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
# Implementation Plan — Route A (revised after plan review)
|
||||
|
||||
## Objective alignment
|
||||
|
||||
Add a stats dashboard widget immediately before the existing Scrobble Activity section that shows scrobbled track row counts for the last 30 local calendar days. The solution maps directly to the issue by adding a `MusicLibrary.ListeningStats` daily-count query for today plus the previous 29 days, rendering the fixed 30-day result as a horizontal bar chart on `MusicLibraryWeb.StatsLive.Index`, and refreshing the same data through the existing listening-stats PubSub update path.
|
||||
|
||||
## Chosen approach and alternatives considered
|
||||
|
||||
**Chosen approach: Route A — bounded context range scan + Elixir local-day grouping.** `MusicLibrary.ListeningStats` will own a new public function that accepts `timezone`, `days`, and testable `current_time` options. It will compute the exact local calendar window, query only `scrobbled_at_uts` values from `scrobbled_tracks` within that window, convert timestamps to the requested IANA timezone in Elixir, group by `Date`, fill missing days with zero counts, and return 30 oldest-to-newest entries.
|
||||
|
||||
The query must use both bounds for the local window:
|
||||
|
||||
- `scrobbled_at_uts >= start_of_first_local_day`
|
||||
- `scrobbled_at_uts < start_of_tomorrow_local_day`
|
||||
|
||||
This keeps the query bounded to exactly today plus the previous 29 local calendar days and prevents future/out-of-window rows from appearing.
|
||||
|
||||
This remains the simplest viable approach because it needs no migration, no cache, no background worker, no external API call, and no new infrastructure while preserving correct local-day semantics.
|
||||
|
||||
Alternatives evaluated:
|
||||
|
||||
- **SQL UTC aggregation:** deferred because SQLite can cheaply group by UTC day, but the widget must respect the stats page timezone model. UTC grouping would misclassify scrobbles around local midnight and DST boundaries.
|
||||
- **Materialized daily count table:** rejected for now because it adds schema, backfill, write-path maintenance, and timezone-bucket ambiguity for a small 30-day dashboard read.
|
||||
- **Reuse recent activity or call Last.fm:** rejected because recent activity is capped and can miss tracks in busy 30-day windows; Last.fm calls duplicate local data, add latency, and consume external API/rate-limit budget.
|
||||
|
||||
Research details are captured in `doc-30 - ML-200-Research-Scrobble-daily-count-widget-routes.md`.
|
||||
|
||||
## Architecture impact analysis
|
||||
|
||||
- **Schemas:** no schema changes. `LastFm.Track` remains unchanged; counts are derived from existing `scrobbled_tracks.scrobbled_at_uts` rows.
|
||||
- **Contexts:** add a public analytics function to `MusicLibrary.ListeningStats`. The LiveView must not query the database directly.
|
||||
- **Database/indexes:** use the existing unique index on `[:scrobbled_at_uts, :title]` for an indexed bounded range scan. No migration is expected unless `EXPLAIN QUERY PLAN` shows the final bounded range query is not using that index.
|
||||
- **LiveView/UI:** update `MusicLibraryWeb.StatsLive.Index` to assign/render the new chart immediately before `.scrobble_activity`. Add a stable wrapper id such as `id="daily-scrobble-counts"` so tests can verify ordering relative to the existing `id="scrobble-activity"` section.
|
||||
- **Components:** prefer reusing `MusicLibraryWeb.ChartComponents.horizontal_bar_chart/1`; add only a small stats-specific wrapper if needed for labels, empty state, or accessible text.
|
||||
- **PubSub:** reuse existing `ListeningStats.subscribe/0` topic and `%{track_count: n}` messages. Update the existing refresh helper so daily counts refresh with the total count and recent activity on nonzero updates.
|
||||
- **Routes:** no route changes.
|
||||
- **External APIs:** no Last.fm/API calls for this widget; data comes from local persistence.
|
||||
- **Supervision tree/workers:** no supervision or Oban worker changes.
|
||||
- **Gettext:** new visible strings must be wrapped and catalogs updated.
|
||||
|
||||
## Performance profile
|
||||
|
||||
- **Runtime complexity:** O(n + d), where `n` is scrobbles in the exact 30-day local window and `d = 30` fixed output days.
|
||||
- **Database query pattern:** one indexed bounded range scan selecting only `scrobbled_at_uts`, ordered ascending. No joins and no per-day query loop, so no N+1 risk.
|
||||
- **Expected SQL shape:** `SELECT scrobbled_at_uts FROM scrobbled_tracks WHERE scrobbled_at_uts >= ? AND scrobbled_at_uts < ? ORDER BY scrobbled_at_uts ASC`.
|
||||
- **Memory footprint:** one integer per scrobble in the recent window plus a 30-entry result list/map. Current local data observed during research had about 1,002 scrobbles in the last 30 days.
|
||||
- **Latency/throughput:** expected dashboard impact is small for a personal collection app. The query runs on page mount and when new scrobbles are imported via existing PubSub notifications.
|
||||
- **Failure mode:** if the query fails unexpectedly, it should fail like existing synchronous stats assigns unless implemented as async; do not introduce external-service failure paths.
|
||||
|
||||
## Benchmarking requirements
|
||||
|
||||
No ongoing benchmark is required before implementation because the query is a bounded 30-day indexed range scan and current observed row volume is modest.
|
||||
|
||||
One-off validation is required during implementation:
|
||||
|
||||
1. Run `EXPLAIN QUERY PLAN` for the final bounded SQL shape and confirm SQLite uses `scrobbled_tracks_scrobbled_at_uts_title_index` or an equivalent covering/range index for `scrobbled_at_uts >= ? AND scrobbled_at_uts < ?`.
|
||||
2. Use development data or seeded test data to confirm the number of fetched rows is limited to the exact local 30-day window.
|
||||
3. If local or production-like data shows the query is slow enough to affect dashboard rendering, stop and revisit Route C rather than adding an ad hoc cache.
|
||||
|
||||
Acceptable threshold for this task: one dashboard read should perform a single indexed bounded range query for daily counts and should not introduce any query loop proportional to 30 days.
|
||||
|
||||
## Cost profile
|
||||
|
||||
No paid resources are consumed. The implementation uses local SQLite data and existing LiveView rendering. It does not call Last.fm or any other external API, does not increase storage, and does not require additional compute services.
|
||||
|
||||
## Production Changes
|
||||
|
||||
No manual production changes are expected.
|
||||
|
||||
- **Environment variables:** none.
|
||||
- **Service provisioning:** none.
|
||||
- **Database migrations:** none expected.
|
||||
- **DNS/firewall:** none.
|
||||
- **Rollout:** deploy the code normally through the existing release pipeline.
|
||||
- **Rollback:** revert the application code if the widget causes issues; no data rollback is required because no schema or persistent data changes are planned.
|
||||
|
||||
## Documentation updates
|
||||
|
||||
- `docs/architecture.md`: update only if the implementation adds a new named component/module or materially changes the stats dashboard/context map. A small function in `ListeningStats` and markup in `StatsLive.Index` may not require an architecture update unless a new component module is introduced.
|
||||
- `.agents/skills/*`: no updates expected.
|
||||
- README/API docs: no updates expected; this is an internal dashboard UI feature.
|
||||
- Gettext catalogs: update `priv/gettext/default.pot` and locale files after adding user-facing strings.
|
||||
|
||||
## Sequential implementation steps with verification
|
||||
|
||||
1. **Add the context function in `MusicLibrary.ListeningStats`.**
|
||||
- Implement a public function such as `daily_scrobble_counts/1` or `scrobble_counts_by_day/1` with options for `timezone`, `days: 30`, and `current_time`.
|
||||
- Prefer returning a fixed list of maps such as `%{date: Date.t(), count: non_neg_integer()}` ordered oldest to newest.
|
||||
- Compute calendar boundaries by first shifting `current_time` into the supplied timezone, deriving `today` from that local time, then using `Date.add/2` for calendar-day math. Do not subtract fixed 24-hour second intervals to derive local dates because DST boundaries matter.
|
||||
- Compute `first_date = Date.add(today, -(days - 1))` and `tomorrow = Date.add(today, 1)`.
|
||||
- Convert local midnights for `first_date` and `tomorrow` to Unix timestamps in the supplied timezone; use those as the lower and upper SQL bounds.
|
||||
- Query only `scrobbled_at_uts` values where `scrobbled_at_uts >= start_uts and scrobbled_at_uts < end_uts`, ordered ascending.
|
||||
- Convert each returned timestamp to the supplied timezone, group by local `Date`, count rows, fill all 30 dates with zeroes, and return oldest-to-newest entries.
|
||||
- Verification before moving on: add/run focused context tests for exact 30-day length, oldest-to-newest order, zero-filled missing days, counting rows rather than distinct timestamps, exclusion before the window, exclusion at/after tomorrow, inclusion on today, and timezone boundary behavior.
|
||||
|
||||
2. **Validate the SQL query shape.**
|
||||
- Inspect the generated SQL or equivalent query and run `EXPLAIN QUERY PLAN` against SQLite.
|
||||
- Verification before moving on: confirm the plan uses the existing `scrobbled_at_uts` range/covering index and does not perform a full table scan or 30 separate queries.
|
||||
|
||||
3. **Render the widget on `StatsLive.Index`.**
|
||||
- Assign the daily-count data during mount using the user timezone from the socket.
|
||||
- Add a section immediately before the existing `.scrobble_activity` call.
|
||||
- Wrap the section with a stable id such as `daily-scrobble-counts` so tests can assert it appears before `scrobble-activity`.
|
||||
- Reuse `ChartComponents.horizontal_bar_chart/1` where practical, with labels suitable for local dates and counts as values. If a small wrapper is needed, keep it in the stats/chart component layer and follow project UI conventions: gettext strings, paired dark-mode classes, CSS Grid charting, and accessible text/labels.
|
||||
- Verification before moving on: run the focused stats LiveView test and inspect rendered HTML ordering to confirm the daily chart section precedes Scrobble Activity.
|
||||
|
||||
4. **Refresh the widget when scrobbles update.**
|
||||
- Extend the existing `assign_scrobble_activity/1` refresh path or equivalent so `%{track_count: n}` PubSub updates refresh total scrobbles, recent activity, and daily counts together.
|
||||
- Preserve the existing `%{track_count: 0}` no-op behavior unless there is a product reason to reload on zero inserted tracks.
|
||||
- Verification before moving on: add/run a connected LiveView test that inserts or broadcasts a nonzero listening-stats update and proves the chart data/counts change after the update is received.
|
||||
|
||||
5. **Update gettext catalogs.**
|
||||
- Wrap new headings, empty-state text, accessible labels, and any date/count labels requiring text in `gettext/1` or `ngettext/3` as appropriate.
|
||||
- Run gettext extraction/merge.
|
||||
- Verification before moving on: run the gettext up-to-date check or the project task/check that covers it.
|
||||
|
||||
6. **Run focused and relevant verification.**
|
||||
- Run `mix test test/music_library/listening_stats_test.exs`.
|
||||
- Run `mix test test/music_library_web/live/stats_live/index_test.exs`.
|
||||
- Run formatting and relevant project verification tasks identified in `docs/available-tasks.md`, such as `mise run dev:lint` or the conditional precommit task, before finalizing implementation.
|
||||
- Verification before moving on: all focused tests pass, formatting is clean, and no gettext drift remains.
|
||||
|
||||
7. **Visual verification.**
|
||||
- With the dev server running, open the stats page and capture screenshots after the chart is visible.
|
||||
- Verify both light and dark mode if custom colors/classes are added or changed.
|
||||
- Verification before completion: chart appears immediately before Scrobble Activity, has 30 bars/rows, remains readable at common viewport widths, excludes future/out-of-window rows, and dark-mode colors are paired/correct.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
Updated plan and research doc after the existing chart component was renamed from `vertical_bar_chart/1` to `horizontal_bar_chart/1`, so implementers should use the corrected component name.
|
||||
|
||||
<!-- SECTION:NOTES:END -->
|
||||
Reference in New Issue
Block a user