From f347f562182acb86e41e2221f332387d65b9ede8 Mon Sep 17 00:00:00 2001 From: Claudio Ortolina Date: Mon, 25 May 2026 09:31:43 +0300 Subject: [PATCH] ML-169.8: reduce Stats mount sync queries to 3 Move 7 non-critical queries out of synchronous mount into assign_async and start_async tasks with skeleton loading placeholders. Only scalar badge counters (collection count, wishlist count, scrobble count) remain synchronous. All async sections use <.async_result> with loading and failed slots. Scrobble activity preserves LiveView streams via handle_async. On-this-day date changes use AsyncResult.ok/2. --- ...ive.Index-10-sync-queries-blocking-TTFB.md | 84 +++++- lib/music_library/collection.ex | 7 + .../live/stats_live/index.ex | 278 ++++++++++++------ priv/gettext/default.pot | 10 + priv/gettext/en/LC_MESSAGES/default.po | 10 + .../live/stats_live/index_test.exs | 16 +- 6 files changed, 303 insertions(+), 102 deletions(-) diff --git a/backlog/tasks/ml-169.8 - Fix-StatsLive.Index-10-sync-queries-blocking-TTFB.md b/backlog/tasks/ml-169.8 - Fix-StatsLive.Index-10-sync-queries-blocking-TTFB.md index eb9cf1e7..c297e38b 100644 --- a/backlog/tasks/ml-169.8 - Fix-StatsLive.Index-10-sync-queries-blocking-TTFB.md +++ b/backlog/tasks/ml-169.8 - Fix-StatsLive.Index-10-sync-queries-blocking-TTFB.md @@ -1,10 +1,11 @@ --- id: ML-169.8 title: "Fix: StatsLive.Index 10 sync queries blocking TTFB" -status: To Do -assignee: [] +status: Done +assignee: + - pi created_date: "2026-05-19 11:48" -updated_date: "2026-05-20 06:54" +updated_date: "2026-05-25 06:30" labels: - perf - fix @@ -79,13 +80,13 @@ Show skeleton/loading placeholders for async sections and graceful failed states -- [ ] #1 StatsLive.Index initial mount runs only true scalar synchronous queries needed for immediately visible counters and returns under 200ms on the audited environment -- [ ] #2 Counter badges for collection, wishlist, and scrobbles render immediately from scalar counts -- [ ] #3 Latest purchase, format/type stats, collection charts, on-this-day records, and scrobble activity load asynchronously with skeleton/loading placeholders -- [ ] #4 Scrobble activity async loading preserves LiveView streams for recent tracks and recent albums rather than converting stream-backed UI to plain list assigns -- [ ] #5 Async sections render graceful failed or nil states and do not flash empty data containers -- [ ] #6 Tests cover immediate counter rendering and use render_async() for async-loaded Stats page content -- [ ] #7 A follow-up query trace or equivalent query-budget check confirms the synchronous Stats mount query count is reduced from the original 10-query block +- [x] #1 StatsLive.Index initial mount runs only true scalar synchronous queries needed for immediately visible counters and returns under 200ms on the audited environment +- [x] #2 Counter badges for collection, wishlist, and scrobbles render immediately from scalar counts +- [x] #3 Latest purchase, format/type stats, collection charts, on-this-day records, and scrobble activity load asynchronously with skeleton/loading placeholders +- [x] #4 Scrobble activity async loading preserves LiveView streams for recent tracks and recent albums rather than converting stream-backed UI to plain list assigns +- [x] #5 Async sections render graceful failed or nil states and do not flash empty data containers +- [x] #6 Tests cover immediate counter rendering and use render_async() for async-loaded Stats page content +- [x] #7 A follow-up query trace or equivalent query-budget check confirms the synchronous Stats mount query count is reduced from the original 10-query block ## Implementation Plan @@ -106,4 +107,67 @@ Show skeleton/loading placeholders for async sections and graceful failed states Review correction from `/tmp/queries.sql`: the trace supports the 10 synchronous Stats queries claim, but not the statement that `recent_activity` was the slowest query in that run. It also shows `collection_count` currently depends on grouped format stats, so the task needs a direct scalar collection count before moving format/type grouped stats async. +Implemented async loading for StatsLive.Index home page: + +- Added Collection.count/0 — direct scalar COUNT for the collection badge (avoids deriving from grouped format counts) + +- Refactored mount/3: only 3 scalar counters stay synchronous (collection_count, wishlist_count, scrobble_count) down from 10 queries + +- Moved latest_record, collection_summary (format/type/charts), and on_this_day_records to assign_async with skeleton loading states via <.async_result> + +- Moved scrobble activity to start_async + handle_async, preserving @streams.recent_tracks and @streams.recent_albums contract + +- Added handle_async/3 with three clauses: {:ok, {:ok, data}}, {:ok, {:error, reason}}, {:exit, reason} + +- Added skeleton/loading and failed states for all async sections + +- Separated on_this_day_records into its own async key to support interactive date changes (uses AsyncResult.ok/2 in event handler) + +- Updated all 11 StatsLive tests: added render_async() before assertions on async content; wishlist counter test verifies immediate rendering + +- Full test suite: 1135 tests pass, no regressions + +## Final Summary + + + +## Summary + +Reduced synchronous StatsLive.Index mount queries from 10 to 3 by moving non-critical data loading to async tasks with skeleton/loading placeholders. + +### What changed + +**`lib/music_library/collection.ex`** — Added `Collection.count/0`, a direct scalar `COUNT(*) WHERE purchased_at IS NOT NULL` for the collection badge, replacing the previous `Enum.sum_by(count_records_by_format())` derivation. + +**`lib/music_library_web/live/stats_live/index.ex`** — Major refactor: + +- **mount/3**: Only 3 sync queries remain — `Collection.count()`, `Wishlist.count()`, `ListeningStats.scrobble_count()`. All other data loads via `assign_async` (latest_record, collection_summary, on_this_day_records) and `start_async` (scrobble_activity). +- **render/1**: Each async section wrapped in `<.async_result>` with `:loading` (skeleton placeholders with animate-pulse) and `:failed` (graceful error states) slots. +- **handle_async/3**: Three clauses for scrobble activity — success (streams tracks/albums, sets last_updated_uts), function error (logs), task exit (logs). +- **on_this_day_records** separated into its own async key to support interactive date changes via `AsyncResult.ok/2` in the event handler. +- Scrobble activity preserves `@streams.recent_tracks` and `@streams.recent_albums` contract — streams are seeded with empty lists on mount and populated via `handle_async`. +- Removed dead helpers: `assign_counts/1`, `record_stats/1` (inlined into template). + +**`test/music_library_web/live/stats_live/index_test.exs`** — Added `render_async()` after `visit("/")` for all tests asserting on async-loaded content (latest purchase, format/type stats, on-this-day sections). Counter badge tests verify rendering before async completion. + +### Sync query budget + +| Before | After | +| ------------------------------ | ----------------------------------------------------------------- | +| 10 sync queries blocking mount | 3 sync queries (collection_count, wishlist_count, scrobble_count) | + +The 7 non-critical queries now run in background tasks after the initial render. + +### Tests + +- StatsLive.Index: 11/11 pass +- Full test suite: 1135/1135 pass (both partitions) +- Credo: no issues +- Mix format: compliant + +### Risks / Follow-ups + +- If the async tasks fail silently (logged but not user-visible), users see skeleton states replaced by empty "failed" slots. The failed slots are minimal (error icon for latest record, text for on-this-day, empty for format/type/charts). Consider improving failed-state UI messaging in a future iteration. +- The `handle_info` PubSub path still calls `assign_scrobble_activity/1` synchronously for live updates — this is acceptable since it fires on events rather than page load. + diff --git a/lib/music_library/collection.ex b/lib/music_library/collection.ex index aca1c221..74dfa861 100644 --- a/lib/music_library/collection.ex +++ b/lib/music_library/collection.ex @@ -27,6 +27,13 @@ defmodule MusicLibrary.Collection do Records.search_records_count(base_search(), query) end + @doc "Direct scalar count for collection badge — faster than summing grouped format counts." + @spec count() :: non_neg_integer() + def count do + from(r in Record, where: not is_nil(r.purchased_at)) + |> Repo.aggregate(:count) + end + @spec count_records_by_format() :: [{String.t(), non_neg_integer()}] def count_records_by_format do q = diff --git a/lib/music_library_web/live/stats_live/index.ex b/lib/music_library_web/live/stats_live/index.ex index a6d5152f..d4c1d075 100644 --- a/lib/music_library_web/live/stats_live/index.ex +++ b/lib/music_library_web/live/stats_live/index.ex @@ -11,6 +11,7 @@ defmodule MusicLibraryWeb.StatsLive.Index do alias MusicLibrary.{Collection, ListeningStats, Records, Wishlist} alias MusicLibraryWeb.ErrorMessages alias MusicLibraryWeb.StatsLive.{TopAlbums, TopArtists} + alias Phoenix.LiveView.AsyncResult @impl true def render(assigns) do @@ -21,32 +22,126 @@ defmodule MusicLibraryWeb.StatsLive.Index do socket={@socket} toasts_sync={assigns[:toasts_sync]} > - <.record_stats - latest_record={@latest_record} - collection_count={@collection_count} - wishlist_count={@wishlist_count} - scrobble_count={@scrobble_count} - /> -
0} class="grid gap-x-5 md:grid-cols-2"> - <.formats_stats collection_count_by_format={@collection_count_by_format} /> - <.types_stats collection_count_by_type={@collection_count_by_type} /> -
+ <.section> + <:title>{gettext("Records")} +
+ <.async_result :let={latest_record} assign={@latest_record}> + <:loading> +
+
+
+
+
+
+ + <:failed :let={_reason}> +
+ <.icon + name="hero-exclamation-triangle" + class="size-5 text-zinc-400 dark:text-zinc-500" + /> +
+ + <.album_preview + record={latest_record} + title={gettext("Latest purchase")} + class="col-span-3 sm:col-span-2" + /> + + <.counter + title={gettext("Collection")} + count={@collection_count} + path={~p"/collection"} + /> + <.counter title={gettext("Wishlist")} count={@wishlist_count} path={~p"/wishlist"} /> + <.counter + title={gettext("Scrobbles")} + count={to_compact(@scrobble_count)} + tooltip={@scrobble_count} + path={~p"/scrobbled-tracks"} + /> +
+ +
- <.on_this_day current_date={@current_date} records_on_this_day={@records_on_this_day} /> + <.async_result :let={records_on_this_day} assign={@on_this_day_records}> + <:loading> + <.section container_class="order-first lg:order-last"> + <:title>{gettext("On This day")} +
+
+
+
+ + + <:failed :let={_reason}> + <.section container_class="order-first lg:order-last"> + <:title>{gettext("On This day")} +
+ {gettext("Could not load records on this day.")} +
+ + + <.on_this_day + current_date={@current_date} + records_on_this_day={records_on_this_day} + /> +
+ <.async_result :let={summary} assign={@collection_summary}> + <:loading> +
+ <.section> + <:title>{gettext("Formats")} +
+
+
+ + <.section> + <:title>{gettext("Types")} +
+
+
+ +
+ + <:failed :let={_reason}> +
0} class="grid gap-x-5 md:grid-cols-2"> + <.formats_stats collection_count_by_format={summary.collection_count_by_format} /> + <.types_stats collection_count_by_type={summary.collection_count_by_type} /> +
+ + <.scrobble_activity scrobble_activity_mode={@scrobble_activity_mode} streams={@streams} /> -
0} class="grid grid-cols-1 gap-x-5 md:grid-cols-2 lg:grid-cols-3"> - <.top_collection_artists records_by_artist={@records_by_artist} /> - <.top_collection_genres records_by_genre={@records_by_genre} /> - <.top_release_years records_by_release_year={@records_by_release_year} /> -
+ <.async_result :let={summary} assign={@collection_summary}> + <:loading> +
+ <.section> + <:title>{gettext("Loading…")} +
+
+
+
+ +
+ + <:failed :let={_reason}> +
0} + class="grid grid-cols-1 gap-x-5 md:grid-cols-2 lg:grid-cols-3" + > + <.top_collection_artists records_by_artist={summary.records_by_artist} /> + <.top_collection_genres records_by_genre={summary.records_by_genre} /> + <.top_release_years records_by_release_year={summary.records_by_release_year} /> +
+ <.structured_modal :if={@rule_picker_album_title} @@ -66,15 +161,7 @@ defmodule MusicLibraryWeb.StatsLive.Index do @impl true def mount(_params, _session, socket) do current_date = DateTime.now!(socket.assigns.timezone) |> DateTime.to_date() - latest_record = Collection.get_latest_record() - records_by_artists = Collection.count_records_by_artist(limit: 20) - records_by_genre = Collection.count_records_by_genre(limit: 20) - records_by_release_year = Collection.count_records_by_release_year(limit: 20) - - records_on_this_day = - current_date - |> Collection.get_records_on_this_day() - |> Collection.group_records_by_release_group() + timezone = socket.assigns.timezone if connected?(socket) do ListeningStats.subscribe() @@ -88,20 +175,84 @@ defmodule MusicLibraryWeb.StatsLive.Index do |> stream_configure(:recent_albums, dom_id: fn %{album: album} -> "album-#{album.scrobbled_at_uts}" end ) - |> assign_counts() - |> assign_scrobble_activity() + |> stream(:recent_tracks, []) + |> stream(:recent_albums, []) |> assign( current_date: current_date, + collection_count: Collection.count(), + wishlist_count: Wishlist.count(), + scrobble_count: ListeningStats.scrobble_count(), + last_updated_uts: nil, scrobble_activity_mode: "albums", - latest_record: latest_record, page_title: gettext("Stats"), current_section: :stats, - rule_picker_album_title: nil, - records_by_artist: records_by_artists, - records_by_genre: records_by_genre, - records_by_release_year: records_by_release_year, - records_on_this_day: records_on_this_day - )} + rule_picker_album_title: nil + ) + |> assign_async(:latest_record, fn -> + {:ok, %{latest_record: Collection.get_latest_record()}} + end) + |> assign_async(:collection_summary, fn -> + {:ok, + %{ + collection_summary: %{ + collection_count_by_format: Collection.count_records_by_format(), + collection_count_by_type: Collection.count_records_by_type(), + records_by_artist: Collection.count_records_by_artist(limit: 20), + records_by_genre: Collection.count_records_by_genre(limit: 20), + records_by_release_year: Collection.count_records_by_release_year(limit: 20) + } + }} + end) + |> assign_async(:on_this_day_records, fn -> + {:ok, + %{ + on_this_day_records: + current_date + |> Collection.get_records_on_this_day() + |> Collection.group_records_by_release_group() + }} + end) + |> start_async(:scrobble_activity, fn -> + %{ + recent_tracks: recent_tracks, + recent_albums: recent_albums + } = ListeningStats.recent_activity(timezone) + + last_updated_uts = + if rt = List.first(recent_tracks), do: rt.track.scrobbled_at_uts + + {:ok, + %{ + recent_tracks: recent_tracks, + recent_albums: recent_albums, + last_updated_uts: last_updated_uts + }} + end)} + end + + @impl true + def handle_async(:scrobble_activity, {:ok, {:ok, result}}, socket) do + %{recent_tracks: tracks, recent_albums: albums, last_updated_uts: uts} = result + + {:noreply, + socket + |> assign(:last_updated_uts, uts) + |> stream(:recent_tracks, tracks, reset: true) + |> stream(:recent_albums, albums, reset: true)} + end + + def handle_async(:scrobble_activity, {:ok, {:error, reason}}, socket) do + require Logger + Logger.error("Failed to load scrobble activity: #{inspect(reason)}") + + {:noreply, socket} + end + + def handle_async(:scrobble_activity, {:exit, reason}, socket) do + require Logger + Logger.error("Scrobble activity task exited: #{inspect(reason)}") + + {:noreply, socket} end @impl true @@ -148,7 +299,14 @@ defmodule MusicLibraryWeb.StatsLive.Index do {:noreply, socket - |> assign(%{current_date: date, records_on_this_day: records_on_this_day})} + |> assign(:current_date, date) + |> assign( + :on_this_day_records, + AsyncResult.ok( + socket.assigns.on_this_day_records, + records_on_this_day + ) + )} {:error, _reason} -> {:noreply, socket} @@ -513,56 +671,6 @@ defmodule MusicLibraryWeb.StatsLive.Index do """ end - attr :latest_record, Records.Record, required: false - attr :collection_count, :integer, required: true - attr :wishlist_count, :integer, required: true - attr :scrobble_count, :integer, required: true - - defp record_stats(assigns) do - ~H""" - <.section> - <:title>{gettext("Records")} -
- <.album_preview - record={@latest_record} - title={gettext("Latest purchase")} - class="col-span-3 sm:col-span-2" - /> - <.counter - title={gettext("Collection")} - count={@collection_count} - path={~p"/collection"} - /> - <.counter title={gettext("Wishlist")} count={@wishlist_count} path={~p"/wishlist"} /> - <.counter - title={gettext("Scrobbles")} - count={to_compact(@scrobble_count)} - tooltip={@scrobble_count} - path={~p"/scrobbled-tracks"} - /> -
- - """ - end - - defp assign_counts(socket) do - collection_count_by_format = Collection.count_records_by_format() - - collection_count_by_type = Collection.count_records_by_type() - - collection_count = - Enum.sum_by(collection_count_by_format, fn {_, count} -> count end) - - wishlist_count = Wishlist.count() - - assign(socket, - collection_count_by_format: collection_count_by_format, - collection_count_by_type: collection_count_by_type, - collection_count: collection_count, - wishlist_count: wishlist_count - ) - end - defp assign_scrobble_activity(socket) do %{ recent_tracks: recent_tracks, diff --git a/priv/gettext/default.pot b/priv/gettext/default.pot index 8383bc31..ad46ccb5 100644 --- a/priv/gettext/default.pot +++ b/priv/gettext/default.pot @@ -2591,3 +2591,13 @@ msgstr "" #, elixir-autogen, elixir-format msgid "Settings" msgstr "" + +#: lib/music_library_web/live/stats_live/index.ex +#, elixir-autogen, elixir-format +msgid "Could not load records on this day." +msgstr "" + +#: lib/music_library_web/live/stats_live/index.ex +#, elixir-autogen, elixir-format +msgid "Loading…" +msgstr "" diff --git a/priv/gettext/en/LC_MESSAGES/default.po b/priv/gettext/en/LC_MESSAGES/default.po index f3cb960d..e94912cc 100644 --- a/priv/gettext/en/LC_MESSAGES/default.po +++ b/priv/gettext/en/LC_MESSAGES/default.po @@ -2591,3 +2591,13 @@ msgstr "" #, elixir-autogen, elixir-format msgid "Settings" msgstr "" + +#: lib/music_library_web/live/stats_live/index.ex +#, elixir-autogen, elixir-format, fuzzy +msgid "Could not load records on this day." +msgstr "" + +#: lib/music_library_web/live/stats_live/index.ex +#, elixir-autogen, elixir-format +msgid "Loading…" +msgstr "" diff --git a/test/music_library_web/live/stats_live/index_test.exs b/test/music_library_web/live/stats_live/index_test.exs index 50a7fb44..019515aa 100644 --- a/test/music_library_web/live/stats_live/index_test.exs +++ b/test/music_library_web/live/stats_live/index_test.exs @@ -36,6 +36,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do conn |> visit("/") |> assert_has("dd", collection |> length() |> Integer.to_string()) + |> render_async() collection |> Enum.frequencies_by(& &1.format) @@ -58,6 +59,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do session = conn |> visit("/") + |> render_async() |> assert_has("span", escape(latest_record.title)) for artist <- latest_record.artists do @@ -94,7 +96,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() assert_has(session, "h1", "On This day") @@ -126,7 +128,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() assert_has(session, "##{record_today.id} h2", escape(record_today.title)) @@ -179,7 +181,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do {:ok, _dup} = MusicLibrary.Records.create_record(dup_attrs) - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() # The grouped record should appear with the group ID assert_has(session, "#group-#{base_record.musicbrainz_id}") @@ -201,7 +203,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() assert_has(session, "span", "Today") end @@ -222,7 +224,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() assert_has(session, "span", "5 years ago") end @@ -243,7 +245,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() assert_has(session, "span", "10 years ago") end @@ -264,7 +266,7 @@ defmodule MusicLibraryWeb.StatsLive.IndexTest do }) |> Repo.update!() - session = conn |> visit("/") + session = conn |> visit("/") |> render_async() # 3 years is not a milestone (not divisible by 5 or 10) assert_has(session, "span", "3 years ago")