cloud/music_library
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
19e097e514242e6b65064e704b2359eba9619e86
music_library/backlog/tasks/ml-156 - Explore-alternatives-to-reduce-token-usage-when-providing-collection-context-to-LLM-for-collection-chat.md
T
Claudio Ortolina 69245d8d15 ML-156: extend and improve plan
2026-05-22 23:46:04 +01:00

23 KiB
Raw Blame History

id, title, status, assignee, created_date, updated_date, labels, dependencies, references, documentation, priority
id title status assignee created_date updated_date labels dependencies references documentation priority
ML-156 Explore alternatives to reduce token usage when providing collection context to LLM for collection chat To Do
2026-05-02 16:02 2026-05-22 22:40
ready
backlog://document/doc-1
lib/music_library/chats/collection_chat.ex
lib/music_library/collection.ex
lib/music_library/chats/prompt.ex
lib/music_library/chats/stream_provider.ex
lib/open_ai/api.ex
lib/music_library_web/components/chat.ex
lib/music_library_web/live/collection_live/index.ex
docs/architecture.md
docs/production-infrastructure.md
high

Description

Currently, every new collection chat sends the ENTIRE collection catalog (all records formatted as "Artist - Title (year, format) [genres]") plus aggregated stats as the instructions parameter to the OpenAI Responses API. For a collection of 500+ records, this burns ~9,000+ input tokens on EVERY new chat start — regardless of what the user asks.

The goal of this task is to analyze alternatives, pick the best one, and implement it. The selected approach should:

  • Significantly reduce per-chat token usage
  • Preserve or improve the quality of LLM responses about the collection
  • Not require architectural overhauls beyond the chat/streaming layer

The Collection.collection_summary/0 function loads ALL records from the DB, formats them, and returns {summary, count}. This is computed asynchronously in CollectionLive.Index.mount/3 and passed to the Chat component as chat_context. CollectionChat.build_instructions/2 then embeds the full summary into the instructions string sent to OpenAI.

Acceptance Criteria

  • #1 CollectionChat.build_instructions/2 no longer interpolates the full collection catalog into the instructions sent to OpenAI — only aggregated stats and a record count are included
  • #2 A file_search tool with the collection's vector store ID is included in every collection chat request to the OpenAI Responses API
  • #3 A CollectionChat.FileStore module manages the collection file lifecycle: upload to OpenAI Files API, vector store creation, and file-to-store attachment, persisting IDs via Secrets
  • #4 When a record is added, edited, or deleted, the collection file at OpenAI is refreshed (async, non-blocking) so the LLM always searches up-to-date data
  • #5 If the file/vector store is unavailable (upload failed, first deploy), the chat falls back to stats-only instructions without errors
  • #6 Existing record and artist chats continue to work without changes (no regression in streaming)
  • #7 Tests cover: Files API endpoints, file upload/create/refresh lifecycle, file_search tool inclusion in chat requests, fallback when file store is unavailable, empty collection edge case
  • #8 Per-chat input tokens for the collection chat instructions are O(1) relative to collection size (the file is searched by OpenAI on demand, not embedded in instructions)
  • #9 Refresh scheduling uses a true debounce/coalescing strategy so rapid/bulk mutations and mutations during an executing refresh cannot leave the active file stale
  • #10 FileStore only switches the active vector store after vector-store-file indexing succeeds; failed refreshes keep the previous active store available
  • #11 Production documentation includes OpenAI file_search/vector-store cost profile, initial refresh behavior, cleanup/rollback notes, and whether new environment variables are required

Implementation Plan

Approach: OpenAI file_search tool + Oban-managed collection catalog lifecycle

Upload the collection catalog to OpenAI, index it in a vector store, and include the Responses API file_search tool in collection chat requests. Keep only aggregated stats and record count in the prompt instructions. This makes per-chat instructions O(1) relative to collection size while preserving the ability to answer record-specific questions through retrieval.

Research and alternative analysis: see ML-156 Research document.

Why this approach

  • Stats-only is much simpler, but it removes record-specific collection awareness.
  • Custom function calling would avoid OpenAI file storage, but requires a streaming tool-call orchestration loop and changes shared chat streaming code for all chat types.
  • Local RAG with SQLite FTS/sqlite-vec avoids external vector-store lifecycle, but still requires retrieval orchestration before/inside every chat request and would either be mostly lexical or depend on the existing embedding pipeline being complete and fresh. It remains a good future option if OpenAI file_search quality/cost is poor.
  • OpenAI file_search gives semantic retrieval with minimal streaming changes: file-search events are response.* events and can be ignored by the existing catch-all while output text streams normally.

Performance and cost profile

  • Per chat: instructions contain stats + count only (~100-300 tokens), not the full catalog (~9,000+ tokens for ~500 records). File-search result tokens are paid only when the model searches.
  • Per refresh: catalog generation, upload, and indexing are O(collection size). This is moved out of LiveView/chat paths and into debounced Oban jobs.
  • Page load: CollectionLive.Index loads a stats-only context, not the full catalog. Measure stats_summary/0 once on realistic data; if it is noticeably slow, switch it from in-memory grouping to SQL aggregate queries.
  • Hash skip: store a SHA-256 hash of the uploaded catalog. Cron/manual refreshes skip upload/index work when the catalog content is unchanged.
  • Benchmarks: no recurring benchmark is required. Do one implementation-time measurement comparing before/after instruction byte/token estimates for representative collection sizes (current data, 1,000 records if easy to synthesize) and record catalog refresh runtime.
  • Paid resources: document current OpenAI pricing at implementation time in docs/production-infrastructure.md because prices change. Expected profile for current scale is negligible storage for a small text file/vector store plus occasional indexing after debounced mutations, and per-chat retrieval/tool charges only when file_search runs.

Architecture decisions

  1. Chat sessions never upload or refresh files. They only read the active vector store ID from FileStore.get_vector_store_id/0. Missing/unavailable store falls back to stats-only instructions without surfacing an error to the user.
  2. Use a new vector store per successful refresh. Build and index the new file/store off to the side, then switch the active secret only after vector-store-file indexing completes. Failed refreshes leave the previous active store untouched.
  3. Persist FileStore state as one JSON secret. Store %{file_id, vector_store_id, catalog_hash, updated_at} in a single encrypted Secrets entry to avoid partial multi-secret updates. get_vector_store_id/0 only returns {:ok, id} when the state contains a usable active file and vector store.
  4. True debounce/coalescing. Record mutations schedule a refresh for 5 minutes in the future. Additional mutations while a refresh is scheduled replace scheduled_at, pushing the refresh out. Mutations that happen while a worker is already executing can schedule a follow-up refresh, so no changes are dropped.
  5. Refresh triggers cover all catalog-affecting mutations. CRUD paths enqueue refreshes, and direct enrichment Repo.update paths that affect catalog content (notably genres and any title/artist/release-date/format/purchased-state changes) either go through Records.update_record/2 or explicitly enqueue on success.
  6. OpenAI API work uses an :open_ai Oban queue. Add a serialized queue (open_ai: 1) for OpenAI-calling workers, in addition to the existing Req rate limiter.
  7. Cleanup is scoped and safe. Only delete OpenAI files/vector stores with the ML-managed name/filename prefix, and never delete unrelated OpenAI resources.

Phase 0: Baseline measurement and API documentation check

Before implementing:

  1. Confirm current OpenAI Responses API file_search, Files, and Vector Stores endpoint shapes.
  2. Confirm the status values for vector-store file indexing and poll the vector-store-file endpoint, not the generic file metadata endpoint.
  3. Record a one-off baseline:
    • current collection chat instruction byte length / estimated tokens;
    • expected stats-only instruction byte length / estimated tokens;
    • current Collection.collection_summary/0 runtime on realistic data.

Do not add a permanent benchmark unless this measurement shows refresh/page-load costs are high enough to justify ongoing tracking.

Phase 1: OpenAI API extensions

Add public functions to OpenAI.API following the existing new_request/1 pattern, Req rate limiting, and OpenAI.API.ErrorResponse.from_response/1 error handling:

  • upload_file(file_content, filename, config) — POST /v1/files with multipart body (purpose: "assistants", file text content).
  • create_vector_store(name, config) — POST /v1/vector_stores with an ML-managed name such as music-library-collection-chat-<timestamp>-<hash>.
  • add_file_to_vector_store(vector_store_id, file_id, config) — POST /v1/vector_stores/{vector_store_id}/files.
  • get_vector_store_file(vector_store_id, file_id, config) — GET /v1/vector_stores/{vector_store_id}/files/{file_id}. Use this for indexing status polling.
  • delete_vector_store(vector_store_id, config) — DELETE /v1/vector_stores/{vector_store_id} for cleanup/rollback.
  • delete_file(file_id, config) — DELETE /v1/files/{file_id}.
  • list_files(config) — GET /v1/files, used by scoped cleanup.
  • list_vector_stores(config) — GET /v1/vector_stores, used by scoped cleanup.
  • Add remove_file_from_vector_store/3 only if the confirmed API flow requires detach without deleting the whole old vector store.

Keep the existing 6-arity chat_stream/6 intact and add a 7-arity chat_stream/7 that accepts a tools list.

Phase 2: Collection context changes

Add lightweight collection context functions:

@spec count_records() :: non_neg_integer()
def count_records

@spec stats_summary() :: {String.t(), non_neg_integer()}
def stats_summary

@spec file_search_catalog() :: {String.t(), non_neg_integer(), String.t()}
def file_search_catalog
  • stats_summary/0 returns {stats_text, release_group_count} with no per-record catalog lines. Empty collection returns { "", 0 }.
  • file_search_catalog/0 returns deterministic text for upload plus record count and SHA-256 hash. Include the stats header and per-record lines, ordered deterministically, with only collection records (purchased_at not nil).
  • Keep any existing collection_summary/0 tests passing if the function remains. If renamed/extracted, update tests and callers so only FileStore uses the full catalog.
  • The LiveView/chat path must never receive the full catalog.

If the one-off measurement shows stats_summary/0 is too expensive, compute stats with SQL aggregates rather than loading all essential fields.

Phase 3: CollectionChat.FileStore

Create MusicLibrary.Chats.CollectionChat.FileStore with @moduledoc, specs, and public functions:

@spec upload_or_refresh() :: :ok | {:error, term()}
def upload_or_refresh

@spec get_vector_store_id() :: {:ok, String.t()} | {:error, :not_uploaded}
def get_vector_store_id

@spec cleanup_orphaned_resources() :: :ok | {:error, term()}
def cleanup_orphaned_resources

Active-state secret

Use one encrypted secret value, JSON-encoded, for example under collection_chat_file_store:

{
  "file_id": "file_...",
  "vector_store_id": "vs_...",
  "catalog_hash": "sha256...",
  "updated_at": "2026-05-22T...Z"
}

Upload/refresh flow

  1. Build {catalog_text, record_count, hash} via Collection.file_search_catalog/0.
  2. If record_count == 0:
    • delete old active vector store/file best-effort;
    • clear the active FileStore secret;
    • return :ok so collection chat falls back to stats-only.
  3. Read the active state secret.
  4. If active state exists and catalog_hash == hash, return :ok without OpenAI API calls.
  5. Upload a new file with an ML-managed filename prefix and the hash in the name.
  6. Create a new vector store with an ML-managed name prefix and the hash/timestamp in the name.
  7. Attach the uploaded file to the new vector store.
  8. Poll get_vector_store_file/3 until indexing is complete. Use bounded attempts/backoff; return {:error, reason} for failed/cancelled/timeouts.
  9. Only after indexing succeeds, persist the new active-state JSON secret.
  10. Best-effort delete the previous active vector store and file. Log cleanup failures and rely on the cleanup worker to retry later.

If any step before the active secret switch fails, best-effort delete the newly created resources and return {:error, reason}. The old active store remains available to chat users.

Read path

get_vector_store_id/0 is read-only and returns:

  • {:ok, vector_store_id} only when the active JSON secret has both a file_id and vector_store_id;
  • {:error, :not_uploaded} otherwise.

Cleanup flow

cleanup_orphaned_resources/0:

  1. Read the active state.
  2. List OpenAI vector stores and files.
  3. Select only resources with the ML-managed collection-chat prefix.
  4. Delete inactive vector stores first.
  5. Delete inactive files.
  6. Never delete resources without the ML-managed prefix.

Phase 4: Oban workers and queue config

Add :open_ai to Oban queues in config/config.exs:

queues: [default: 10, heavy_writes: 1, music_brainz: 1, discogs: 1, wikipedia: 1, last_fm: 1, open_ai: 1]

Create MusicLibrary.Worker.RefreshCollectionFile as a thin wrapper:

defmodule MusicLibrary.Worker.RefreshCollectionFile do
  use Oban.Worker,
    queue: :open_ai,
    max_attempts: 3,
    unique: [fields: [:worker], states: [:scheduled], period: :infinity],
    replace: [scheduled: [:scheduled_at]]

  @impl true
  def perform(_job) do
    case MusicLibrary.Chats.CollectionChat.FileStore.upload_or_refresh() do
      :ok -> :ok
      {:error, reason} -> MusicLibrary.Worker.ErrorHandler.to_oban_result(reason)
    end
  end

  @spec enqueue_debounced() :: {:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t()}
  def enqueue_debounced do
    %{}
    |> new(schedule_in: {5, :minutes})
    |> Oban.insert()
  end
end

Important debounce semantics:

  • uniqueness checks only :scheduled jobs, so mutations during an executing refresh can enqueue a follow-up scheduled refresh;
  • replace: [scheduled: [:scheduled_at]] means rapid mutations push the single scheduled refresh later instead of first-writer-wins;
  • the hourly cron and catalog hash skip provide recovery without repeated uploads.

Create MusicLibrary.Worker.CleanupOrphanedOpenAICollectionResources:

  • queue :open_ai;
  • max_attempts: 2;
  • delegates to FileStore.cleanup_orphaned_resources/0;
  • routes errors through Worker.ErrorHandler.

Cron:

  • production: hourly RefreshCollectionFile for initial upload/recovery; monthly cleanup after existing monthly jobs;
  • development: every 15 minutes for refresh if desired; monthly cleanup can stay production-only unless useful in dev.

Phase 5: Record mutation triggers

Add a shared enqueue helper in the Records context, e.g.:

@spec enqueue_collection_file_refresh() :: :ok
def enqueue_collection_file_refresh do
  case MusicLibrary.Worker.RefreshCollectionFile.enqueue_debounced() do
    {:ok, _job} -> :ok
    {:error, reason} ->
      Logger.warning("Failed to enqueue collection file refresh: #{inspect(reason)}")
      :ok
  end
end

Use it after successful mutations, non-blocking:

  • Records.create_record/1
  • Records.update_record/2
  • Records.delete_record/1
  • enrichment/direct update paths that affect catalog content, especially Records.Enrichment.populate_genres/1
  • any other direct update path discovered during implementation that changes title, artists, release date, format, genres, MusicBrainz ID, or purchased status.

Do not enqueue on failed changesets/API errors. It is acceptable to enqueue for some harmless non-catalog changes because the FileStore hash skip avoids unnecessary uploads.

Phase 6: LiveView and collection chat changes

CollectionLive.Index

Replace full catalog loading:

|> assign(:collection_stats, {"", 0})
|> start_async(:collection_stats, fn -> Collection.stats_summary() end)

Pass chat_context={@collection_stats} to the Chat component. Remove the @collection_summary assign from the LiveView path.

OpenAI facade/API

Update OpenAI.chat_stream/2 to accept :vector_store_ids:

vector_store_ids = Keyword.get(opts, :vector_store_ids, [])
tools = build_tools(vector_store_ids)
API.chat_stream(messages, instructions, model, temperature, open_ai_config(), on_chunk, tools)

Build tools as:

  • no vector store: [%{type: "web_search_preview"}]
  • vector store present: [%{type: "file_search", vector_store_ids: ids}, %{type: "web_search_preview"}]

Preserve the existing API.chat_stream/6 behavior for record and artist chats.

CollectionChat

CollectionChat.stream_response/3:

  1. Build instructions from stats + count only.
  2. Call FileStore.get_vector_store_id/0.
  3. Include vector_store_ids: [id] only when available.
  4. Fall back to stats-only instructions when not uploaded.

Instructions must:

  • tell the model to use stats for aggregate questions;
  • tell the model to use file search for artists/albums/genres/formats;
  • tell the model not to invent records when search returns no results;
  • keep the existing [[artist/album]] linking rule.

Phase 7: Tests

OpenAI.API tests

Add coverage for new endpoints:

  • multipart upload includes purpose: "assistants", filename, and file content;
  • vector store creation request body/name;
  • file-to-vector-store attachment;
  • vector-store-file status polling endpoint, including completed, in-progress, and failed statuses;
  • delete vector store and delete file;
  • list files and list vector stores for cleanup;
  • representative 429/5xx responses return OpenAI.API.ErrorResponse with the expected kind.

FileStore tests

Use Req.Test stubs and the Secrets context (not raw DB writes):

  • first upload creates file + vector store, attaches file, polls vector-store-file until complete, and stores active JSON state;
  • refresh creates a new file/store and does not switch active state until indexing succeeds;
  • failed refresh leaves previous active state available;
  • unchanged catalog hash skips all OpenAI API calls;
  • empty collection clears/deletes active resources so old records are not searchable;
  • partial failures best-effort clean up newly created resources;
  • cleanup deletes only inactive ML-managed resources and skips unrelated OpenAI files/vector stores;
  • get_vector_store_id/0 returns {:ok, id} only for complete active state and {:error, :not_uploaded} otherwise.

Worker tests

  • perform/1 delegates to FileStore and returns :ok on success;
  • structured OpenAI errors are routed through Worker.ErrorHandler;
  • debounced enqueue creates one scheduled job and later enqueue replaces scheduled_at rather than first-writer-wins;
  • a mutation can enqueue a follow-up job when another refresh job is already executing or no scheduled job exists.

Records/enrichment tests

  • create_record/1, update_record/2, and delete_record/1 enqueue RefreshCollectionFile via assert_enqueued;
  • failed changes do not enqueue;
  • catalog-affecting enrichment/direct update paths enqueue after success, especially genre population;
  • bulk import or repeated creates coalesce into a single scheduled refresh with the latest scheduled time.

Collection/LiveView tests

  • stats_summary/0 returns stats string and correct count;
  • stats_summary/0 returns { "", 0 } for empty collection;
  • file_search_catalog/0 returns deterministic catalog text and hash;
  • CollectionLive.Index uses @collection_stats, not @collection_summary, and Chat receives the stats tuple.

CollectionChat/OpenAI request tests

  • instructions include stats and count;
  • instructions do not include per-record catalog lines, even with records present;
  • active FileStore secret causes request tools to include file_search with the vector store ID;
  • missing/incomplete FileStore secret omits file_search and keeps web_search_preview;
  • web_search_preview remains present for all chat types;
  • SSE fixtures including response.file_search_call.* events stream successfully via the existing response.* catch-all;
  • existing record and artist chat request shapes remain unchanged.

Phase 8: Verification

Run the normal checks required by the task DoD:

  • mix compile --warnings-as-errors
  • mix test
  • mix format --check-formatted
  • mix credo

Implementation-time manual verification:

  1. Compare before/after collection chat instruction byte/token estimates.
  2. Run one FileStore refresh against test stubs and inspect logs for successful upload/index/switch/cleanup ordering.
  3. Confirm a simulated failed refresh leaves the previous vector store ID active.
  4. Confirm empty collection does not leave the old catalog searchable.
  5. If UI-visible code changes beyond the assign rename, use browser tooling for the collection page; otherwise no visual verification is required.

Phase 9: Production and documentation

No new server environment variables are expected; the implementation uses the existing OPENAI_KEY. Document this explicitly. Confirm the OpenAI project/key has access to Files, Vector Stores, and Responses file_search.

Production rollout behavior:

  • initial upload happens through the hourly cron after deploy;
  • if the user wants immediate availability, ask before performing any production action, then trigger RefreshCollectionFile through existing admin tooling/Oban UI;
  • rollback/fallback is safe because collection chat works stats-only when the active FileStore secret is missing or incomplete;
  • cleanup worker removes stale ML-managed OpenAI files/vector stores, never unrelated resources.

Update documentation:

  • docs/architecture.md — add CollectionChat.FileStore, OpenAI file/vector-store endpoints, :open_ai queue, refresh/cleanup workers, and updated collection chat data flow.
  • docs/production-infrastructure.md — add OpenAI Files/Vector Stores usage, cost profile with current pricing checked at implementation time, no-new-env-var note, initial refresh behavior, cleanup/rollback notes, and operational cautions.
  • Update any relevant tests/docs comments if function names change (collection_summary/0, stats_summary/0, file_search_catalog/0).

Definition of Done

  • #1 All new and modified modules have @moduledoc
  • #2 All public functions have @spec and @doc
  • #3 Mix compile --warnings-as-errors passes
  • #4 mix test passes with no failures
  • #5 mix format --check-formatted passes
  • #6 mix credo passes
  • #7 Documentation updated (architecture.md if new modules/schemas added)
  • #8 Commit subject references ML-156
Reference in New Issue View Git Blame Copy Permalink