From dac946bbd689f8e14dd8ca885b1d4dc1c4eba181 Mon Sep 17 00:00:00 2001 From: Claudio Ortolina Date: Sat, 2 May 2026 17:10:29 +0100 Subject: [PATCH] ML-156: initial research --- ...or-reducing-collection-chat-token-usage.md | 250 ++++++++++++++++++ ...tion-context-to-LLM-for-collection-chat.md | 231 ++++++++++++++++ 2 files changed, 481 insertions(+) create mode 100644 backlog/docs/doc-1 - ML-156-Research-Alternatives-for-reducing-collection-chat-token-usage.md create mode 100644 backlog/tasks/ml-156 - Explore-alternatives-to-reduce-token-usage-when-providing-collection-context-to-LLM-for-collection-chat.md diff --git a/backlog/docs/doc-1 - ML-156-Research-Alternatives-for-reducing-collection-chat-token-usage.md b/backlog/docs/doc-1 - ML-156-Research-Alternatives-for-reducing-collection-chat-token-usage.md new file mode 100644 index 00000000..532135f1 --- /dev/null +++ b/backlog/docs/doc-1 - ML-156-Research-Alternatives-for-reducing-collection-chat-token-usage.md @@ -0,0 +1,250 @@ +--- +id: doc-1 +title: 'ML-156 Research: Alternatives for reducing collection chat token usage' +type: other +created_date: '2026-05-02 16:12' +--- +# ML-156 Research: Alternatives for reducing collection chat token usage + +Research document for [ML-156 - Explore alternatives to reduce token usage when providing collection context to LLM for collection chat](backlog://task/ML-156). + +--- + +## Current state + +**Token flow per new collection chat:** +1. `Collection.collection_summary/0` runs on mount via `start_async` +2. Loads ALL records: `from(r in Record, where: not is_nil(r.purchased_at), order_by: [order_alphabetically()], select: ^essential_fields())` +3. Groups by `musicbrainz_id`, formats each group as `"Artist - Title (year, formats) [genre1, genre2]"` +4. Builds stats header: `"# Stats: N releases, M artists\nGenres: ...\nFormats: ...\nEras: ..."` +5. Returns `{stats + "\n\n" + catalog, group_count}` +6. Stored in `@collection_summary` assign on the LiveView +7. Passed to Chat component as `chat_context={@collection_summary}` +8. When user sends first message, `do_send_message` calls `chat_module.stream_response(messages, chat_context, callback)` +9. `CollectionChat.stream_response/3` calls `build_instructions(summary, record_count)` +10. `build_instructions/2` calls `Prompt.build/2` which wraps the summary in identity + approach templates +11. Full instructions string is sent as the `instructions` field in the OpenAI Responses API request + +**Token estimate per catalog entry:** ~15-20 tokens (e.g., "Radiohead - OK Computer (1997, cd/vinyl) [alternative rock, art rock]\n") +**For 500 releases:** ~9,000 input tokens for catalog alone + ~100 tokens for stats + ~500 tokens for prompt template = ~9,600 tokens + +**Key files:** +- `lib/music_library/collection.ex:203-235` — `collection_summary/0` (loads and formats all records) +- `lib/music_library/chats/collection_chat.ex:18-31` — `build_instructions/2` (embeds summary in prompt) +- `lib/music_library/chats/prompt.ex` — `Prompt.build/2` (wraps in identity + approach) +- `lib/open_ai/api.ex:54-73` — `chat_stream/6` (sends to Responses API with `tools: [%{type: "web_search_preview"}]`) +- `lib/open_ai/api.ex:158-178` — `decode_responses_event/2` (SSE parser, currently only handles `response.output_text.delta`, `response.failed`, generic `response.*`) +- `lib/music_library_web/components/chat.ex:196-234` — `do_send_message/2` (dispatches to `chat_module.stream_response`) +- `lib/music_library_web/live/collection_live/index.ex:226-233,246-254,304-305` — Chat component mount and summary async loading + +### Streaming architecture constraints + +The Chat component dispatches streaming to a `Task.Supervisor` child: +```elixir +Task.Supervisor.start_child(MusicLibrary.TaskSupervisor, fn -> + case chat_module.stream_response(stream_messages, chat_context, fn chunk -> + LiveView.send_update(parent_pid, __MODULE__, id: component_id, chunk: chunk) + end) do + :ok -> LiveView.send_update(parent_pid, __MODULE__, id: component_id, done: true) + {:error, reason} -> ...send error update... + end +end) +``` + +The callback sends `[chunk: chunk]` updates; `update/2` in the Chat component accumulates text via `MDEx.Document.put_markdown`. The response is rendered as streaming markdown. + +--- + +## Alternatives + +### Alternative A: Stats-only instructions (no catalog) + +**Description:** Remove the catalog lines from instructions. Keep only the aggregated stats (release count, artist count, top genres, formats, eras). + +**Token savings:** ~9,000 → ~100 input tokens (~99% reduction) + +**Pros:** +- Simplest possible change; < 10 lines of code +- No architectural changes needed +- No streaming infrastructure changes +- The LLM can still answer statistical questions ("what's my most common genre?", "how many jazz records do I have?") + +**Cons:** +- LLM loses ability to answer specific questions ("do I have Kid A?", "which Radiohead albums do I own?", "show me my 90s electronic albums") +- User experience degrades for record-specific queries +- The LLM will hallucinate or say "I don't have access to your specific collection" frequently + +**Impact on code:** +1. `CollectionChat.build_instructions/2` — remove `#{collection_summary}` interpolation, keep only stats +2. `Collection.collection_summary/0` — could be simplified to return only stats (or keep as-is, the function is also tested independently) + +--- + +### Alternative B: Function calling (tool-based search) + +**Description:** Provide the LLM with a `search_collection` function tool. The LLM calls this tool when it needs to look up specific records in the user's collection. The instructions only include aggregated stats. + +**Token savings:** ~9,000 → ~100 input tokens base + tool call overhead + tool results (~200-500 tokens when actually searching) + +**How it works:** +1. Add a tool definition to the Responses API request: +```json +{ + "type": "function", + "name": "search_collection", + "description": "Search the user's music collection by artist name, album title, genre, format, or any combination", + "parameters": { + "type": "object", + "properties": { + "query": {"type": "string", "description": "Search query"} + }, + "required": ["query"] + } +} +``` + +2. The SSE streaming flow changes: instead of simple text-delta → done, we need to handle: + - `response.function_call_arguments.delta` / `response.function_call_arguments.done` + - Build the function call from accumulated deltas + - Execute `Collection.search_records(query)` locally + - Submit the function result back to the Responses API (second request) + - Continue streaming the text response + +3. The Chat component's streaming architecture needs to handle this multi-turn flow. + +**Pros:** +- Maximal token efficiency — only pay for what's actually needed +- LLM can answer arbitrary specific questions with real data +- Scales to any collection size +- The same tool pattern could be reused for artist chat, record chat, etc. +- Leverages the existing `tools` infrastructure in the Responses API request + +**Cons:** +- **Significantly more complex** — requires: + - Changes to `OpenAI.API.chat_stream/6` to support function calls in streaming mode + - Changes to `decode_responses_event/2` to handle function call SSE events + - A function call execution loop (model calls function → execute → submit result → model responds) + - Changes to the Chat component's `do_send_message/2` to orchestrate multi-turn tool use + - The streaming Task process becomes stateful (needs to handle the submit-then-continue loop) +- Tool results still consume tokens (but only the matching records, not the entire catalog) +- User may experience a brief pause while the tool executes (mitigated by streaming the tool call status) +- More error states to handle (function execution failures, parse errors) + +**Implementation scope:** +1. **Tool definition module** — New module ~30 lines defining the OpenAI function tool schema +2. **Function executor** — New module or function in `CollectionChat` that executes the tool call (~20 lines) +3. **Streaming changes in `OpenAI.API`** — New `chat_stream_with_tools/7` or modify `chat_stream/6` to accept tools and handle function call events (~80 lines) +4. **SSE event handling** — Add cases to `decode_responses_event/2` for function call events (~30 lines) +5. **Chat component orchestration** — Modify `do_send_message/2` or the Task function to handle tool call loops (~50 lines) +6. **`CollectionChat.build_instructions/2`** — Remove catalog, keep stats, add tool usage guidance (~10 lines) + +**Estimated lines of change:** ~200-300 lines across 6-8 files + +--- + +### Alternative C: Cached LLM-generated summary + +**Description:** Generate a concise, human-readable summary of the collection using an LLM call (e.g., "Your collection spans from 1967 to 2024 with a focus on progressive rock, featuring 3 albums by Radiohead, 2 by Pink Floyd..."). Store this summary in a `Chat` record or asset, and regenerate it when the collection changes (record added/removed/edited). + +**Token savings:** ~9,000 → ~400 tokens (summary text + stats) + +**Pros:** +- Good middle ground — compact but informative +- The LLM has a narrative understanding of the collection +- No streaming architecture changes needed +- Single one-time cost to generate (amortized across many chats) + +**Cons:** +- Still doesn't give the LLM ability to answer specific queries ("do I have Kid A?") +- Summary can go stale if not regenerated on collection changes +- Requires an initial LLM call to generate the summary (token cost + latency) +- When to regenerate is tricky — every record add/edit/delete? +- The summary is only as good as the LLM's compression; important details may be lost +- Adds a new background job / async concern + +**Implementation scope:** +1. Store the cached summary (new DB field on a canonical collection `Chat` record, or a new schema) +2. A function/worker to generate the LLM summary (calls OpenAI, stores result) +3. Invalidation triggers (PubSub on record add/edit/delete, regenerate async) +4. Changes to `CollectionChat.build_instructions/2` to use the cached summary +5. Fallback to stats-only if cache is stale/missing + +--- + +### Alternative D: Hybrid — stats in instructions + tool for catalog search + +**Description:** Combine Alternative A (stats always included) with Alternative B (function calling for specific queries). The instructions include aggregated stats and guidance to use the `search_collection` tool when the user asks about specific records. + +**Token savings:** ~9,000 → ~100 tokens base + tool results (0-500 tokens per use) + +**Pros:** +- Best of both worlds: statistical awareness always available, specific lookup on demand +- Token-efficient — base cost is minimal +- LLM knows to reach for the tool when appropriate + +**Cons:** +- Same implementation complexity as Alternative B for the tool infrastructure +- Slightly more prompt engineering to ensure the model uses the tool appropriately + +**Implementation scope:** Same as Alternative B, plus refined prompt instructions. + +--- + +### Alternative E: OpenAI `file_search` tool (recommended) + +Upload the collection catalog as a file to OpenAI, create a vector store, and use the Responses API's built-in `file_search` tool. OpenAI automatically performs semantic search over the file and includes relevant results inline. + +**Token savings:** ~9,000 → ~100 tokens base + retrieval overhead (~200-500 when model searches) + +**How it works:** +1. Format collection catalog as text (same as current `collection_summary/0` output) +2. Upload to OpenAI: `POST /v1/files` with `purpose: "assistants"` → `file_id` +3. Create vector store: `POST /v1/vector_stores` → `vector_store_id` +4. Attach file: `POST /v1/vector_stores/{id}/files` → OpenAI indexes it +5. In `chat_stream`, add `%{type: "file_search", vector_store_ids: [store_id]}` to tools +6. Model searches file when needed — results appear inline, same as web_search_preview today + +**Critical difference from Alt B:** No SSE event handling changes, no orchestration loop, no custom tool execution. `file_search` works exactly like `web_search_preview` (already in use) — OpenAI handles retrieval automatically. + +**Pros:** +- ~150 lines vs ~250 for Alt B +- No changes to `decode_responses_event/2` or Chat streaming loop +- Semantic search (finds "upbeat 80s rock" not just keywords) +- Files only uploaded on collection change (amortized cost) +- Reusable for artist bios, notes, etc. + +**Cons:** +- File can go stale if not updated on collection change +- 2-3 new API endpoints needed (Files API, Vector Stores API) +- Vector store indexing is async (may need brief poll) +- First chat after deploy has upload+index latency +- File storage cost at OpenAI (negligible for text) +- Search quality depends on OpenAI's embedding model, not directly controllable + +--- + +## Comparison + +| Criterion | A (stats-only) | B (custom func) | C (cached) | E (file_search) | +|---|---|---|---|---| +| Implementation complexity | ★☆☆☆☆ | ★★★★☆ | ★★★☆☆ | ★★☆☆☆ | +| Response quality | ★★☆☆☆ | ★★★★★ | ★★★☆☆ | ★★★★★ | +| Token efficiency | ★★★★★ | ★★★★★ | ★★★★☆ | ★★★★★ | +| Infrastructure risk | ★★★★★ | ★★★☆☆ | ★★★★☆ | ★★★★☆ | +| Reusability | ★☆☆☆☆ | ★★★★☆ | ★★☆☆☆ | ★★★☆☆ | + +--- + +## Why Alternative E over B + +Alternative E (`file_search`) was chosen over Alternative B (custom function calling) for these reasons: + +1. **Same response quality with half the code** (~150 lines vs ~250 lines). Both approaches let the LLM answer arbitrary specific questions with real collection data. + +2. **No streaming infrastructure changes.** Alternative B requires modifying `decode_responses_event/2`, the SSE event loop, and `do_send_message/2` to handle function call orchestration. Alternative E simply adds a `file_search` tool entry alongside the existing `web_search_preview` — no new SSE event types to parse, no orchestration loop, no stateful streaming. + +3. **Lower risk.** The core streaming code path is untouched. The new code is additive (new API endpoints, new `FileStore` module) rather than modifying the shared streaming infrastructure that serves all three chat types (record, artist, collection). + +4. **Better search quality.** OpenAI's semantic search is likely superior to FTS5 for natural language queries like "upbeat 80s rock with synths". + +5. **Alternative B would be the right choice if** the application needed parameterized queries (artist+format+year range), real-time data that changes mid-chat, or complex database filtering. For a static-ish catalog that changes infrequently, `file_search` is the pragmatic choice. diff --git a/backlog/tasks/ml-156 - Explore-alternatives-to-reduce-token-usage-when-providing-collection-context-to-LLM-for-collection-chat.md b/backlog/tasks/ml-156 - Explore-alternatives-to-reduce-token-usage-when-providing-collection-context-to-LLM-for-collection-chat.md new file mode 100644 index 00000000..402cb386 --- /dev/null +++ b/backlog/tasks/ml-156 - Explore-alternatives-to-reduce-token-usage-when-providing-collection-context-to-LLM-for-collection-chat.md @@ -0,0 +1,231 @@ +--- +id: ML-156 +title: >- + Explore alternatives to reduce token usage when providing collection context + to LLM for collection chat +status: To Do +assignee: [] +created_date: '2026-05-02 16:02' +updated_date: '2026-05-02 16:13' +labels: + - chat + - collection + - openai + - token-optimization +dependencies: [] +references: + - 'backlog://document/doc-1' +documentation: + - lib/music_library/chats/collection_chat.ex + - lib/music_library/collection.ex + - lib/music_library/chats/prompt.ex + - lib/music_library/chats/stream_provider.ex + - lib/open_ai/api.ex + - lib/music_library_web/components/chat.ex + - lib/music_library_web/live/collection_live/index.ex +priority: high +--- + +## Description + + +Currently, every new collection chat sends the ENTIRE collection catalog (all records formatted as "Artist - Title (year, format) [genres]") plus aggregated stats as the `instructions` parameter to the OpenAI Responses API. For a collection of 500+ records, this burns ~9,000+ input tokens on EVERY new chat start — regardless of what the user asks. + +The goal of this task is to analyze alternatives, pick the best one, and implement it. The selected approach should: +- Significantly reduce per-chat token usage +- Preserve or improve the quality of LLM responses about the collection +- Not require architectural overhauls beyond the chat/streaming layer + +The `Collection.collection_summary/0` function loads ALL records from the DB, formats them, and returns `{summary, count}`. This is computed asynchronously in `CollectionLive.Index.mount/3` and passed to the Chat component as `chat_context`. `CollectionChat.build_instructions/2` then embeds the full summary into the instructions string sent to OpenAI. + + +## Acceptance Criteria + +- [ ] #1 `CollectionChat.build_instructions/2` no longer interpolates the full collection catalog into the instructions sent to OpenAI — only aggregated stats and a record count are included +- [ ] #2 A `file_search` tool with the collection's vector store ID is included in every collection chat request to the OpenAI Responses API +- [ ] #3 A `CollectionChat.FileStore` module manages the collection file lifecycle: upload to OpenAI Files API, vector store creation, and file-to-store attachment, persisting IDs via `Secrets` +- [ ] #4 When a record is added, edited, or deleted, the collection file at OpenAI is refreshed (async, non-blocking) so the LLM always searches up-to-date data +- [ ] #5 If the file/vector store is unavailable (upload failed, first deploy), the chat falls back to stats-only instructions without errors +- [ ] #6 Existing record and artist chats continue to work without changes (no regression in streaming) +- [ ] #7 Tests cover: Files API endpoints, file upload/create/refresh lifecycle, `file_search` tool inclusion in chat requests, fallback when file store is unavailable, empty collection edge case +- [ ] #8 Per-chat input tokens for the collection chat instructions are O(1) relative to collection size (the file is searched by OpenAI on demand, not embedded in instructions) + + +## Implementation Plan + + +## Approach: OpenAI `file_search` tool + +Upload the collection catalog as a file to OpenAI, create a vector store, and use the Responses API's built-in `file_search` tool. OpenAI automatically performs semantic search over the file and includes relevant results inline in the response stream — no SSE event handling changes, no orchestration loop. + +Token savings: ~9,000 → ~100 tokens per chat (99% reduction). File search results consume ~200-500 tokens only when the model actually searches. + +Research and alternative analysis: see [ML-156 Research document](backlog://document/doc-1). + +--- + +### Phase 1: OpenAI API extensions + +Add to `OpenAI.API` (following existing `new_request/1` pattern with `Req.RateLimiter` on `:open_ai` bucket): + +- `upload_file(file_content, config)` — `POST /v1/files` with multipart body, `purpose: "assistants"` +- `create_vector_store(name, config)` — `POST /v1/vector_stores` +- `add_file_to_vector_store(store_id, file_id, config)` — `POST /v1/vector_stores/{id}/files` +- `delete_file(file_id, config)` — `DELETE /v1/files/{id}` (cleanup on re-upload) + +~60 lines. + +### Phase 2: File management module + +Create `MusicLibrary.Chats.CollectionChat.FileStore`: + +``` +defmodule MusicLibrary.Chats.CollectionChat.FileStore do + @moduledoc """ + Manages the collection catalog file lifecycle at OpenAI. + Persists file_id and vector_store_id via Secrets. + """ + + @spec ensure_uploaded() :: {:ok, String.t()} | {:error, term()} + def ensure_uploaded do + # Check Secrets for existing file_id + vector_store_id + # If missing, call Collection.collection_summary/0 + # Upload to OpenAI, create vector store, attach file + # Persist IDs via Secrets.store/2 + # Return {:ok, vector_store_id} + end + + @spec refresh() :: :ok | {:error, term()} + def refresh do + # Delete old file from OpenAI (if exists) + # Regenerate summary, upload new file + # Attach to existing vector store (re-indexes automatically) + end + + @spec get_vector_store_id() :: {:ok, String.t()} | {:error, :not_uploaded} + def get_vector_store_id do + # Read vector_store_id from Secrets + end +end +``` + +- `ensure_uploaded/0` — lazy init on first chat; idempotent +- `refresh/0` — called when records are added/edited/deleted (async, non-blocking) +- `get_vector_store_id/0` — reads from `Secrets`; returns error if never uploaded + +~50 lines. + +### Phase 3: Chat streaming changes + +**`OpenAI.chat_stream/2`** — add optional `vector_store_ids` option: + +```elixir +def chat_stream(messages, opts) do + model = Keyword.get(opts, :model, "gpt-4.1") + vector_store_ids = Keyword.get(opts, :vector_store_ids, []) + + tools = [%{type: "web_search_preview"}] + tools = if vector_store_ids != [], + do: [%{type: "file_search", vector_store_ids: vector_store_ids} | tools], + else: tools + + # ... rest unchanged +end +``` + +**`OpenAI.API.chat_stream/6`** — accept tools as parameter instead of hardcoding (or add `tools` parameter): + +```elixir +def chat_stream(messages, instructions, model, temperature, config, cb, tools \\ nil) do + tools = tools || [%{type: "web_search_preview"}] + # ... use tools in json body +end +``` + +**`CollectionChat.stream_response/3`:** + +```elixir +def stream_response(messages, {_summary, record_count}, callback) do + instructions = build_instructions(record_count) + + vector_store_opts = case FileStore.get_vector_store_id() do + {:ok, id} -> [vector_store_ids: [id]] + {:error, _} -> [] # fall back to stats-only + end + + OpenAI.chat_stream(messages, [ + on_chunk: callback, + instructions: instructions, + model: "gpt-5.1" + ] ++ vector_store_opts) +end +``` + +~15 lines. + +### Phase 4: Prompt changes + +Update `CollectionChat.build_instructions/2`: + +```elixir +defp build_instructions(record_count) do + Prompt.build(""" + Answer questions about the user's music collection. + The collection contains #{record_count} records. + Use file search to find specific records when the user asks about \ + artists, albums, genres, or formats in their collection. + + # Mentioning artists/albums + + **IF YOU MENTION AN ARTIST NAME OR ALBUM NAME, wrap it in "[[name]]", \ + for example "[[Steven Wilson]]" + """) +end +``` + +Key changes: +- Remove `#{collection_summary}` interpolation entirely +- Remove `Collection catalog:` section +- Remove `Use the provided collection catalog as your primary reference` +- Add guidance to use file search for specific record lookup +- `stream_response/3` signature changes from `{summary, count}` to just `count` (summary only used for stats, which we no longer pass) + +~10 lines. + +### Phase 5: Trigger refresh on collection changes + +In `CollectionLive.Index`, handle record add/edit/delete events: + +```elixir +# In handle_info for RecordForm saved / AddRecord imported / delete: +def handle_info({MusicLibraryWeb.Components.RecordForm, {:saved, _record}}, socket) do + Task.start(&MusicLibrary.Chats.CollectionChat.FileStore.refresh/0) + IndexActions.handle_record_saved(socket) +end +``` + +Or use the existing PubSub topic `"records:#{id}"` to trigger refresh from a central place. Debounce rapid changes (multiple quick adds) with a short timer. + +~20 lines. + +### Phase 6: Testing + +- `test/open_ai/api_test.exs` — test `upload_file`, `create_vector_store`, `add_file_to_vector_store`, `delete_file` endpoints via `Req.Test` stubs +- `test/music_library/chats/collection_chat/file_store_test.exs` — test `ensure_uploaded` (first-time upload), `refresh` (re-upload), `get_vector_store_id` (missing, present), and `Secrets` persistence +- `test/music_library/chats/collection_chat_test.exs` — verify instructions no longer contain catalog; verify `file_search` tool included when vector store is available; verify fallback to stats-only when unavailable +- `test/music_library/collection_test.exs` — existing `collection_summary/0` tests continue to pass unchanged + +Estimated total: ~150 lines across 5-6 files. Low risk — no changes to core streaming infrastructure. + + +## Definition of Done + +- [ ] #1 All new and modified modules have @moduledoc +- [ ] #2 All public functions have @spec and @doc +- [ ] #3 Mix compile --warnings-as-errors passes +- [ ] #4 mix test passes with no failures +- [ ] #5 mix format --check-formatted passes +- [ ] #6 mix credo passes +- [ ] #7 Documentation updated (architecture.md if new modules/schemas added) +- [ ] #8 Commit subject references ML-156 +