ML-156: initial research
This commit is contained in:
Claudio Ortolina
+250
@@ -0,0 +1,250 @@
|
||||
---
|
||||
id: doc-1
|
||||
title: 'ML-156 Research: Alternatives for reducing collection chat token usage'
|
||||
type: other
|
||||
created_date: '2026-05-02 16:12'
|
||||
---
|
||||
# ML-156 Research: Alternatives for reducing collection chat token usage
|
||||
|
||||
Research document for [ML-156 - Explore alternatives to reduce token usage when providing collection context to LLM for collection chat](backlog://task/ML-156).
|
||||
|
||||
---
|
||||
|
||||
## Current state
|
||||
|
||||
**Token flow per new collection chat:**
|
||||
1. `Collection.collection_summary/0` runs on mount via `start_async`
|
||||
2. Loads ALL records: `from(r in Record, where: not is_nil(r.purchased_at), order_by: [order_alphabetically()], select: ^essential_fields())`
|
||||
3. Groups by `musicbrainz_id`, formats each group as `"Artist - Title (year, formats) [genre1, genre2]"`
|
||||
4. Builds stats header: `"# Stats: N releases, M artists\nGenres: ...\nFormats: ...\nEras: ..."`
|
||||
5. Returns `{stats + "\n\n" + catalog, group_count}`
|
||||
6. Stored in `@collection_summary` assign on the LiveView
|
||||
7. Passed to Chat component as `chat_context={@collection_summary}`
|
||||
8. When user sends first message, `do_send_message` calls `chat_module.stream_response(messages, chat_context, callback)`
|
||||
9. `CollectionChat.stream_response/3` calls `build_instructions(summary, record_count)`
|
||||
10. `build_instructions/2` calls `Prompt.build/2` which wraps the summary in identity + approach templates
|
||||
11. Full instructions string is sent as the `instructions` field in the OpenAI Responses API request
|
||||
|
||||
**Token estimate per catalog entry:** ~15-20 tokens (e.g., "Radiohead - OK Computer (1997, cd/vinyl) [alternative rock, art rock]\n")
|
||||
**For 500 releases:** ~9,000 input tokens for catalog alone + ~100 tokens for stats + ~500 tokens for prompt template = ~9,600 tokens
|
||||
|
||||
**Key files:**
|
||||
- `lib/music_library/collection.ex:203-235` — `collection_summary/0` (loads and formats all records)
|
||||
- `lib/music_library/chats/collection_chat.ex:18-31` — `build_instructions/2` (embeds summary in prompt)
|
||||
- `lib/music_library/chats/prompt.ex` — `Prompt.build/2` (wraps in identity + approach)
|
||||
- `lib/open_ai/api.ex:54-73` — `chat_stream/6` (sends to Responses API with `tools: [%{type: "web_search_preview"}]`)
|
||||
- `lib/open_ai/api.ex:158-178` — `decode_responses_event/2` (SSE parser, currently only handles `response.output_text.delta`, `response.failed`, generic `response.*`)
|
||||
- `lib/music_library_web/components/chat.ex:196-234` — `do_send_message/2` (dispatches to `chat_module.stream_response`)
|
||||
- `lib/music_library_web/live/collection_live/index.ex:226-233,246-254,304-305` — Chat component mount and summary async loading
|
||||
|
||||
### Streaming architecture constraints
|
||||
|
||||
The Chat component dispatches streaming to a `Task.Supervisor` child:
|
||||
```elixir
|
||||
Task.Supervisor.start_child(MusicLibrary.TaskSupervisor, fn ->
|
||||
case chat_module.stream_response(stream_messages, chat_context, fn chunk ->
|
||||
LiveView.send_update(parent_pid, __MODULE__, id: component_id, chunk: chunk)
|
||||
end) do
|
||||
:ok -> LiveView.send_update(parent_pid, __MODULE__, id: component_id, done: true)
|
||||
{:error, reason} -> ...send error update...
|
||||
end
|
||||
end)
|
||||
```
|
||||
|
||||
The callback sends `[chunk: chunk]` updates; `update/2` in the Chat component accumulates text via `MDEx.Document.put_markdown`. The response is rendered as streaming markdown.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Alternative A: Stats-only instructions (no catalog)
|
||||
|
||||
**Description:** Remove the catalog lines from instructions. Keep only the aggregated stats (release count, artist count, top genres, formats, eras).
|
||||
|
||||
**Token savings:** ~9,000 → ~100 input tokens (~99% reduction)
|
||||
|
||||
**Pros:**
|
||||
- Simplest possible change; < 10 lines of code
|
||||
- No architectural changes needed
|
||||
- No streaming infrastructure changes
|
||||
- The LLM can still answer statistical questions ("what's my most common genre?", "how many jazz records do I have?")
|
||||
|
||||
**Cons:**
|
||||
- LLM loses ability to answer specific questions ("do I have Kid A?", "which Radiohead albums do I own?", "show me my 90s electronic albums")
|
||||
- User experience degrades for record-specific queries
|
||||
- The LLM will hallucinate or say "I don't have access to your specific collection" frequently
|
||||
|
||||
**Impact on code:**
|
||||
1. `CollectionChat.build_instructions/2` — remove `#{collection_summary}` interpolation, keep only stats
|
||||
2. `Collection.collection_summary/0` — could be simplified to return only stats (or keep as-is, the function is also tested independently)
|
||||
|
||||
---
|
||||
|
||||
### Alternative B: Function calling (tool-based search)
|
||||
|
||||
**Description:** Provide the LLM with a `search_collection` function tool. The LLM calls this tool when it needs to look up specific records in the user's collection. The instructions only include aggregated stats.
|
||||
|
||||
**Token savings:** ~9,000 → ~100 input tokens base + tool call overhead + tool results (~200-500 tokens when actually searching)
|
||||
|
||||
**How it works:**
|
||||
1. Add a tool definition to the Responses API request:
|
||||
```json
|
||||
{
|
||||
"type": "function",
|
||||
"name": "search_collection",
|
||||
"description": "Search the user's music collection by artist name, album title, genre, format, or any combination",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "Search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. The SSE streaming flow changes: instead of simple text-delta → done, we need to handle:
|
||||
- `response.function_call_arguments.delta` / `response.function_call_arguments.done`
|
||||
- Build the function call from accumulated deltas
|
||||
- Execute `Collection.search_records(query)` locally
|
||||
- Submit the function result back to the Responses API (second request)
|
||||
- Continue streaming the text response
|
||||
|
||||
3. The Chat component's streaming architecture needs to handle this multi-turn flow.
|
||||
|
||||
**Pros:**
|
||||
- Maximal token efficiency — only pay for what's actually needed
|
||||
- LLM can answer arbitrary specific questions with real data
|
||||
- Scales to any collection size
|
||||
- The same tool pattern could be reused for artist chat, record chat, etc.
|
||||
- Leverages the existing `tools` infrastructure in the Responses API request
|
||||
|
||||
**Cons:**
|
||||
- **Significantly more complex** — requires:
|
||||
- Changes to `OpenAI.API.chat_stream/6` to support function calls in streaming mode
|
||||
- Changes to `decode_responses_event/2` to handle function call SSE events
|
||||
- A function call execution loop (model calls function → execute → submit result → model responds)
|
||||
- Changes to the Chat component's `do_send_message/2` to orchestrate multi-turn tool use
|
||||
- The streaming Task process becomes stateful (needs to handle the submit-then-continue loop)
|
||||
- Tool results still consume tokens (but only the matching records, not the entire catalog)
|
||||
- User may experience a brief pause while the tool executes (mitigated by streaming the tool call status)
|
||||
- More error states to handle (function execution failures, parse errors)
|
||||
|
||||
**Implementation scope:**
|
||||
1. **Tool definition module** — New module ~30 lines defining the OpenAI function tool schema
|
||||
2. **Function executor** — New module or function in `CollectionChat` that executes the tool call (~20 lines)
|
||||
3. **Streaming changes in `OpenAI.API`** — New `chat_stream_with_tools/7` or modify `chat_stream/6` to accept tools and handle function call events (~80 lines)
|
||||
4. **SSE event handling** — Add cases to `decode_responses_event/2` for function call events (~30 lines)
|
||||
5. **Chat component orchestration** — Modify `do_send_message/2` or the Task function to handle tool call loops (~50 lines)
|
||||
6. **`CollectionChat.build_instructions/2`** — Remove catalog, keep stats, add tool usage guidance (~10 lines)
|
||||
|
||||
**Estimated lines of change:** ~200-300 lines across 6-8 files
|
||||
|
||||
---
|
||||
|
||||
### Alternative C: Cached LLM-generated summary
|
||||
|
||||
**Description:** Generate a concise, human-readable summary of the collection using an LLM call (e.g., "Your collection spans from 1967 to 2024 with a focus on progressive rock, featuring 3 albums by Radiohead, 2 by Pink Floyd..."). Store this summary in a `Chat` record or asset, and regenerate it when the collection changes (record added/removed/edited).
|
||||
|
||||
**Token savings:** ~9,000 → ~400 tokens (summary text + stats)
|
||||
|
||||
**Pros:**
|
||||
- Good middle ground — compact but informative
|
||||
- The LLM has a narrative understanding of the collection
|
||||
- No streaming architecture changes needed
|
||||
- Single one-time cost to generate (amortized across many chats)
|
||||
|
||||
**Cons:**
|
||||
- Still doesn't give the LLM ability to answer specific queries ("do I have Kid A?")
|
||||
- Summary can go stale if not regenerated on collection changes
|
||||
- Requires an initial LLM call to generate the summary (token cost + latency)
|
||||
- When to regenerate is tricky — every record add/edit/delete?
|
||||
- The summary is only as good as the LLM's compression; important details may be lost
|
||||
- Adds a new background job / async concern
|
||||
|
||||
**Implementation scope:**
|
||||
1. Store the cached summary (new DB field on a canonical collection `Chat` record, or a new schema)
|
||||
2. A function/worker to generate the LLM summary (calls OpenAI, stores result)
|
||||
3. Invalidation triggers (PubSub on record add/edit/delete, regenerate async)
|
||||
4. Changes to `CollectionChat.build_instructions/2` to use the cached summary
|
||||
5. Fallback to stats-only if cache is stale/missing
|
||||
|
||||
---
|
||||
|
||||
### Alternative D: Hybrid — stats in instructions + tool for catalog search
|
||||
|
||||
**Description:** Combine Alternative A (stats always included) with Alternative B (function calling for specific queries). The instructions include aggregated stats and guidance to use the `search_collection` tool when the user asks about specific records.
|
||||
|
||||
**Token savings:** ~9,000 → ~100 tokens base + tool results (0-500 tokens per use)
|
||||
|
||||
**Pros:**
|
||||
- Best of both worlds: statistical awareness always available, specific lookup on demand
|
||||
- Token-efficient — base cost is minimal
|
||||
- LLM knows to reach for the tool when appropriate
|
||||
|
||||
**Cons:**
|
||||
- Same implementation complexity as Alternative B for the tool infrastructure
|
||||
- Slightly more prompt engineering to ensure the model uses the tool appropriately
|
||||
|
||||
**Implementation scope:** Same as Alternative B, plus refined prompt instructions.
|
||||
|
||||
---
|
||||
|
||||
### Alternative E: OpenAI `file_search` tool (recommended)
|
||||
|
||||
Upload the collection catalog as a file to OpenAI, create a vector store, and use the Responses API's built-in `file_search` tool. OpenAI automatically performs semantic search over the file and includes relevant results inline.
|
||||
|
||||
**Token savings:** ~9,000 → ~100 tokens base + retrieval overhead (~200-500 when model searches)
|
||||
|
||||
**How it works:**
|
||||
1. Format collection catalog as text (same as current `collection_summary/0` output)
|
||||
2. Upload to OpenAI: `POST /v1/files` with `purpose: "assistants"` → `file_id`
|
||||
3. Create vector store: `POST /v1/vector_stores` → `vector_store_id`
|
||||
4. Attach file: `POST /v1/vector_stores/{id}/files` → OpenAI indexes it
|
||||
5. In `chat_stream`, add `%{type: "file_search", vector_store_ids: [store_id]}` to tools
|
||||
6. Model searches file when needed — results appear inline, same as web_search_preview today
|
||||
|
||||
**Critical difference from Alt B:** No SSE event handling changes, no orchestration loop, no custom tool execution. `file_search` works exactly like `web_search_preview` (already in use) — OpenAI handles retrieval automatically.
|
||||
|
||||
**Pros:**
|
||||
- ~150 lines vs ~250 for Alt B
|
||||
- No changes to `decode_responses_event/2` or Chat streaming loop
|
||||
- Semantic search (finds "upbeat 80s rock" not just keywords)
|
||||
- Files only uploaded on collection change (amortized cost)
|
||||
- Reusable for artist bios, notes, etc.
|
||||
|
||||
**Cons:**
|
||||
- File can go stale if not updated on collection change
|
||||
- 2-3 new API endpoints needed (Files API, Vector Stores API)
|
||||
- Vector store indexing is async (may need brief poll)
|
||||
- First chat after deploy has upload+index latency
|
||||
- File storage cost at OpenAI (negligible for text)
|
||||
- Search quality depends on OpenAI's embedding model, not directly controllable
|
||||
|
||||
---
|
||||
|
||||
## Comparison
|
||||
|
||||
| Criterion | A (stats-only) | B (custom func) | C (cached) | E (file_search) |
|
||||
|---|---|---|---|---|
|
||||
| Implementation complexity | ★☆☆☆☆ | ★★★★☆ | ★★★☆☆ | ★★☆☆☆ |
|
||||
| Response quality | ★★☆☆☆ | ★★★★★ | ★★★☆☆ | ★★★★★ |
|
||||
| Token efficiency | ★★★★★ | ★★★★★ | ★★★★☆ | ★★★★★ |
|
||||
| Infrastructure risk | ★★★★★ | ★★★☆☆ | ★★★★☆ | ★★★★☆ |
|
||||
| Reusability | ★☆☆☆☆ | ★★★★☆ | ★★☆☆☆ | ★★★☆☆ |
|
||||
|
||||
---
|
||||
|
||||
## Why Alternative E over B
|
||||
|
||||
Alternative E (`file_search`) was chosen over Alternative B (custom function calling) for these reasons:
|
||||
|
||||
1. **Same response quality with half the code** (~150 lines vs ~250 lines). Both approaches let the LLM answer arbitrary specific questions with real collection data.
|
||||
|
||||
2. **No streaming infrastructure changes.** Alternative B requires modifying `decode_responses_event/2`, the SSE event loop, and `do_send_message/2` to handle function call orchestration. Alternative E simply adds a `file_search` tool entry alongside the existing `web_search_preview` — no new SSE event types to parse, no orchestration loop, no stateful streaming.
|
||||
|
||||
3. **Lower risk.** The core streaming code path is untouched. The new code is additive (new API endpoints, new `FileStore` module) rather than modifying the shared streaming infrastructure that serves all three chat types (record, artist, collection).
|
||||
|
||||
4. **Better search quality.** OpenAI's semantic search is likely superior to FTS5 for natural language queries like "upbeat 80s rock with synths".
|
||||
|
||||
5. **Alternative B would be the right choice if** the application needed parameterized queries (artist+format+year range), real-time data that changes mid-chat, or complex database filtering. For a static-ish catalog that changes infrequently, `file_search` is the pragmatic choice.
|
||||
Reference in New Issue
Block a user