diff --git a/backlog/tasks/ml-144 - Implement-magazine-issue-ingestion-pipeline.md b/backlog/tasks/ml-144 - Implement-magazine-issue-ingestion-pipeline.md index d18a1dda..abc8822c 100644 --- a/backlog/tasks/ml-144 - Implement-magazine-issue-ingestion-pipeline.md +++ b/backlog/tasks/ml-144 - Implement-magazine-issue-ingestion-pipeline.md @@ -4,6 +4,7 @@ title: Implement magazine issue ingestion pipeline status: To Do assignee: [] created_date: '2026-04-22 15:38' +updated_date: '2026-04-24 06:51' labels: - feature - magazines @@ -21,7 +22,7 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad ## Approach (Approved) **Approach 1 — Text-first, single LLM pass with structured output:** -- Extract text locally with pdftotext (poppler, shell from Elixir) +- Extract text locally with pdftotext (poppler, shell from Elixir) or [kreuzberg](https://hex.pm/packages/kreuzberg) - Feed chunks to OpenAI Responses API with JSON schema for structured output - One Oban worker per issue, one API call per chunk - Cost estimate: ~$0.01–0.10 per issue with gpt-4o-mini @@ -32,8 +33,8 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad 3. Extract text locally (pdftotext) 4. Split by section boundaries (page breaks, headings) 5. Per-chunk LLM call requesting structured payload: - - `mentions`: [{type: "artist"|"album"|"track", name, artist_name?, release_date?, context_snippet}] - - `topic`: "review"|"news"|"upcoming"|"feature" + - `mentions`: `[{type: "artist"|"album"|"track", name, artist_name?, release_date?, context_snippet}]` + - `topic`: `"review"|"news"|"upcoming"|"feature"` 6. Persist to new `Magazines` context (Issue, Mention schemas) 7. Resolve each mention against existing records/artists by name (MusicBrainz lookup fallback for unknowns) @@ -61,3 +62,10 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad - [ ] #5 Magazine mentions surface on record/artist detail pages - [ ] #6 Cost per issue documented and under budget estimate + +## Definition of Done + +- [ ] #1 I can open a page in the application, drag and drop a magazine PDF, and start ingestion +- [ ] #2 For each artist/album in the collection and Wishlist, I can see relevant informational articles coming from ingested magazines +- [ ] #3 I can see a list of ALL ingested items, even those which don't have direct correlation with existing records and artists +