Run prettier on backlog folder
This commit is contained in:
@@ -3,8 +3,8 @@ id: ML-144
|
||||
title: Implement magazine issue ingestion pipeline
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: '2026-04-22 15:38'
|
||||
updated_date: '2026-05-04 08:19'
|
||||
created_date: "2026-04-22 15:38"
|
||||
updated_date: "2026-05-04 08:19"
|
||||
labels: []
|
||||
dependencies: []
|
||||
priority: medium
|
||||
@@ -13,16 +13,20 @@ priority: medium
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Build a pipeline to ingest music magazine issues (PDF), extract structured metadata (artist mentions, album reviews, upcoming releases, features), and surface relevant information in the app.
|
||||
|
||||
## Approach (Approved)
|
||||
|
||||
**Approach 1 — Text-first, single LLM pass with structured output:**
|
||||
|
||||
- Extract text locally with pdftotext (poppler, shell from Elixir) or [kreuzberg](https://hex.pm/packages/kreuzberg)
|
||||
- Feed chunks to OpenAI Responses API with JSON schema for structured output
|
||||
- One Oban worker per issue, one API call per chunk
|
||||
- Cost estimate: ~$0.01–0.10 per issue with gpt-4o-mini
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
1. Upload PDF → store as Asset
|
||||
2. Enqueue `IngestIssue` worker (new `magazines` queue, concurrency 1)
|
||||
3. Extract text locally (pdftotext)
|
||||
@@ -34,6 +38,7 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad
|
||||
7. Resolve each mention against existing records/artists by name (MusicBrainz lookup fallback for unknowns)
|
||||
|
||||
## Personalization Layer
|
||||
|
||||
- Load collection's artist MBIDs once per issue
|
||||
- Match mentions by name → MBID (exact match, then fuzzy / MusicBrainz search)
|
||||
- For matched artists/albums: create Note linked via musicbrainz_id with article snippet
|
||||
@@ -41,6 +46,7 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad
|
||||
- Upcoming releases: new schema or flag on notes with reminder job
|
||||
|
||||
## Integration Points
|
||||
|
||||
- New context: `MusicLibrary.Magazines` with `Issue` and `Mention` schemas
|
||||
- New Oban queue: `magazines` (concurrency 1)
|
||||
- Workers: `IngestIssue`, `ExtractArticle`, `ResolveMentions`
|
||||
@@ -49,7 +55,9 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [ ] #1 Pipeline can ingest a magazine PDF and extract text without OCR
|
||||
- [ ] #2 Structured extraction (artist/album mentions, topics) is persisted to database
|
||||
- [ ] #3 Mentions are resolved against existing collection with MusicBrainz fallback lookup
|
||||
@@ -59,7 +67,9 @@ Build a pipeline to ingest music magazine issues (PDF), extract structured metad
|
||||
<!-- AC:END -->
|
||||
|
||||
## Definition of Done
|
||||
|
||||
<!-- DOD:BEGIN -->
|
||||
|
||||
- [ ] #1 I can open a page in the application, drag and drop a magazine PDF, and start ingestion
|
||||
- [ ] #2 For each artist/album in the collection and Wishlist, I can see relevant informational articles coming from ingested magazines
|
||||
- [ ] #3 I can see a list of ALL ingested items, even those which don't have direct correlation with existing records and artists
|
||||
|
||||
Reference in New Issue
Block a user