Ml-162: plan
This commit is contained in:
Claudio Ortolina
@@ -4,9 +4,10 @@ title: Expose production errors via JSON API endpoint
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: '2026-05-04 08:08'
|
||||
updated_date: '2026-05-04 08:19'
|
||||
updated_date: '2026-05-04 09:12'
|
||||
labels:
|
||||
- api
|
||||
- ready
|
||||
dependencies: []
|
||||
parent_task_id: DRAFT-1
|
||||
priority: medium
|
||||
@@ -43,3 +44,261 @@ This subtask covers the server-side work only: controller, JSON serialization, c
|
||||
- Uses the `error_tracker_errors` and `error_tracker_occurrences` tables (already exist)
|
||||
- Auth via existing `require_api_token` plug (already in use by `/api/v1` pipeline)
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
## Implementation Plan
|
||||
|
||||
### Objective alignment
|
||||
|
||||
Add two JSON API endpoints under `/api/v1/errors` (already behind the existing Bearer token auth plug `require_api_token`) that expose `error_tracker_errors` and `error_tracker_occurrences` table data. The endpoints enable programmatic access to production errors for the pi tooling (ML-163) and extension (ML-164) subtasks.
|
||||
|
||||
**Data source correction**: The task description states "Uses `MusicLibrary.TelemetryRepo`" but `error_tracker` is configured with `repo: MusicLibrary.Repo` in `config/config.exs`. The error_tracker tables (`error_tracker_errors`, `error_tracker_occurrences`) live in the main app database, not the telemetry database. **All queries in this plan use `MusicLibrary.Repo`** — the repo that actually holds the data. (The TelemetryRepo holds only telemetry metrics data.)
|
||||
|
||||
**PK type correction**: The task description describes `id` as UUID, but `error_tracker` uses auto-increment INTEGER primary keys. The API will return integer IDs.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
1. **Query error_tracker tables directly from the controller** — Rejected. Violates the "Context modules own all queries" architecture convention. All LiveViews and controllers call context functions.
|
||||
|
||||
2. **Add error query functions to an existing context** — Rejected. No existing context owns these tables. A new `MusicLibrary.Errors` context is the cleanest home.
|
||||
|
||||
3. **Use error_tracker's built-in query functions** — Rejected. `ErrorTracker` exposes CRUD operations (`report/3`, `resolve/1`, `mute/1`, etc.) but no listing/querying API for retrieval. We need raw `Ecto.Query` against the schemas.
|
||||
|
||||
4. **Add the context under `MusicLibraryWeb`** — Rejected. Contexts live under `lib/music_library/` by convention.
|
||||
|
||||
5. **Pagination via cursor vs offset** — Offset chosen. No requirement for cursor pagination; the data volume is small (hundreds of unique errors, not millions); offset pagination is simpler and maps to the task's `limit`/`offset` params.
|
||||
|
||||
### Architecture impact analysis
|
||||
|
||||
| Touchpoint | Impact |
|
||||
|---|---|
|
||||
| **New context: `lib/music_library/errors.ex`** | New module. Two public functions: `list_errors/1` (filtered + paginated list with counts) and `get_error!/1` (single error with occurrences). |
|
||||
| **New controller: `lib/music_library_web/controllers/error_controller.ex`** | New module. Two actions: `index/2`, `show/2`. Parses query params with `parse_int` helper. |
|
||||
| **New JSON view: `lib/music_library_web/controllers/error_json.ex`** | New module. Two render functions: `index/1`, `show/1`. Serializes errors and occurrences. |
|
||||
| **Router: `lib/music_library_web/router.ex`** | Add two new routes under the existing `scope "/api/v1"` block (`GET /api/v1/errors`, `GET /api/v1/errors/:id`). No new pipeline or scope. |
|
||||
| **No PubSub impact** | These are read-only endpoints. No real-time updates needed. |
|
||||
| **No supervision tree impact** | The context is stateless. No new processes. |
|
||||
| **No external API impact** | Internal API only. |
|
||||
| **No migration needed** | Tables already exist (created by error_tracker's own migrations in `20260226212444_add_error_tracker.exs`). |
|
||||
| **No deprecation path** | Net-new addition. |
|
||||
|
||||
### Performance profile
|
||||
|
||||
**List endpoint (`GET /api/v1/errors`)**: Two queries.
|
||||
1. COUNT query with same filters (no OFFSET/LIMIT). Complexity: O(n) scan of filtered rows on `error_tracker_errors` table. With typical production volumes (hundreds of unique errors), this is trivially fast.
|
||||
2. SELECT with filters + ORDER BY + LIMIT + OFFSET. Same O(n) scan profile. No joins — all data is in the single table for the list view.
|
||||
|
||||
No N+1 risk in the list endpoint: we return error-level data only, no occurrence preloading.
|
||||
|
||||
**Single error endpoint (`GET /api/v1/errors/:id`)**: Three queries.
|
||||
1. `Repo.get!` on `error_tracker_errors` by ID. PK lookup is O(1).
|
||||
2. `Repo.preload` occurrences sorted by `inserted_at DESC`. This produces a single LEFT JOIN or an IN query (depends on preload strategy). For errors with hundreds of occurrences, the preload could return substantial data.
|
||||
3. A subquery COUNT on occurrences for `occurrence_count` (if not already preloaded). Can be merged with the preload.
|
||||
|
||||
No pagination on occurrences in this version — the task spec shows all occurrences returned. If an error has many occurrences (e.g., thousands for a noisy bot error), this could produce a large response. **Mitigation**: the task explicitly says "all its occurrences", so we follow the spec. A future version could add `limit`/`offset` query params for occurrences if needed.
|
||||
|
||||
**Memory**: JSON response is built in memory from Ecto structs. For an error with 100 occurrences each containing full stacktraces, response size could reach ~100KB-500KB. Acceptable for an authenticated internal tooling API accessed infrequently.
|
||||
|
||||
**Latency**: Sub-100ms for typical queries on SQLite with in-memory page cache. No external API calls.
|
||||
|
||||
### Benchmarking requirements
|
||||
|
||||
No dedicated benchmarks needed for this change. The endpoints are read-only against SQLite with simple queries on small tables (hundreds of rows). Standard test coverage will verify correctness.
|
||||
|
||||
If latency becomes a concern when occurrences grow large, we could add a `limit` param for occurrences in a follow-up. This is documented as a future consideration, not a current requirement.
|
||||
|
||||
### Cost profile
|
||||
|
||||
No paid resources consumed. The endpoints are internal HTTP handlers that only query the existing SQLite database and return JSON. No external API calls, no additional compute, no storage.
|
||||
|
||||
### Implementation steps (sequential order)
|
||||
|
||||
---
|
||||
|
||||
#### Step 1: Create `MusicLibrary.Errors` context module
|
||||
|
||||
**What**: New `lib/music_library/errors.ex` with two public functions and private query helpers.
|
||||
|
||||
**Functions**:
|
||||
- `list_errors(opts)` — accepts keyword list: `[status: :unresolved, muted: false, search: "syntax error", limit: 50, offset: 0]`. Returns `%{errors: [...], total: n}`.
|
||||
- Filters: `status` → `where(status: ^status)`, `muted` → `where(muted: ^muted)` (boolean), `search` → `where(ilike(e.reason, ^"%#{search}%"))`
|
||||
- Count query: `Repo.aggregate/3` with same filters
|
||||
- List query: `order_by(desc: :last_occurrence_at)` + `limit/offset`
|
||||
- Computed fields `occurrence_count` and `first_occurrence_at` are omitted from the list endpoint to keep queries simple. They are computed only in the single-error endpoint. This avoids a correlated subquery per row in the list.
|
||||
- `get_error!(id)` — fetches single error, preloads all occurrences sorted `desc: inserted_at`, computes `occurrence_count` and `first_occurrence_at`.
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
mix test test/music_library/errors_test.exs
|
||||
```
|
||||
Write tests for: empty list, filtering by status, filtering by muted, search by reason substring, pagination (limit + offset), error not found raises, single error includes occurrence_count and occurrences.
|
||||
|
||||
---
|
||||
|
||||
#### Step 2: Create `MusicLibraryWeb.ErrorController`
|
||||
|
||||
**What**: New `lib/music_library_web/controllers/error_controller.ex` following `CollectionController` patterns.
|
||||
|
||||
**Actions**:
|
||||
- `index(conn, params)` — parses `status`, `muted`, `search`, `limit`, `offset` from query params; calls `Errs.list_errors(opts)`; renders `:index`.
|
||||
- `show(conn, %{"id" => id})` — fetches error by integer ID; renders `:show`.
|
||||
|
||||
Use `parse_int/2` helper (same pattern as `CollectionController`).
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
Write tests for: auth required (401 without Bearer token), list returns JSON structure, filter params applied, single error returns JSON with occurrences.
|
||||
|
||||
---
|
||||
|
||||
#### Step 3: Create `MusicLibraryWeb.ErrorJSON` view
|
||||
|
||||
**What**: New `lib/music_library_web/controllers/error_json.ex` following `CollectionJSON` patterns.
|
||||
|
||||
**Render functions**:
|
||||
- `index(%{errors: errors, total: total, limit: limit, offset: offset})` → `%{errors: [...], total: n, limit: n, offset: n}`
|
||||
- `show(%{error: error})` → single error with nested occurrences
|
||||
|
||||
**Serialization details**:
|
||||
- `id` → integer
|
||||
- `status` → atom-to-string (`"resolved"` / `"unresolved"`)
|
||||
- `fingerprint` → hex string (already stored as hex string in SQLite TEXT column)
|
||||
- `muted` → boolean
|
||||
- `last_occurrence_at`, `inserted_at`, `updated_at`, `first_occurrence_at` → ISO8601 strings
|
||||
- `context` → decoded map (already a map in Ecto, stored as JSON in SQLite)
|
||||
- `breadcrumbs` → array of strings
|
||||
- `stacktrace.lines` → array of maps `%{application, module, function, arity, file, line}`
|
||||
- `occurrence_count` → integer
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
Tests from Step 2 already validate the JSON structure via `json_response/2`. Verify serialization of all fields including nested stacktrace lines.
|
||||
|
||||
---
|
||||
|
||||
#### Step 4: Add routes to the router
|
||||
|
||||
**What**: Add two entries to the existing `scope "/api/v1"` block in `lib/music_library_web/router.ex`:
|
||||
|
||||
```elixir
|
||||
get "/errors", ErrorController, :index
|
||||
get "/errors/:id", ErrorController, :show
|
||||
```
|
||||
|
||||
Place after existing collection routes, before `assets` and `backup`.
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
mix phx.routes | grep errors
|
||||
```
|
||||
Should show the two new API routes under `/api/v1/errors`. Also run existing tests to ensure no route conflicts:
|
||||
```bash
|
||||
mix test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Step 5: Integration test and documentation update
|
||||
|
||||
**What**: Run the full test suite, update architecture docs.
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
mix test
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
All tests pass including existing tests. Update `docs/architecture.md`:
|
||||
- Add `Errors` context to the Contexts table
|
||||
- Add `ErrorController` to the Controllers table
|
||||
|
||||
---
|
||||
|
||||
### Production Changes
|
||||
|
||||
No manual production infrastructure changes required. The endpoints use:
|
||||
- Existing `error_tracker_errors` and `error_tracker_occurrences` tables (already migrated)
|
||||
- Existing `require_api_token` plug (already configured with `API_TOKEN` env var)
|
||||
- Existing `MusicLibrary.Repo` (already configured)
|
||||
- No new environment variables, no new services, no DNS changes, no firewall changes
|
||||
|
||||
---
|
||||
|
||||
### Test data seeding strategy
|
||||
|
||||
**`ErrorTracker.report/3` cannot be used to seed test data.** Two blockers:
|
||||
|
||||
1. **ErrorTracker is disabled in test.** `config/config.exs` sets `enabled: false`, with no override in `test.exs`. `report/3` checks `enabled?()` at the top and returns `:noop` when disabled — nothing is persisted.
|
||||
2. **ErrorTracker's supervision tree isn't started.** The application supervision tree only starts `ErrorTracker.ErrorNotifier`; the telemetry handlers and process-dictionary state that `report/3` depends on (`get_context()`, `get_breadcrumbs()`) are not initialized.
|
||||
|
||||
**Instead, seed via direct Ecto inserts using `MusicLibrary.Repo`** with ErrorTracker's own structs:
|
||||
|
||||
```elixir
|
||||
# In test/support/fixtures/errors_fixtures.ex
|
||||
|
||||
alias ErrorTracker.{Error, Occurrence, Stacktrace}
|
||||
|
||||
def error_fixture(attrs \\ []) do
|
||||
defaults = %{
|
||||
kind: "RuntimeError",
|
||||
reason: "Something went wrong",
|
||||
source_line: "lib/my_module.ex:42",
|
||||
source_function: "MyModule.do_thing/0",
|
||||
status: :unresolved,
|
||||
fingerprint: error_fingerprint(:runtime_error, "lib/my_module.ex:42", "MyModule.do_thing/0"),
|
||||
last_occurrence_at: DateTime.utc_now(),
|
||||
muted: false
|
||||
}
|
||||
|
||||
defaults
|
||||
|> Map.merge(Map.new(attrs))
|
||||
|> then(&MusicLibrary.Repo.insert!(struct!(Error, &1)))
|
||||
end
|
||||
|
||||
def occurrence_fixture(error, attrs \\ []) do
|
||||
defaults = %{
|
||||
reason: error.reason,
|
||||
context: %{user_id: 1},
|
||||
breadcrumbs: ["step 1"],
|
||||
stacktrace: %Stacktrace{
|
||||
lines: [
|
||||
%Stacktrace.Line{
|
||||
application: "music_library",
|
||||
module: "MyModule",
|
||||
function: "do_thing",
|
||||
arity: 0,
|
||||
file: "lib/my_module.ex",
|
||||
line: 42
|
||||
}
|
||||
]
|
||||
},
|
||||
error_id: error.id
|
||||
}
|
||||
|
||||
defaults
|
||||
|> Map.merge(Map.new(attrs))
|
||||
|> then(&MusicLibrary.Repo.insert!(struct!(Occurrence, &1)))
|
||||
end
|
||||
|
||||
defp error_fingerprint(kind, source_line, source_function) do
|
||||
[kind, source_line, source_function]
|
||||
|> Enum.join()
|
||||
|> then(&:crypto.hash(:sha256, &1))
|
||||
|> Base.encode16()
|
||||
end
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- `fingerprint` is stored as TEXT in SQLite (the hex string from `Base.encode16/1`), despite the Ecto schema declaring `:binary`. Use hex strings when inserting.
|
||||
- `status` is an `Ecto.Enum` — pass atoms (`:resolved` / `:unresolved`).
|
||||
- `muted` is INTEGER in SQLite (`0`/`1`), but the Ecto schema accepts booleans.
|
||||
- `stacktrace` is stored as JSON text (embedded schema serialized by Ecto). Pass a `%Stacktrace{}` struct and Ecto handles serialization.
|
||||
- `context` and `breadcrumbs` are stored as JSON text. Pass native Elixir maps and lists.
|
||||
- The fixture module follows the existing pattern (`RecordsFixtures`, `RecordSetsFixtures`, etc.) — helper functions that return inserted structs via `MusicLibrary.Repo`.
|
||||
- Include a variant fixture that produces multiple errors with different status/fingerprint/muted values to exercise filtering and pagination in tests.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
Reference in New Issue
Block a user