diff --git a/backlog/docs/doc-7 - Pi-access-to-production-errors.md b/backlog/docs/doc-7 - Pi-access-to-production-errors.md new file mode 100644 index 00000000..cd2184e5 --- /dev/null +++ b/backlog/docs/doc-7 - Pi-access-to-production-errors.md @@ -0,0 +1,163 @@ +--- +id: doc-7 +title: Pi access to production errors +type: other +created_date: '2026-05-04 08:07' +updated_date: '2026-05-04 08:13' +--- +# Implementation Routes: pi access to production errors + +## Problem + +The project captures production errors via the `error_tracker` Elixir dependency (stored in `MusicLibrary.TelemetryRepo` SQLite database). Errors can currently only be viewed through the built-in web dashboard at `/dev/errors` (behind login auth, only with `:monitoring_routes` enabled). There is no programmatic access — no API, no tooling, no pi extension. + +The goal is to enable pi (the coding agent) to fetch and browse production errors. This requires three layers: +1. A programmatic API to expose error data (behind auth) +2. Pi tools the LLM can call to fetch errors +3. A pi extension for interactive browsing + +## Routes Evaluated + +### Route A: JSON API endpoint on Phoenix (RECOMMENDED) + +**How it works:** +Add a new API controller (`MusicLibraryWeb.ErrorsController`) with two endpoints under `/api/v1/errors`: +- `GET /api/v1/errors` — list errors with optional filtering (status, muted, search) and pagination +- `GET /api/v1/errors/:id` — single error detail with its occurrences and stacktraces + +The controller queries the `error_tracker_errors` and `error_tracker_occurrences` tables via Ecto. Auth is handled by the existing `require_api_token` plug (Bearer token), which is already used by the `/api/v1` pipeline. + +Pi tools (`fetch_production_errors` and `fetch_production_error`) make HTTP requests to this API using `fetch()` or `pi.exec("curl", ...)`. The pi extension builds on these tools for an interactive TUI. + +**Pros:** +- Follows existing patterns exactly (see `CollectionController`, `require_api_token`) +- Clean separation of concerns: API layer, tool layer, extension layer +- Pi tools work remotely — no need for SSH or filesystem access to the production server +- Can be tested independently at each layer +- Auth is handled at the plug level, no new auth infrastructure needed +- API can be reused by other consumers (scripts, monitoring, future tooling) +- CORS not needed (pi tools call from the same origin or via CLI) +- The TelemetryRepo already exists and has the tables — no new database work + +**Cons:** +- Requires a server code change and deployment +- Adds two new routes + +**Changes needed:** +| Layer | Change | +|-------|--------| +| Server | New `ErrorsController` + `ErrorJSON` view or serializer | +| Server | 2 new routes in `router.ex` under `/api/v1` scope | +| Server | New context module (or inline queries) for error_tracker data | +| Pi | New tool registrations in a pi extension | +| Pi | Optional TUI extension for browsing | + +--- + +### Route B: Direct SQLite access from pi tools + +**How it works:** +Pi tools execute SQL queries directly against the TelemetryRepo SQLite database file. In development this is trivial (local file). In production, the pi session would need SSH access to the production server, or the SQLite file would need to be synced locally (e.g., via `mise run prod:backup`). + +No server-side API changes needed. The pi tools would use `pi.exec("sqlite3", ...)` or read the database file directly. + +**Pros:** +- Zero server code changes +- Immediate access to ALL data — no API shape limitations +- Can run complex ad-hoc queries without API changes + +**Cons:** +- **Production access requires SSH or filesystem access** — violates the project's existing API-based pattern for pi access (cf. `fetch_production_logs` which uses Coolify API, not SSH) +- No auth layer — tools have full read access to the entire database +- Tightly couples pi tools to the error_tracker schema — any migration could break tools +- No API reuse — cannot be consumed by other clients +- Each pi tool call requires a separate SQLite process (or a WS/SSE connection to keep a persistent session) +- Production SQLite is under concurrent write load from the app and Litestream — reading directly could impact performance +- Production database file may be locked or busy + +**Verdict: Rejected.** +The direct-access approach introduces deployment friction (SSH key management, filesystem access) that contradicts existing patterns. The app already has an authenticated API pipeline — extending it is simpler and safer. + +--- + +### Route C: Tidewave MCP / existing pi infrastructure only + +**How it works:** +Use the existing Tidewave MCP tools (`tidewave_execute_sql_query`) to query the error_tracker tables directly. This is available in dev but would need the Tidewave MCP server to be accessible in production (e.g., via SSH tunnel or a production-side MCP server). + +**Pros:** +- No new code at all — uses what's already there +- `tidewave_execute_sql_query` already understands the repo structure + +**Cons:** +- **Tidewave MCP server does not run in production** — it's a development-only tool +- Even if it did, running a dev tool against production is architecturally wrong +- No browsing UX, no filtering, no pagination — basic SQL results only +- Auth is missing — would need a separate mechanism + +**Verdict: Rejected.** +Tidewave is a development tool. Exposing it to production would be an architectural antipattern and a security concern. + +--- + +### Route D: Expose via Coolify API (like production logs) + +**How it works:** +Add an endpoint to the existing Coolify-like API pattern used by `fetch_production_logs`. This would require Coolify to expose error_tracker data, which it doesn't natively support. Could potentially parse production logs for error patterns, but that's unstructured and duplicative. + +**Pros:** +- Consistent with the existing `fetch_production_logs` pattern + +**Cons:** +- Coolify doesn't expose error_tracker data +- ErrorTracker already has structured data — parsing it from raw logs is backwards and lossy +- Would require Coolify API changes or custom Coolify plugin + +**Verdict: Rejected.** +Coolify is the wrong layer for structured error data. ErrorTracker already stores structured errors — adding Coolify as an intermediary adds complexity without benefit. + +--- + +## Recommendation + +**Route A: JSON API on Phoenix** is the clear winner. It: + +1. **Follows existing patterns** — the project already has an authenticated API pipeline with Bearer tokens (see `CollectionController`, `require_api_token`) +2. **Has precedent** — `fetch_production_logs` (ML-160) established the pattern of pi tools calling an authenticated API +3. **Is testable** — each layer (API, tool, extension) can be tested independently +4. **Is reusable** — the API can serve other consumers beyond pi +5. **Has minimal production impact** — new controller + routes, no infrastructure changes +6. **Respects auth boundaries** — the existing `require_api_token` plug handles authentication + +### Architecture + +``` +┌─────────────────────────────────────────────────┐ +│ pi session │ +│ │ +│ ┌──────────────┐ ┌──────────────────────────┐ │ +│ │ pi extension │ │ pi tools │ │ +│ │ (TUI browse) │──│ fetch_production_errors │ │ +│ │ │ │ fetch_production_error │ │ +│ └──────────────┘ └──────────┬───────────────┘ │ +└───────────────────────────────┼───────────────────┘ + │ HTTP (Bearer token) + ▼ +┌───────────────────────────────┼───────────────────┐ +│ Phoenix Server │ +│ │ +│ GET/POST /api/v1/errors ───► ErrorsController │ +│ GET /api/v1/errors/:id ───► ErrorsController │ +│ │ │ +│ ▼ │ +│ TelemetryRepo (SQLite) │ +│ error_tracker_errors │ +│ error_tracker_occurrences │ +└───────────────────────────────────────────────────┘ +``` + +### Subtask breakdown + +1. **ML-161: Expose production errors via JSON API** — Server-side: controller, routes, JSON views, context queries +2. **ML-162: Create pi tools to fetch errors** — Pi extension: `fetch_production_errors` and `fetch_production_error` tools via `pi.registerTool()` +3. **ML-163: Create pi extension for error browsing** — Pi extension: interactive TUI for browsing errors via `ctx.ui.custom()` diff --git a/backlog/tasks/ml-161 - pi-access-to-production-errors.md b/backlog/tasks/ml-161 - pi-access-to-production-errors.md new file mode 100644 index 00000000..4c37c187 --- /dev/null +++ b/backlog/tasks/ml-161 - pi-access-to-production-errors.md @@ -0,0 +1,59 @@ +--- +id: ML-161 +title: pi access to production errors +status: To Do +assignee: [] +created_date: '2026-05-04 08:06' +updated_date: '2026-05-04 08:10' +labels: + - pi + - api + - observability +dependencies: + - ML-162 + - ML-163 + - ML-164 +references: + - 'backlog://document/doc-7' +priority: medium +ordinal: 4000 +--- + +## Description + + +Errors in production are captured via the `error_tracker` Elixir dependency and accessed via a dedicated dashboard at `/dev/errors`. There is no built-in tooling or endpoint to pull production errors programmatically in a pi session. This task covers the investigation and implementation of the best approach to expose production error data to pi, broken into three subtasks: + +1. **Expose production errors via a programmatic API (behind auth)** — Add a JSON API endpoint under `/api/v1/errors` that lists errors and shows individual error details from the error_tracker tables. +2. **Create pi tools to pull errors** — Implement `fetch_production_errors` (list with filtering/pagination) and `fetch_production_error` (single error detail) as pi tools. +3. **Create a pi extension to browse errors** — Build an interactive TUI browsing experience via a pi extension that uses the tools from subtask 2. + +### Error data attributes + +These are the attributes that subtask implementations must treat as the canonical data model: + +**Error (from `error_tracker_errors` table):** +- `id` — UUID, unique error identifier +- `kind` — string, e.g. "error", "throw", "exit" +- `reason` — string, the error message/reason +- `source_line` — string, e.g. "lib/foo.ex:42" +- `source_function` — string, e.g. "MusicLibrary.Foo.bar/2" +- `status` — atom: `:resolved` | `:unresolved` +- `fingerprint` — hex string, deterministic hash of (kind, source_line, source_function) +- `last_occurrence_at` — UTC datetime (microsecond precision) +- `muted` — boolean +- `inserted_at` / `updated_at` — UTC datetime + +**Occurrence (from `error_tracker_occurrences` table):** +- `id` — UUID +- `reason` — string, error reason at time of occurrence +- `context` — map, includes `live_view.view`, `request.path`, etc. +- `breadcrumbs` — array of strings +- `stacktrace` — embedded struct with `lines` array (each having: `application`, `module`, `function`, `arity`, `file`, `line`) +- `error_id` — FK to error +- `inserted_at` — UTC datetime + +**Aggregated/per-error metadata (computed, not stored):** +- `occurrence_count` — total occurrences for this error +- `first_occurrence_at` — earliest occurrence timestamp + diff --git a/backlog/tasks/ml-162 - Expose-production-errors-via-JSON-API-endpoint.md b/backlog/tasks/ml-162 - Expose-production-errors-via-JSON-API-endpoint.md new file mode 100644 index 00000000..8222671f --- /dev/null +++ b/backlog/tasks/ml-162 - Expose-production-errors-via-JSON-API-endpoint.md @@ -0,0 +1,47 @@ +--- +id: ML-162 +title: Expose production errors via JSON API endpoint +status: To Do +assignee: [] +created_date: '2026-05-04 08:08' +updated_date: '2026-05-04 08:10' +labels: + - api + - elixir + - backend +dependencies: [] +parent_task_id: DRAFT-1 +priority: medium +ordinal: 7000 +--- + +## Description + + +Add an API controller and routes under `/api/v1/errors` to expose ErrorTracker data as JSON, behind the existing Bearer token auth. + +This subtask covers the server-side work only: controller, JSON serialization, context queries, and routes. The pi tooling and extensions are covered by separate subtasks. + +### API design + +**`GET /api/v1/errors`** — List errors +- Query params: `status` (resolved/unresolved), `muted` (true/false), `search` (substring match on reason), `limit` (default 50), `offset` (default 0) +- Returns: `{ errors: [...], total: n, limit: n, offset: n }` +- Each error includes: id, kind, reason, source_line, source_function, status, fingerprint, last_occurrence_at, muted, inserted_at, updated_at, occurrence_count, first_occurrence_at + +**`GET /api/v1/errors/:id`** — Single error detail +- Returns the error with all its occurrences (with stacktraces), sorted by inserted_at desc +- Each occurrence includes: id, reason, context, breadcrumbs, stacktrace (lines), inserted_at + +### Data attributes (canonical — shared with all subtasks) + +**Error fields:** id, kind, reason, source_line, source_function, status, fingerprint, last_occurrence_at, muted, inserted_at, updated_at +**Occurrence fields:** id, reason, context, breadcrumbs, stacktrace (with lines), error_id, inserted_at +**Computed:** occurrence_count, first_occurrence_at + +### Dependencies + +- Uses `MusicLibrary.TelemetryRepo` (already exists) +- Uses the `error_tracker_errors` and `error_tracker_occurrences` tables (already exist) +- Auth via existing `require_api_token` plug (already in use by `/api/v1` pipeline) + diff --git a/backlog/tasks/ml-163 - Create-pi-tools-to-fetch-production-errors.md b/backlog/tasks/ml-163 - Create-pi-tools-to-fetch-production-errors.md new file mode 100644 index 00000000..ae9aaa81 --- /dev/null +++ b/backlog/tasks/ml-163 - Create-pi-tools-to-fetch-production-errors.md @@ -0,0 +1,51 @@ +--- +id: ML-163 +title: Create pi tools to fetch production errors +status: To Do +assignee: [] +created_date: '2026-05-04 08:08' +updated_date: '2026-05-04 08:15' +labels: + - pi + - tool + - typescript +dependencies: [] +parent_task_id: DRAFT-1 +priority: medium +ordinal: 5000 +--- + +## Description + + +Create `fetch_production_errors` and `fetch_production_error` pi tools that call the `/api/v1/errors` JSON API endpoint. + +These tools are registered via `pi.registerTool()` in a pi extension. They make authenticated HTTP requests to the production server using the same HURL variable pattern established by `fetch_production_logs` (ML-160). + +### Tools + +**`fetch_production_errors`** — List/filter production errors +- Parameters: `status` (optional: "resolved" | "unresolved"), `muted` (optional: boolean), `search` (optional: string, substring match on reason), `limit` (optional: number, default 50), `offset` (optional: number, default 0) +- Calls `GET /api/v1/errors` with query params +- Returns: formatted list of errors with counts and timestamps +- Truncates output with `truncateTail` (50KB / 2000 lines) + +**`fetch_production_error`** — Get a single error with full details +- Parameters: `id` (required: UUID string) +- Calls `GET /api/v1/errors/:id` +- Returns: error details with all occurrences, stacktraces, and context +- Truncates output with `truncateTail` + +### Tool guidelines (promptGuidelines) +- Use fetch_production_errors when investigating what errors are occurring in production +- Use fetch_production_error when you need full details on a specific error, including stacktraces and context +- Start with a small limit (e.g., 20) and filter by status or search before fetching large result sets + +### Auth + +Uses the same HURL variable pattern as `fetch_production_logs`: +- `PI_API_TOKEN` — Bearer token for API auth +- `PI_SERVICE_FQDN_WEB` — Production domain + +They need to be configured in the pi environment as local secrets. + diff --git a/backlog/tasks/ml-164 - Create-pi-extension-for-interactive-error-browsing.md b/backlog/tasks/ml-164 - Create-pi-extension-for-interactive-error-browsing.md new file mode 100644 index 00000000..311200d7 --- /dev/null +++ b/backlog/tasks/ml-164 - Create-pi-extension-for-interactive-error-browsing.md @@ -0,0 +1,79 @@ +--- +id: ML-164 +title: Create pi extension for interactive error browsing +status: To Do +assignee: [] +created_date: '2026-05-04 08:08' +updated_date: '2026-05-04 08:10' +labels: + - pi + - extension + - typescript + - tui +dependencies: [] +parent_task_id: DRAFT-1 +priority: medium +ordinal: 6000 +--- + +## Description + + +Build a pi extension that provides an interactive TUI for browsing production errors, using the `fetch_production_errors` and `fetch_production_error` tools from the parent task. + +This extension gives the user (and LLM) a browseable interface for production errors, accessible via a slash command like `/prod-errors`. + +### Extension features + +1. **`/prod-errors` command** — Opens an interactive TUI using `ctx.ui.custom()`: + - Lists recent errors (unresolved first, then by last_occurrence_at desc) + - Shows key metadata: reason (truncated), kind, source location, occurrence count, last seen, status badge, muted indicator + - Keyboard navigation: up/down to select, Enter to view details, Escape to close + - Filter toggle: show/hide resolved, show/hide muted (keyboard shortcuts) + - Pagination: load more errors as user scrolls + +2. **Error detail view** (Enter on an error): + - Full reason text + - Source location with link-like formatting + - Status, muted, fingerprint + - Timeline of occurrences with timestamps + - Stacktrace display (collapsible per occurrence) + - Context display (request path, LiveView, etc.) + - Breadcrumbs if present + +3. **UI patterns** (consistent with existing `/prod-logs` extension): + - Uses `pi.exec("curl", ...)` to call the API + - Reads credentials from `resolveVar()` (same pattern as prod-logs) + - Keyboard shortcuts displayed as hints + - Theme-aware rendering via `ctx.ui.theme` + +### TUI component design + +``` +╔══════════════════════════════════════════════════╗ +║ Production Errors [12 errors] ║ +╠══════════════════════════════════════════════════╣ +║ ▶ [UNRESOLVED] FunctionClauseError ║ +║ MusicLibrary.Foo.bar/2 lib/foo.ex:42 ║ +║ 23 occurrences · last seen 2h ago ║ +║ ──────────────────────────────────────────────── ║ +║ [RESOLVED] MatchError ║ +║ MusicLibrary.Baz.qux/1 lib/baz.ex:15 ║ +║ 5 occurrences · last seen 3d ago ║ +║ ──────────────────────────────────────────────── ║ +║ [MUTED] KeyError (key :foo not found) ║ +║ MusicLibrary.Other.func/3 lib/other.ex:99 ║ +║ 1 occurrence · last seen 7d ago ║ +║ ──────────────────────────────────────────────── ║ +║ ║ +║ ↑↓ navigate ↵ details r toggle resolved ║ +║ m toggle muted q quit ║ +╚══════════════════════════════════════════════════╝ +``` + +### File location + +`.pi/extensions/prod-errors/index.ts` (new extension, separate from `prod-logs`) + +The prod-logs extension already provides the `resolveVar` pattern and `fetchLogs` function. This extension follows the same conventions but for error_tracker data. +