cloud/music_library
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ML-184: presto testing harness (plan)

Browse Source
This commit is contained in:
Claudio Ortolina
2026-05-15 09:50:23 +01:00
parent 7338825606
commit 2ec19f3a88
2 changed files with 529 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backlog/docs/doc-23 - Research-Presto-test-harness-implementation-routes.md/doc-23 - Research-Presto-test-harness-implementation-routes.md
+261
View File
@@ -0,0 +1,261 @@
---
id: doc-23
title: "Research: Presto test harness implementation routes"
type: specification
created_date: "2026-05-15 07:22"
tags:
- presto
- testing
- micropython
- research
---
# Research: Presto test harness implementation routes
Status: **Awaiting feedback** — 3 routes presented below.
## Current state
- `presto/main.py` is a 2717-line monolithic MicroPython app for the Pimoroni Presto (480×480 touch display).
- Hardware init happens at module level: `presto = Presto(full_res=True)` creates the display, touch, and WiFi objects on import.
- `main()` runs an infinite `while True` event loop with touch dispatch, display sleep, and WiFi management.
- `main()` is called unconditionally at the bottom of the file — importing the module always starts the app.
- `pimoroni-emulator` v0.5.0 is already configured in `presto/mise.toml` with an `emulator` task.
- The emulator's `DeviceTest` class provides: headless display, `run_frames()`, `touch()`, `click_button()`, `screenshot()`, `assert_display_matches()`.
- The emulator installs mock modules for `presto`, `picographics`, `network`, `jpegdec`, etc. before running the app, so hardware calls work in the test environment.
## Key challenge
The current `main.py` architecture couples three concerns at module level:
1. **Hardware initialization** — creates Presto/display/touch on import
2. **State + rendering** — all functions are module-level, operating on globals
3. **Event loop** — `main()` is called unconditionally at the bottom
For a test harness to work, we need to decouple at least #3 so that the event loop doesn't run when we import the module for testing. The emulator's mock system handles #1 (hardware mocks), and the rendering functions (#2) are already callable once imported.
---
## Route A: Minimal guard — add testing flag to main.py
**Approach:** Add a single `if __name__ == "__main__":` guard (or environment-variable check) around the `main()` call at the bottom of `main.py`. Tests import the module as a library, set up app state, and call render functions directly. The emulator's `DeviceTest` provides the headless display.
### What changes in main.py
```python
# Bottom of main.py — only change needed:
if __name__ == "__main__":
main()
```
When the emulator's `DeviceTest.setUp()` runs the app via `runpy.run_path()`, `__name__` is set to `"__main__"`, so the guard would NOT prevent `main()` from running. We need an alternative signal.
**Better guard:** Use an environment variable:
```python
# Bottom of main.py:
import os
if os.environ.get("PRESTO_TEST_MODE") != "1":
main()
```
This works because the emulator runs on CPython where `os.environ` is available. The test harness sets `PRESTO_TEST_MODE=1` before importing.
### Test structure
```
presto/
tests/
__init__.py
conftest.py # Sets PRESTO_TEST_MODE=1, imports main, sets up emulator
test_screens.py # Smoke tests: render each screen, capture screenshot
test_touch.py # Touch interaction tests
fixtures/ # Reference screenshots for visual regression
```
### Example test
```python
import os
os.environ["PRESTO_TEST_MODE"] = "1"
from emulator.testing import DeviceTest
import main # imports app module without running main()
class TestScreens(DeviceTest):
device = "presto"
def test_home_screen(self):
app = main.AppState()
main.init_display()
main.draw_home_screen(app)
self.assert_display_matches("fixtures/home_screen.png")
```
### Pros
- **Smallest change to main.py** — one guard at the bottom, zero structural changes.
- **Tests actual render code** — calls the exact same functions that run on the device.
- **Fast** — no WiFi/NTP wait, no touch simulation delays.
- **Easy to extend** — each new test is just: set up state → call render → screenshot.
- **CI-ready** — headless execution works in any Python 3 environment with `pimoroni-emulator` pip installed.
### Cons
- Testing concern leaks into app code (the guard).
- `main.py` remains monolithic — doesn't force better architecture.
- All module-level globals still exist on import; state isolation between tests requires care.
---
## Route B: Extract rendering and state into separate modules
**Approach:** Refactor `main.py` into a package structure. Rendering functions move to `presto/lib/display.py`, state classes move to `presto/lib/state.py`, API/networking stays in `presto/lib/api.py`. `main.py` becomes a thin entry point. Tests import from `lib/*` modules directly.
### Package structure
```
presto/
main.py # Thin entry point: init hardware, run event loop
lib/
__init__.py
state.py # AppState, DayListState, SearchState, DetailState, TouchState
display.py # All draw_* functions, pen init, helpers
api.py # fetch_records, fetch_search_results, fetch_thumbnail
network.py # connect_wifi, sync_time, wifi_connected
config.py # Layout constants, colors, timing
layout.py # px(), text helpers, wrapping
tests/
__init__.py
conftest.py
test_screens.py
test_state.py
fixtures/
```
### What changes in main.py
`main.py` shrinks to ~50 lines:
```python
"""Presto Music Library — application entry point."""
from presto import Presto
from lib.state import AppState
from lib.display import init_display, draw_home_screen, draw_status, ...
from lib.network import connect_wifi, sync_time, set_today
from lib.api import ...
# Hardware init
presto = Presto(full_res=True)
display = presto.display
touch = presto.touch
WIDTH, HEIGHT = display.get_bounds()
def main():
# ... event loop using imported functions ...
main()
```
### Test structure (same as Route A but cleaner)
```python
from lib.state import AppState
from lib.display import init_display, draw_home_screen, draw_month_view
def test_home_screen():
app = AppState()
init_display()
draw_home_screen(app)
# ...
```
### Pros
- **Clean separation** — rendering, state, and I/O are in distinct modules.
- **No test code in production files** — no guard needed.
- **Enables unit testing** — can test state transitions, text wrapping, layout math without the emulator.
- **Reusable** — `lib/display.py` could be shared with other Presto apps.
- **Aligns with project conventions** — matches the Elixir-side pattern of separating contexts, schemas, and workers.
### Cons
- **Large diff** — touches every function in main.py (moving import paths, updating globals).
- **Risk of regressions** — moving 2717 lines of code across files can introduce subtle bugs.
- **Must test on physical device after refactoring** — cannot rely on emulator alone.
- **Module-level globals are tricky** — `display`, `_pen_bg`, etc. need to be accessible from `lib/display.py` but initialized by hardware code.
---
## Route C: End-to-end emulator tests with touch simulation (no code changes)
**Approach:** Write `DeviceTest` subclasses that run `main.py` as-is in the emulator. Use touch simulation and frame waiting to navigate through the app. Capture screenshots at each state.
### Test structure
```python
from emulator.testing import DeviceTest
class TestScreensE2E(DeviceTest):
device = "presto"
app = "main.py" # Runs the full app
def test_can_reach_home_screen(self):
self.run_frames(30) # Wait for boot, WiFi fail, status screens
self.screenshot("home_screen.png")
def test_can_navigate_to_month_view(self):
self.run_frames(30)
# Tap "Today's Records" button (coordinates from layout constants)
self.touch(240, 195) # HOME_BUTTON2_Y center
self.run_frames(10)
self.screenshot("month_view.png")
```
### Pros
- **Zero code changes** — `main.py` stays exactly as-is.
- **Tests the full app** — WiFi, NTP, API calls all run (or fail) as they would on device.
- **Catches integration bugs** — if a refactoring breaks the event loop, these tests catch it.
### Cons
- **WiFi blocks** — `connect_wifi()` will try to connect and fail (timeout: 30 seconds). Need `--no-wifi` or mock the network module.
- **NTP blocks** — `sync_time()` calls `ntptime.settime()` which tries to reach an NTP server.
- **API calls fail** — all `urequests` calls will get connection errors.
- **Fragile timing** — `run_frames(30)` is guesswork; the app may be in an unexpected state.
- **Slow** — each test takes seconds to minutes due to boot sequence and timeouts.
- **Hard to test specific states** — reaching deep states requires complex touch sequences.
- **No isolation** — tests share emulator state; ordering matters.
### Workarounds for Route C
- Create mock `secrets.py` on the test path with dummy credentials (WiFi will still fail but won't crash).
- Monkey-patch `network.WLAN` to return "connected" before tests run.
- Use `--no-wifi` CLI flag (but this isn't exposed through `DeviceTest` — would need to set `_emulator_state` directly).
- These workarounds add complexity and move us toward Route A/B anyway.
---
## Recommendation
**Route A (minimal guard) is the recommended starting point** because:
1. It gets us rendering smoke tests with the smallest possible change to `main.py` (2 lines).
2. It validates that all screens render without crashing — the stated minimum requirement.
3. It's fast to implement (a few hours) and easy to extend later.
4. It can evolve into Route B later — the test files written for Route A still work after refactoring.
If the team prefers to invest in architecture now, **Route B** is the better long-term solution and aligns with the project's Elixir-side conventions of separating concerns into modules. It just carries more implementation risk.
**Route C** is not recommended as a starting point. It adds complexity without proportional benefit. It could be valuable later as a complement to Route A/B for testing the full boot sequence and touch-driven navigation, but only after the blocking network calls are addressed.
---
## Open questions for feedback
1. **Route A or B?** Minimal guard (2-line change) or full extraction into `lib/` modules?
2. **What to test first?** Smoke-test all screens, or focus on specific views (month, day, detail)?
3. **Visual regression or crash-only?** Should screenshots be compared against reference images (`assert_display_matches`), or is "renders without exception" sufficient for now?
4. **CI integration?** Should the tests run in CI? The `pimoroni-emulator` dependency needs to be installed in CI — is that acceptable?
5. **secrets.py in tests?** The app reads credentials from `secrets.py`. For tests, should we create a test `secrets.py` with dummy values, or refactor to make credentials injectable?
backlog/tasks/ml-184 - test-harness-for-presto-application.md
+268
View File
@@ -0,0 +1,268 @@
---
id: ML-184
title: test harness for presto application
status: To Do
assignee: []
created_date: "2026-05-15 07:18"
updated_date: "2026-05-15 08:50"
labels:
- presto
- ready
dependencies: []
references:
- "https://github.com/iksaif/pimoroni-emu/blob/main/README.md#testing"
- presto/mise.toml
- presto/main.py
- doc-23 - Research-Presto-test-harness-implementation-routes.md
priority: medium
ordinal: 18000
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Build a test harness for the Presto application so that it's possible at the very minimum to render screens and check them. The pimoroni-emulator dependency is already configured in presto/mise.toml and provides a DeviceTest testing framework with headless execution, screenshot capture, button simulation, and touch input. Reference: https://github.com/iksaif/pimoroni-emu/blob/main/README.md#testing
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [ ] #1 main.py can be imported without starting the event loop when PRESTO_TEST_MODE=1 is set
- [ ] #2 Deploying to physical Presto still works (mise run presto)
- [ ] #3 mise run emulator still opens the emulator window
- [ ] #4 All 7 smoke tests pass (home, month, day-empty, day-error, record-detail, search-input, search-results)
- [ ] #5 mise run test executes all tests and reports results
- [ ] #6 presto/README.md includes a Testing section with usage instructions
- [ ] #7 presto/AGENTS.md mentions the test harness
<!-- AC:END -->
## Implementation Plan
<!-- SECTION:PLAN:BEGIN -->
## Implementation Plan: Route A — Minimal guard + emulator smoke tests
### 1. Objective alignment
The goal is a test harness that can, at minimum, render each Presto screen and verify it doesn't crash. Route A achieves this by:
1. Adding a single guard around the `main()` call so the module can be imported without starting the event loop.
2. Writing `DeviceTest`-based tests that import the app module, set up `AppState`, call render functions, and capture screenshots.
3. Configuring a `mise` task (`presto.test`) so tests are runnable with one command.
This directly maps: **problem** = no way to verify screens render correctly without deploying to physical hardware → **solution** = importable module + emulator-based smoke tests.
### 2. Simplicity and alternatives considered
**Chosen: Route A (minimal guard).** The only change to `main.py` is wrapping `main()` in an environment-variable guard. The test harness lives entirely in new files under `presto/tests/`.
**No test `secrets.py` is needed** — `secrets` is imported lazily inside `connect_wifi()` (line 516), not at module level. As long as the guard prevents `main()` from running, `connect_wifi()` is never called and the import never happens. The module-level `from presto import Presto` and `Presto(full_res=True)` call are handled by the emulator's mock system.
**Alternatives evaluated and deferred:**
- **Route B (extract lib/ modules):** Cleaner architecture but a 2717-line refactoring with regression risk. Deferred — can be done later as a follow-up, and all tests written for Route A will still pass.
- **Route C (full E2E with touch simulation):** Zero code changes but blocked by WiFi/NTP timeouts (30s each), fragile frame timing, and complex touch sequences. Not viable as a starting point.
**Justification:** Route A is the simplest change that achieves the stated minimum ("render screens and check them") with the least risk. It amortizes into Route B if desired.
**Decision on visual regression:** Smoke tests for this task will verify "renders without exception" only — no `assert_display_matches` with reference fixtures. Screenshots are saved for manual inspection. Reference-based visual regression can be added in a follow-up task once the harness is stable and a baseline is established.
### 3. Completeness and sequencing
#### Step 1: Add test-mode guard to main.py
- Add `import os` at the top of imports (after `import gc` on line 31).
- Replace the bare `main()` call at the bottom (line 2716) with:
```python
if os.environ.get("PRESTO_TEST_MODE") != "1":
main()
```
- Verify the app still deploys and runs on the physical Presto: `mise run presto`.
- Verify the emulator task still works: `cd presto && mise run emulator`.
**Verification:** Deploy to physical device and confirm the app boots normally. Run `mise run emulator` and confirm the emulator window opens with the splash screen. Confirm `PRESTO_TEST_MODE=1` is NOT set in normal `mise run emulator` (it should not be).
#### Step 2: Create test infrastructure
- Create `presto/tests/` directory.
- Create `presto/tests/__init__.py` (empty).
- Create `presto/tests/conftest.py`:
- Set `os.environ["PRESTO_TEST_MODE"] = "1"` before importing `main`.
- `import main` — this imports the app module without starting the event loop.
- Provide a pytest fixture that calls `main.init_display()` once per test class (pen creation depends on `display` which is set up at module level by the emulator mock).
- Provide a helper `make_mock_record()` that returns a dict with all required fields and cache keys pre-set so no draw function attempts a network call (see Step 3 for the exact shape).
- Monkey-patch `main.fetch_thumbnail` to raise `RuntimeError("Network call in smoke test")` so any accidental network access is caught immediately.
- Create `presto/tests/fixtures/` directory for saved screenshots (not reference images — used for manual inspection).
**Verification:** `cd presto && PRESTO_TEST_MODE=1 python3 -c "import main; print('Import OK, AppState:', main.AppState)"` succeeds without starting the event loop and without importing `secrets`.
#### Step 3: Write smoke tests for all 7 screens
Create `presto/tests/test_screens.py` with one test per screen. Every test follows the pattern: `init_display()`, create `AppState()`, set state fields, call draw function, assert no exception, optionally save screenshot.
**Mock record shape** — records passed to draw functions must include ALL of these keys to avoid triggering thumbnail fetches, text re-measurement, or attribute errors:
```python
{
"id": "rec-1",
"selected_release_id": "release-uuid-1",
"title": "Kind of Blue",
"artists": ["Miles Davis"],
"format": "Vinyl",
"release_date": "1959-08-17",
"genres": ["Jazz", "Modal"],
"record_type": "LP",
"purchased_at": "2024-03-15",
# Pre-computed display strings (set by prepare_record_list / _measure_detail_content)
"_display_title": "Kind of Blue",
"_display_artists": "Miles Davis",
"_display_meta": "Vinyl | 1959",
"_display_title_lines": [("Kind of Blue", 120)],
"_display_artist_lines": [("Miles Davis", 120)],
"_display_meta_lines": [("Vinyl | 1959", 120)],
# Thumbnail cache keys — set to None so draw functions use placeholders
"_thumb_url": "",
"_thumb_data": None,
"_thumb_failed": True,
# Detail-view cache keys
"_detail_thumb_data": None,
"_detail_thumb_failed": True,
"_detail_title_lines": [("Kind of Blue", 22)],
"_detail_artist_lines": [("Miles Davis", 15)],
"_detail_genre_lines": [("Jazz, Modal", 15)],
"_detail_meta_lines": [("LP | Vinyl | 1959", 15)],
"_detail_purchased_lines": [("Purchased: 2024-03-15", 15)],
# Row height (computed by prepare_record_list)
"_row_height": 88,
}
```
The conftest.py `make_mock_record()` helper constructs this dict so individual tests don't repeat it.
**Individual tests:**
1. **test_home_screen_renders** — `init_display()`, create `AppState()`, call `draw_home_screen(app)`, assert no exception, take screenshot.
2. **test_month_view_renders** — set `app.screen = STATE_MONTH`, `app.view_year = 2026`, `app.view_month = 5`, call `draw_month_view(app)`, screenshot.
3. **test_day_view_empty_renders** — set `app.screen = STATE_DAY`, `app.day.records = []`, `app.day.error = False`, `app.view_year = 2026`, `app.view_month = 5`, `app.day.selected_day = 15`, call `draw_day_view(app)`, screenshot.
4. **test_day_view_error_renders** — set `app.day.error = True` (records can be anything), call `draw_day_view(app)`, screenshot.
5. **test_record_detail_renders** — set `app.screen = STATE_RECORD`, populate `app.day.records` with one mock record, set `app.detail.selected_index = 0`, `app.detail.source_screen = STATE_DAY` (required by `draw_record_detail`), call `draw_record_detail(app)`, screenshot.
6. **test_search_input_renders** — set `app.screen = STATE_SEARCH_INPUT`, call `draw_search_input(app)`, screenshot.
7. **test_search_results_renders** — set `app.screen = STATE_SEARCH_RESULTS`, populate `app.search.results` with mock records, pre-compute `app.search.content_height` (or call `prepare_record_list` which won't trigger network calls because `_thumb_failed=True` and `_thumb_data=None`), call `draw_search_results(app)`, screenshot.
**Verification:** All 7 tests pass with `pytest presto/tests/ -v`. The `fetch_thumbnail` monkey-patch confirms no network calls were attempted. Manually inspect generated screenshots to confirm they look correct (no blank screens, text is visible, layouts are intact).
#### Step 4: Configure mise task for testing
- Add a `[tasks.test]` section to `presto/mise.toml`:
```toml
[tasks.test]
run = "pytest tests/ -v"
dir = "{{cwd}}"
```
- Test it: `cd presto && mise run test`.
**Verification:** `mise run test` runs all tests and reports results with the standard pytest summary (7 passed).
#### Step 5: Documentation
- Update `presto/README.md` with a "Testing" section explaining:
- Prerequisites: `pimoroni-emulator` (already installed via `mise`), Python 3, `pytest`.
- How to run tests: `mise run test` (in `presto/` directory) or `cd presto && mise run test`.
- What the tests cover: smoke tests for each screen (home, month, day-empty, day-error, record-detail, search-input, search-results).
- How to add new tests: create a test function in `presto/tests/test_screens.py` following the existing pattern (init display, set up state, call draw function, assert no crash).
- That tests use mock data and do not make network requests.
- That screenshots are saved to `presto/tests/fixtures/` for manual inspection.
- Update `presto/AGENTS.md`:
- Add a new section `## Testing` after the "Deployment And Verification" section:
```markdown
## Testing
The test harness lives in `presto/tests/` and uses the `pimoroni-emulator`'s
`DeviceTest` framework for headless smoke testing.
- Run tests: `mise run test`
- Tests import `main.py` without starting the event loop (guarded by `PRESTO_TEST_MODE=1`).
- Smoke tests cover every screen state. When adding a new screen, add a corresponding test in `test_screens.py`.
- All tests use mock data. No network calls are made during test execution.
- Screenshots are saved to `presto/tests/fixtures/` for manual visual inspection.
```
### 4. Verifiability
Each step has explicit verification:
| Step | Verification |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| 1. Guard | Deploy to physical Presto, app boots normally. `mise run emulator` opens window. `PRESTO_TEST_MODE` is not set in normal emulator runs. |
| 2. Infrastructure | `PRESTO_TEST_MODE=1 python3 -c "import main; print(main.AppState)"` succeeds without event loop. No `secrets` import occurs. |
| 3. Smoke tests | `pytest presto/tests/ -v` — all 7 tests pass. `fetch_thumbnail` monkey-patch confirms zero network calls. Screenshots visually inspected. |
| 4. Mise task | `cd presto && mise run test` runs all tests and reports 7 passed. |
| 5. Docs | README has "Testing" section. AGENTS.md has "Testing" section with conventions. |
### 5. Architecture impact analysis
| Touchpoint | Impact |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `presto/main.py` | **Minimal:** `import os` added to imports (line 31); `main()` call (line 2716) wrapped in `os.environ` guard. No function signatures, globals, layout constants, or other logic changes. |
| `presto/mise.toml` | **Addition:** new `[tasks.test]` task. Existing `emulator` and `presto` tasks unchanged. |
| `presto/tests/` | **New directory:** `__init__.py`, `conftest.py`, `test_screens.py`, `fixtures/`. No impact on existing code. |
| `presto/README.md` | **Addition:** "Testing" section. |
| `presto/AGENTS.md` | **Addition:** "Testing" section with conventions. |
| Physical Presto device | **No impact.** The guard is a no-op when `PRESTO_TEST_MODE` is unset (normal operation). |
| Emulator | **No impact.** The `emulator` task runs `main.py` normally; the env var is not set. |
**No migration or deprecation needed.** All changes are additive except the `main()` guard.
### 6. Performance profile
- **Test execution time:** Each smoke test renders one screen and optionally saves a PNG screenshot. Screen rendering is fast (tens of ms in the headless emulator) but the dominant cost is `DeviceTest.setUp()` emulator startup (~500ms-1s per test class). All 7 tests in a single class should complete in under 5 seconds. If each test gets its own class (worst case), total time could be ~7-10 seconds.
- **Database queries:** None — tests use mock data and don't hit any API or database.
- **N+1 risks:** None — no database or API calls in smoke tests. The `fetch_thumbnail` monkey-patch guarantees this.
- **Memory footprint:** The emulator's headless display holds a single 480×480 framebuffer (~900KB uncompressed). Negligible for test environments.
- **No latency/throughput concerns** — tests are offline and single-threaded.
### 7. Benchmarking requirements
No ongoing benchmarks are needed for this change. The test harness itself is the verification mechanism.
**One-off validation:** After smoke tests pass, run `time pytest presto/tests/ -v` to record baseline execution time. If future changes cause tests to slow down significantly (>5x), investigate.
### 8. Cost profile
**Zero cost.** All testing uses:
- `pimoroni-emulator` (free, open-source, MIT licensed)
- Local headless rendering (no API calls, no cloud resources)
- Mock data in tests (no network requests)
No paid resources are consumed during test execution.
### 9. Production infrastructure steps
**No production changes required.** The test harness runs entirely locally and does not affect:
- The deployed Phoenix application
- The physical Presto device
- API endpoints or databases
- Coolify, Litestream, or any production service
The only "deployment" consideration is that `main.py` continues to work on the physical Presto after the guard is added — verified in Step 1.
### 10. Documentation updates
| File | Change |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `presto/README.md` | Add "Testing" section: prerequisites, how to run, what's covered, how to add tests, mock data note, screenshot location. |
| `presto/AGENTS.md` | Add `## Testing` section after "Deployment And Verification": how to run tests, architecture (guard + DeviceTest), smoke test conventions, mock data guarantee. |
| `docs/architecture.md` | No changes needed — Presto is not covered in the main architecture doc (it's a separate MicroPython app). |
| `docs/project-conventions.md` | No changes needed — existing testing conventions cover Elixir only. |
<!-- SECTION:PLAN:END -->