Add research docs about nerves deployment

2026-05-04 16:12:43 +01:00
parent eb8d4776a5
commit 7efe277fba
3 changed files with 841 additions and 0 deletions
@@ -0,0 +1,222 @@
 ---
 id: doc-10
 title: 'Deepseek v4 Pro, xhigh analysis'
 type: other
 created_date: '2026-05-04 15:11'
 ---
 # Nerves Deployment Feasibility Analysis for Music Library
 > Research report — do not implement. May 2026.
 ## Overview
 This report assesses the viability of deploying the Music Library application (a
 Phoenix LiveView app backed by SQLite + Litestream) as a Nerves firmware image.
 Three areas were identified as potential challenges:
 1.  **Libraries with native extensions (NIFs)** — do they offer precompiled ARM
    binaries or can they be cross-compiled?
 2.  **SQLite extensions** — `vec0` (sqlite-vec) and `unicode` — already support
    Linux/ARM on paper, but are they tested on Nerves targets?
 3.  **Data replication** — the production instance uses Litestream for backup. How
    would a Nerves device replicate data, and what happens with write conflicts?
 Each area is analysed below, followed by a summary of risks and recommended next
 steps.
 ---
 ## 1. Native-extensions audit
 Every dependency that ships a NIF or binary artefact was reviewed for ARM /
 Nerves compatibility.  Dev‑only tools (esbuild, tailwind, credo, etc.) are
 excluded — they never reach the firmware.
 | Dependency | Purpose | NIF type | Precompiled ARM? | Nerves feasibility |
 |---|---|---|---|---|
 | **exqlite** → `ecto_sqlite3` | SQLite driver | C NIF (`cc_precompiler`) | ✅ `aarch64-linux-gnu/musl` (no `armv7-*`) | **🟢 Good** — `force_build: true` compiles from source inside the Nerves toolchain. Explicitly mentioned in Exqlite’s README as an embedded use-case. |
 | **vix** → libvips | Image processing (covers, thumbnails) | C NIF (`elixir_make`) | ❌ No ARM binaries advertised | **🔴 High risk** — libvips is ~30 MB with transitive deps (libjpeg, libpng, libwebp, etc.). Each needs Buildroot integration or cross‑compilation. Biggest blocker for a lean firmware. |
 | **mdex** → comrak + ammonia | Markdown → HTML | Rust NIF (RustlerPrecompiled) | ⚠️ Standard Rustler targets (`aarch64-unknown-linux-gnu`, possibly `arm-*`) | **🟡 Needs cross‑compilation** — RustlerPrecompiled targets use different triples than Nerves (`aarch64-nerves-linux-gnu`). Precompiled `.so` likely won’t load. Cross‑compile from source with a Nerves‑aware Rust toolchain. |
 | **lumis** → tree‑sitter | Syntax highlighting (in mdex) | Rust NIF (RustlerPrecompiled) | Same as mdex | **🟡 Same situation** — part of the mdex stack. Tree‑sitter grammars add per‑language compilation but the NIF itself is manageable. |
 | **typst** | PDF generation (tracklists) | Rust NIF (RustlerPrecompiled) | Same as mdex | **🔴 Heavy** — a full typesetting engine. Large binary, plus the same target‑triple mismatch. Likely overkill for embedded PDFs. |
 | **dominant_colors** → kmeans_colors | Colour extraction from covers | Rust NIF (`rustler_precompiled ~> 0.7`) | Same as mdex | **🟢 Small** — trivial NIF. Cross‑compilation is straightforward. Low risk. |
 | **esbuild** | JS bundling | Go binary | — | **N/A** — dev only, never deployed. |
 | **tailwind** | CSS generation | Go binary | — | **N/A** — dev only, never deployed. |
 ### Target‑architecture specifics
 | Target | CPU | Nerves toolchain triple | Precompiled‑binary match? |
 |---|---|---|---|
 | RPi 4 / 5 | Cortex‑A72 / A76 (64‑bit) | `aarch64-nerves-linux-gnu` | ⚠️ `aarch64-linux-gnu` *may* be ABI‑compatible (exqlite). Rust NIFs need rebuild. |
 | RPi 3 | Cortex‑A53 (32‑bit) | `armv7-nerves-linux-gnueabihf` | ❌ No precompiled binaries match. Everything must be compiled from source. |
 **Recommendation**: Target RPi 4 or 5 (64‑bit) first — the precompiled ecosystem is
 far better and the 32‑bit path adds unnecessary friction.
 ---
 ## 2. SQLite extensions
 The application loads two run‑time extensions via Exqlite’s `load_extensions`
 config:
 | Extension | Source | ARM status | Notes |
 |---|---|---|---|
 | **vec0** | [sqlite-vec](https://github.com/asg017/sqlite-vec) | ✅ **Good** | “Written in pure C, no dependencies, runs anywhere SQLite runs (… Raspberry Pis, etc.)” — explicitly tested on ARM. Compilable for any Nerves target. |
 | **unicode** | likely [sqlean](https://github.com/nalgeon/sqlean) (`text` module or bundle) | ✅ **ARM64 available** | Sqlean ships `sqlean-linux-arm64.zip`. For ARMv7, compile from C source. |
 ### Loading mechanism
 The project already uses an architecture‑aware directory layout:
 ```elixir
 config :myapp, Myapp.Repo,
  load_extensions: ["./priv/sqlite/#{arch_dir}/vector0"]
 ```
 This pattern extends naturally to Nerves — compile the `.so` files with the Nerves
 cross‑compiler, place them in the appropriate arch subdirectory, and Exqlite will
 load them at connection time.
 **Unknown**: Whether the `unicode` extension is built from the exact same sqlean
 release as production, or compiled independently.  Sync the build flags and version
 to avoid subtle behavioural differences.
 ---
 ## 3. Data replication & conflict resolution
 This is the **hardest problem** and the most architecturally open of the three.
 ### What Litestream does (and doesn’t)
 | Capability | Litestream | Needed for device ↔ cloud sync |
 |---|---|---|
 | Unidirectional backup (app → S3) | ✅ | ❌ |
 | Disaster recovery (restore from S3) | ✅ | ❌ |
 | Bidirectional sync | ❌ | ✅ |
 | Conflict detection / resolution | ❌ | ✅ |
 | Multi‑writer support | ❌ | ✅ |
 Litestream is purely an asynchronous backup tool. It continuously streams WAL pages
 to S3. It has **no mechanism** for a second instance to push writes back, no merge
 strategy, and no conflict detection. Adding a Nerves device that writes to the same
 database file would corrupt the Litestream generation (or create a parallel one that
 cannot be merged).
 ### What LiteFS does (and doesn’t)
 [LiteFS](https://github.com/superfly/litefs) (same creators, different project)
 provides multi‑node SQLite replication with a primary/replica model:
 - ✅ Automatic failover via distributed lease (Consul)
 - ✅ LTX‑based transaction‑log replication
 - ✅ Built‑in HTTP proxy for write‑forwarding to the primary
 - ❌ Requires FUSE (needs a Buildroot kernel module)
 - ❌ Requires Consul or static leasing
 - ❌ **Replicas are read‑only** — only one writer at a time
 - ❌ Designed for same‑region clusters, not device ↔ cloud
 - ❌ No offline‑first or intermittent‑connectivity support
 **LiteFS does not fit** the device‑sync use case. If the Nerves device goes offline,
 accumulates writes, then reconnects, LiteFS has no way to merge those writes onto
 the primary.
 ### Realistic approaches
 #### A. Read‑only Nerves device (simplest, safest)
 The Nerves device is a read‑only replica. All writes happen on the production
 server. The device periodically pulls a fresh database snapshot.
 - **Sync**: `GET /api/v1/backup` (already exists), Litestream restore, or HTTP
  range‑request to fetch a compressed DB.
 - **Pros**: Zero conflicts, simple to implement, leverages existing backup API.
 - **Cons**: No local writes — can’t add records, update notes, or scrobble from
  the device.
 #### B. Separate write domains (pragmatic)
 Never write the same record type from both sides:
 - **Production server**: records, embeddings, notes, record sets, artist info.
 - **Nerves device**: scrobbles, listening stats, playback logs.
 - Each instance has its own SQLite database.  They sync via API calls (e.g., device
  pushes scrobbles to production, production pushes new records to device).
 - **Pros**: No database‑level conflicts. Clear ownership boundaries.
 - **Cons**: App‑level routing logic. Two separate databases on the device.
 #### C. CRDT‑backed data model (most robust)
 Model the subset of data writable on both sides using Conflict‑free Replicated Data
 Types (CRDTs).
 - **Libraries**: [Automerge](https://automerge.org/), [Yjs](https://yjs.dev/),
  custom CRDT stores.
 - **Pros**: True offline‑first, automatic conflict resolution, no central
  authority needed.
 - **Cons**: Requires wrapping your data in CRDT types. No mature Elixir CRDT ↔
  SQLite bridge. Significant architecture change.
 #### D. Event sourcing with log shipping
 Write all mutations as an append‑only event log. Ship the log between instances.
 Replay to reconstruct state.
 - **Pros**: Natural fit with SQLite (append to a `mutations` table). Full audit
  trail.
 - **Cons**: Requires event‑schema design, log compaction, eventual consistency
  handling.
 #### E. Application‑level sync protocol
 Define a custom sync protocol with:
 - Vector clocks or timestamps for causality tracking.
 - Last‑write‑wins (LWW) or manual conflict resolution.
 - Periodic pull/push of changed records since the last sync point.
 - **Pros**: Domain‑specific, can optimise for exact data shapes.
 - **Cons**: Most implementation effort. Easy to get wrong.
 ### Litestream’s role on the device
 Even with a sync strategy in place, Litestream could still run on the Nerves device
 for **local disaster recovery** — streaming the device’s own WAL to a separate S3
 path.  This is independent from the production Litestream instance.
 ---
 ## Summary of risks
 | Risk | Severity | Mitigation |
 |---|---|---|
 | **vix / libvips on Nerves** | 🔴 High | Replace image processing with a lighter alternative, or offload to the production server via API. |
 | **Data‑sync strategy** | 🔴 High | No off‑the‑shelf solution. Start with read‑only replica (option A), iterate toward separate write domains (option B). |
 | **typst on embedded** | 🟡 Medium | Switch to server‑side PDF generation; client downloads the result. |
 | **Rust NIF cross‑compilation** | 🟡 Medium | Set up Nerves Rust cross‑compilation. Works but adds build complexity. Each library needs individual testing. |
 | **ARMv7 (RPi 3) vs AArch64 (RPi 4/5)** | 🟡 Medium | RPi 4/5 (64‑bit) has much better precompiled‑binary support. RPi 3 requires everything compiled from source. |
 | **Buildroot package deps (libvips)** | 🟡 Medium | Each transitive C library needs a Buildroot package definition or a custom cross‑compile step. |
 | **Firmware size** | 🟡 Medium | libvips + typst + sqlite extensions → image could exceed 200 MB. Typical Nerves firmware is 20–80 MB. |
 | **sqlite‑vec on ARM** | 🟢 Low | Explicitly supported. Pure C, no deps. Compiles trivially with the Nerves toolchain. |
 | **exqlite on Nerves** | 🟢 Low | Designed for embedded use. Source compilation well‑supported and documented. |
 | **dominant\_colors** | 🟢 Low | Tiny Rust NIF. Straightforward cross‑compilation. |
 ---
 ## Recommended next steps (if proceeding)
 1.  **Target RPi 4 or 5 (64‑bit)** — significantly better precompiled binary
    ecosystem and larger RAM for Phoenix + Oban.
 2.  **Start with a read‑only firmware** — pull database snapshots from the
    production server via the existing `/api/v1/backup` (or `/health`) endpoint.
    Don’t write locally until a sync strategy is decided.
 3.  **Audit which features to keep on‑device** — consider dropping image
    processing (vix) and PDF generation (typst) from the embedded build, or
    replacing them with lighter alternatives / server‑side APIs.
 4.  **Benchmark firmware size incrementally** — start with bare Phoenix + SQLite,
    add extensions, measure. Set a hard size budget (e.g., 120 MB).
 5.  **Design the sync protocol as a separate project** — this is the most complex
    piece. Prototype with a read‑only client first, then evaluate CRDTs, event
    sourcing, or application‑level sync based on actual write patterns.
@@ -0,0 +1,119 @@
 ---
 id: doc-8
 title: 'Opus 4.7, xhigh analysis'
 type: other
 created_date: '2026-05-04 15:08'
 updated_date: '2026-05-04 15:10'
 ---
 # Nerves Deployment Research — Findings
 Three parallel research agents covered NIFs, SQLite extensions, and Litestream replication semantics. The user's concerns were well-founded: there are real blockers, but most are solvable with custom Nerves system work. The Litestream question reframes the architecture more than expected.
 ## 1. Native libraries (NIFs on aarch64-linux-musl)
 | Library | aarch64-gnu | aarch64-musl | Status |
 |---|---|---|---|
 | `mdex` | ✅ | ✅ | Fine |
 | `lumis` | ✅ | ✅ | Fine |
 | `typst` | ✅ | ✅ | **Nerves-aware** — has `@nerves_rust_target_triple_mapping` in mix.exs |
 | `ecto_sqlite3` / `exqlite` | ✅ | ✅ | Driver fine; designed with embedded use in mind. Extensions are the issue (see §2) |
 | `dominant_colors` | likely | unknown | Verify GitHub releases; if missing, request musl target from maintainer |
 | `vix` | partial | ❌ | **Major blocker** |
 | `cloak_ecto` | n/a | n/a | Pure Elixir + `:crypto` |
 **`vix` is the dominant blocker.** Two compounding issues:
 - It loads the NIF at *compile time* (macro-driven introspection), which breaks cross-compilation on stock Nerves systems. fhunleth (Nerves core) discussed this in a 2023 ElixirForum thread; the conclusion was "no easy Elixir-level fix."
 - libvips itself is not in `nerves_system_rpi4` / `nerves_system_rpi5`. You'd need a **custom Nerves system** with `BR2_PACKAGE_LIBVIPS=y` in Buildroot, then `VIX_COMPILATION_MODE=PLATFORM_PROVIDED_LIBVIPS`.
 This is doable (15–30 min Buildroot rebuild, well-documented) but it commits you to maintaining a custom Nerves system. That single decision affects everything downstream — once you're building custom systems, ICU and sqlite-vec become "while we're at it" additions.
 **General Phoenix-on-Nerves status (2025):** Pattern is established but niche. Underjord's "LiveView on Nerves" guide is the canonical reference; a poncho project (sibling `ui` + `firmware`) is preferred over umbrella. No widely-cited "production Phoenix LiveView on Nerves at non-hobby scale" guide exists — most published examples are thermostats and birdhouses.
 ## 2. SQLite extensions
 Both extensions are blockers on stock Nerves systems for the same root reason — Nerves uses musl, prebuilt binaries assume glibc.
 **`sqlite-vec` (`vec0`):**
 - Upstream publishes `linux-aarch64` but it's **glibc** — won't `dlopen` on Nerves.
 - No musl prebuilt exists; PR #199 fixes the musl compile but is **unmerged**.
 - The `sqlite_vec` Hex wrapper (Joel Paul Koch's) downloads the glibc binary and would fail at runtime.
 - Mitigation: cross-compile inside Buildroot, ship via `rootfs_overlay/`. Or compile statically into exqlite at firmware build time (recommended pattern from the SQLCipher-on-Nerves ElixirForum thread).
 **`unicode` (ICU) extension:**
 - ICU is ~30 MB and deliberately excluded from Nerves base systems.
 - Requires a custom Nerves system with `BR2_PACKAGE_ICU=y`.
 - If you're already building custom for `vix`, the marginal cost is minimal.
 **Good news for `exqlite`:** Driver itself ships precompiled musl NIFs (since v0.13.7), supports `load_extension`, and has been used on Nerves in production (the SQLCipher thread). The complexity is purely in shipping the extension `.so`s built against the same musl toolchain.
 ## 3. Litestream replication — this is the architectural pivot
 **Headline:** Litestream is *not* a multi-master replicator. It's a one-way backup tool. If the Nerves device writes locally, those writes are lost on the next restore — no conflict detection, no reconciliation, no merge.
 **Two viable replica modes today (both newer than the docs you may remember):**
 1. **VFS read replica** (stable, in `litestream.io/guides/vfs/`): Nerves device queries S3-backed replica directly. ~1 minute staleness matches your sync interval. Writes return `"litestream is a read only vfs"` at the SQLite layer. **Requires CGO** — verify this works in your Nerves build.
 2. **Restore-then-poll**: simpler, no CGO. Restore on startup, periodically re-restore. Live replication via the embedded HTTP server existed but the example repo was **archived November 2023** — treat as experimental.
 **For the architectures you'd consider:**
 - **Read-only Nerves device** (browse music library at home, no edits) — Litestream VFS replica works, modulo CGO. This is the simplest path.
 - **Read-write Nerves device** (record listening, edit notes) — Litestream alone is insufficient. Three options:
  1. **Split databases**: music library is read-only via Litestream; a separate small SQLite on the device captures writes and pushes to the production API out-of-band. Cleanest fit for your existing app shape, since scrobbles are already a separate domain.
  2. **LiteFS** (Fly.io's project): true primary/replica with write-forwarding. Requires FUSE — not viable on Nerves without kernel module support, so likely off the table.
  3. **Writes go directly to production API**: Nerves device calls the prod Phoenix JSON API for any write; reads are local. Requires the device to be online for writes (acceptable for a home device).
 - **Other distributed-SQLite options** — rqlite (Raft, kills write throughput), Marmot (NATS-based CDC, eventual consistency), cr-sqlite (CRDTs). All exist; all are heavier than the problem warrants for one home device.
 ## Synthesis: where the unknowns concentrate
 The three concerns aren't independent — they collapse into **one decision**: are you willing to maintain a custom Nerves system?
 - If **yes**: `vix`, ICU, and `sqlite-vec` are all solvable in the same Buildroot config. The remaining unknowns are minor (`dominant_colors` musl target, exact poncho structure for the existing umbrella-style codebase).
 - If **no**: the application as currently architected cannot run on Nerves. You'd have to drop image processing, drop FTS unicode collation, drop similarity search — at which point it's a different application.
 The **biggest non-obvious unknown** is the Litestream architecture decision. Going read-only on the Nerves side is dramatically simpler than supporting writes, and it covers most of the value (browsing, search, AI chat against the local collection). Splitting writes to a separate small DB that pushes to prod is the natural compromise — and your codebase is already structured to support it (scrobbles are a separate context, notes are a separate schema).
 **Suggested next step if you want to proceed:** prototype a custom `nerves_system_rpi5` with libvips + ICU on a feature branch, before touching the application. That single artifact resolves three of the four major risks; everything else is application-layer work that can be done iteratively.
 ---
 ## Sources
 ### NIF / Phoenix-on-Nerves
 - [Underjord: LiveView on Nerves](https://underjord.io/liveview-on-nerves.html)
 - [ElixirForum: Cross Compiling with NIF as compilation dependency (vix/Nerves thread)](https://elixirforum.com/t/cross-compiling-with-nif-as-compilation-dependency/59821)
 - [Nerves docs: Compiling Non-BEAM Code](https://hexdocs.pm/nerves/compiling-non-beam-code.html)
 - [rustler_precompiled Precompilation Guide](https://hexdocs.pm/rustler_precompiled/precompilation_guide.html)
 - [mdex GitHub releases (NIF targets)](https://github.com/leandrocp/mdex/releases)
 - [lumis GitHub releases](https://github.com/leandrocp/lumis)
 - [typst Elixir GitHub releases](https://github.com/Hermanverschooten/typst/releases)
 - [vix HexDocs README](https://hexdocs.pm/vix/readme.html)
 - [exqlite README](https://hexdocs.pm/exqlite/readme.html)
 - [NervesConf 2024 Talks thread](https://elixirforum.com/t/nervesconf-2024-talks/75205)
 - [Customizing Your Nerves System](https://hexdocs.pm/nerves/customizing-systems.html)
 ### SQLite extensions
 - [Releases · asg017/sqlite-vec](https://github.com/asg017/sqlite-vec/releases)
 - [Fix for musl compile PR #199 · asg017/sqlite-vec](https://github.com/asg017/sqlite-vec/pull/199)
 - [Build and install exqlite with sqlcipher into nerves rpi5 image — ElixirForum](https://elixirforum.com/t/build-and-install-exqlite-with-sqlcipher-into-nerves-rpi5-image/75089)
 - [Concerning warning compiling under Nerves · Issue #166 · elixir-sqlite/exqlite](https://github.com/elixir-sqlite/exqlite/issues/166)
 - [Compiling sqlite-vec — Alex Garcia](https://alexgarcia.xyz/sqlite-vec/compiling.html)
 - [GitHub — joelpaulkoch/sqlite_vec](https://github.com/joelpaulkoch/sqlite_vec)
 - [Advanced Configuration — Nerves](https://hexdocs.pm/nerves/advanced-configuration.html)
 ### Litestream
 - [Live Read Replication — Litestream (tip)](https://tip.litestream.io/guides/read-replica/)
 - [VFS Read Replicas — Litestream](https://litestream.io/guides/vfs/)
 - [How it works — Litestream](https://litestream.io/how-it-works/)
 - [Tips & Caveats — Litestream](https://litestream.io/tips/)
 - [Alternatives — Litestream](https://litestream.io/alternatives/)
 - [Litestream: Revamped — Fly.io Blog](https://fly.io/blog/litestream-revamped/)
 - [benbjohnson/litestream-read-replica-example (archived 2023)](https://github.com/benbjohnson/litestream-read-replica-example)
 - [Live read replicas — Issue #8](https://github.com/benbjohnson/litestream/issues/8)
 - [LiteFS vs Litestream vs rqlite vs dqlite on VPS in 2025](https://onidel.com/blog/sqlite-replication-vps-2025)
 - [Marmot distributed SQLite replicator](https://github.com/maxpert/marmot)
 - [mvSQLite](https://github.com/losfair/mvsqlite)
 - [cr-sqlite](https://github.com/vlcn-io/cr-sqlite)
@@ -0,0 +1,500 @@
 ---
 id: doc-9
 title: 'GPT 5.5, high analysis'
 type: other
 created_date: '2026-05-04 15:11'
 ---
 # Nerves Deployment Research Report
 Date: 2026-05-04
 ## Scope
 This report explores the feasibility and risk areas for deploying this Phoenix/SQLite music library application as part of a Nerves application. No implementation work was performed.
 The research focused on:
 - Native dependencies and NIF availability for ARM/Nerves targets.
 - SQLite loadable extensions on Linux ARM.
 - Litestream replication behavior, especially conflicts and local writes.
 - Project-specific architecture implications based on the current application docs and dependency graph.
 ## Project Context
 The application is a Phoenix LiveView app backed by three SQLite databases:
 - `MusicLibrary.Repo`: main application data.
 - `MusicLibrary.BackgroundRepo`: Oban jobs.
 - `MusicLibrary.TelemetryRepo`: persistent telemetry metrics.
 The main repo loads two SQLite extensions at runtime:
 - `unicode`
 - `vec0`
 The application currently ships precompiled extension binaries for:
 - `darwin-amd64`
 - `darwin-arm64`
 - `linux-amd64`
 - `linux-arm64`
 The app also uses several native or precompiled-native dependencies:
 - `exqlite`, via `ecto_sqlite3`
 - `vix`, backed by libvips
 - `dominant_colors`, via RustlerPrecompiled
 - `mdex`, via RustlerPrecompiled
 - `lumis`, via RustlerPrecompiled
 - `typst`, via RustlerPrecompiled
 ## Summary
 A Nerves deployment appears feasible as a read-only or mostly read-only edge deployment, but it should not be treated as a second writable production node.
 The central architectural constraint is that Litestream is not an application-level sync or conflict-resolution system. It is physical SQLite/WAL replication. If the Nerves device writes to its local database, those writes either remain local, conflict with read-replica/follow mode, or create a divergent database history.
 The safest complementary architecture is:
 1. The Nerves device keeps a local read model restored from production.
 2. User writes are sent to the production app through an authenticated API.
 3. Production remains the single SQLite writer.
 4. Litestream brings the resulting production state back down to the device.
 5. Offline-capable writes, if required, are handled with an application-level outbox and explicit conflict rules.
 ## Native Dependency Challenges
 ### Nerves Cross-Compilation Model
 Nerves sets environment variables that are intended to support cross-compiling C/C++ NIFs and ports, including `CC`, `CFLAGS`, `LDFLAGS`, `CROSSCOMPILE`, `NERVES_SDK_SYSROOT`, `TARGET_ARCH`, `TARGET_OS`, and `TARGET_ABI`.
 This is helpful for dependencies that use standard `elixir_make`, `cc_precompiler`, Rustler, or RustlerPrecompiled flows. It does not guarantee that every dependency will compile cleanly, especially if the dependency needs system libraries that are not present in the Nerves system.
 Source: https://hexdocs.pm/nerves/environment-variables.html
 ### Exqlite / Ecto SQLite
 `ecto_sqlite3` depends on `exqlite`. Exqlite documents precompiled artifacts and native build options. It can also be forced to use a system SQLite through `EXQLITE_USE_SYSTEM`, `EXQLITE_SYSTEM_CFLAGS`, and `EXQLITE_SYSTEM_LDFLAGS`.
 Risk level: medium.
 Likely challenge:
 - Verifying that the selected Nerves target receives a compatible Exqlite NIF.
 - Deciding whether to use bundled SQLite or a system SQLite from the Nerves image.
 - Ensuring SQLite compile-time options still satisfy app requirements, especially FTS5 and loadable extensions.
 Source: https://hexdocs.pm/exqlite/readme.html
 ### Vix / Libvips
 `vix` is the highest-risk native dependency in this application.
 Vix includes prebuilt binaries for macOS and Linux, and it can alternatively use a platform-provided libvips by setting:
 ```sh
 VIX_COMPILATION_MODE=PLATFORM_PROVIDED_LIBVIPS
 ```
 However, on Nerves this may require a custom Nerves system that includes libvips and its development headers. Vix also has historically been tricky in cross-compilation contexts because it depends on libvips introspection during compilation.
 Risk level: high.
 Likely challenge:
 - Official Nerves systems may not include libvips.
 - A custom Buildroot/Nerves system may be needed.
 - Optional libvips codec support may differ from production.
 - Image processing behavior may not match production unless libvips versions and build options are controlled.
 Sources:
 - https://hexdocs.pm/vix/readme.html
 - https://hexdocs.pm/nerves/customizing-systems.html
 - https://elixirforum.com/t/cross-compiling-with-nif-as-compilation-dependency/59821
 ### RustlerPrecompiled Dependencies
 RustlerPrecompiled supports Nerves-style target detection via `TARGET_ARCH`, `TARGET_ABI`, and `TARGET_OS`. Default targets include common ARM Linux triples such as:
 - `aarch64-unknown-linux-gnu`
 - `aarch64-unknown-linux-musl`
 - `arm-unknown-linux-gnueabihf`
 This is promising for `mdex`, `lumis`, `typst`, and `dominant_colors`, but every package still needs to publish compatible precompiled artifacts for the relevant target and NIF version.
 Risk level: medium.
 Specific concern:
 - `dominant_colors` is old and low-activity. It depends on `rustler_precompiled ~> 0.7`, and there is no guarantee that all current Nerves targets are covered.
 Sources:
 - https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html
 - https://hex.pm/packages/dominant_colors
 ## SQLite Extension Challenges
 The app already includes `priv/sqlite_extensions/linux-arm64/unicode.so` and `vec0.so`. This is a good starting point, but it does not prove Nerves compatibility.
 Open questions:
 - Were the `linux-arm64` extensions built against assumptions compatible with the target Nerves system?
 - Are they dynamically linked against glibc or musl expectations that differ from the Nerves target?
 - Do they require symbols unavailable in the SQLite build used by Exqlite?
 - Does the target allow SQLite loadable extensions as used by Ecto/Exqlite?
 The current platform detection only maps Linux `aarch64` to `linux-arm64`. That means 32-bit ARM targets such as Raspberry Pi Zero-class systems would fail unless new binaries and platform mapping were added.
 Risk level: medium for aarch64, high for arm32.
 Recommended spike:
 - Build a minimal Nerves firmware for the intended target.
 - Boot it on hardware or QEMU if possible.
 - Run only the repo extension tests:
  - `SELECT vec_version()`
  - Unicode extension behavior currently covered by `test/music_library/repo_test.exs`
 ## Nerves Filesystem and Runtime Considerations
 Nerves root filesystems are read-only by default. Persistent data should live on the writable data partition, typically `/data`.
 This affects:
 - `DATABASE_PATH`
 - `BACKGROUND_DATABASE_PATH`
 - `TELEMETRY_DATABASE_PATH`
 - Litestream restore target paths
 - Litestream buffers, if using VFS write mode
 - Temporary files used by image/PDF processing
 Source: https://hexdocs.pm/nerves/faq.html
 The current application expects runtime environment variables for production configuration. A Nerves deployment would need a target-specific runtime configuration story for:
 - `SECRET_KEY_BASE`
 - `CLOAK_ENCRYPTION_KEY`
 - `LOGIN_PASSWORD`
 - `API_TOKEN`
 - External API keys, if those features are enabled
 - SQLite DB paths under `/data`
 ## Litestream Findings
 ### What Litestream Does
 Litestream is a streaming replication and disaster recovery tool for SQLite. It runs as a separate process and continuously copies WAL data to a replica store such as S3-compatible object storage.
 It is physical replication, not row-level sync.
 Source: https://litestream.io/how-it-works/
 ### Restore Mode
 `litestream restore` can restore a database from a replica. It can also restore to a specific transaction ID or timestamp.
 `restore -f` enables follow mode, continuously applying new data as it becomes available. Litestream explicitly states that the restored database in follow mode should only be opened read-only.
 Source: https://litestream.io/reference/restore/
 Implication:
 - Follow mode is a good fit for a local read replica.
 - It is not a fit for local writes.
 ### VFS Read Replica
 The Litestream VFS can serve reads directly from object storage. It is read-only by default, fetches pages on demand, caches them, and polls for new LTX files.
 Important constraints:
 - Write attempts fail unless write mode is enabled.
 - It requires a CGO-enabled SQLite VFS extension.
 - It is network-latency sensitive.
 - It needs contiguous LTX coverage.
 - Initial snapshot availability is required.
 Source: https://litestream.io/reference/vfs/
 Implication for Nerves:
 - The VFS is probably more complex than a restore-to-`/data` approach.
 - Building and loading `litestream-vfs` on Nerves would be a separate native extension challenge.
 - A fully restored local DB is likely simpler and more robust for embedded use.
 ### VFS Write Mode
 Litestream VFS write mode allows writes to a remote-first database by buffering local changes and periodically uploading them.
 However, Litestream documents that write mode assumes a single writer. Multiple writers trigger conflict detection, not conflict prevention or automatic conflict resolution.
 Limitations include:
 - Single-writer assumption.
 - Sync latency.
 - Local write buffer disk usage.
 - Crash window for unsynced buffered writes.
 - Application-owned conflict resolution.
 Source: https://litestream.io/guides/vfs-write-mode/
 Implication:
 - VFS write mode should not be used to let production and a Nerves device both write to the same logical database unless an external single-writer lock exists.
 - It does not solve offline bidirectional sync.
 ## What Happens If the Device Writes?
 There are several possible scenarios.
 ### Scenario 1: Device Restores Production DB Once and Writes Locally
 The device diverges from production.
 Production does not receive those writes. Later restores or updates from production may overwrite or invalidate the local changes depending on the restore strategy.
 There is no automatic merge.
 ### Scenario 2: Device Uses `restore -f` and Also Writes
 This is unsupported. Litestream says follow-mode restored databases should only be opened read-only.
 Expected outcome:
 - Conflict or corruption risk.
 - Follow mode may be unable to apply incoming production changes cleanly.
 ### Scenario 3: Device Uses Litestream VFS Read Replica and Writes
 By default, writes fail because the VFS is read-only.
 ### Scenario 4: Device Uses VFS Write Mode Against Same Replica Path as Production
 This is a multiple-writer design. Litestream detects conflicts but does not reconcile them.
 Expected outcome:
 - Conflict errors.
 - Need for application retry/rebase logic.
 - Possible divergent generations or failed sync depending on exact mode.
 ### Scenario 5: Device Sends Writes to Production API
 This is the cleanest model.
 Production remains the only writer to the production SQLite database. The device reads a replicated copy and submits commands to production when it needs to mutate state.
 This works well when the device has network connectivity.
 ### Scenario 6: Device Supports Offline Writes
 This requires application-level sync.
 A likely shape:
 - Device stores local commands/events in a separate outbox database.
 - Each command has an idempotency key.
 - The production API accepts commands and performs validation/conflict checks.
 - Production returns accepted/rejected/conflicted status.
 - Device periodically pushes pending commands.
 - Device refreshes its read model from Litestream.
 This is not something Litestream provides.
 ## Data Architecture Recommendations
 ### Recommended Baseline: Read Replica + Write-Through API
 Use Litestream to hydrate the main application database onto the device under `/data`.
 Only replicate the main app DB:
 - Replicate `MusicLibrary.Repo`: yes.
 - Replicate `MusicLibrary.BackgroundRepo`: probably no.
 - Replicate `MusicLibrary.TelemetryRepo`: no.
 Make the local app behave as read-only for production-backed tables, or route mutations through the production API.
 Benefits:
 - Avoids multi-writer SQLite.
 - Avoids Litestream conflict semantics.
 - Keeps production as source of truth.
 - Preserves the current production deployment model.
 Challenges:
 - The UI may currently assume writes are local Ecto writes.
 - Many LiveView workflows may need capability checks.
 - Oban workers on device must be carefully disabled or narrowed.
 ### Alternative: Device-Local Features Only
 Allow writes only to separate device-local tables/databases:
 - Device settings.
 - Local telemetry.
 - Local cache state.
 - Sync metadata.
 - Offline outbox.
 Keep production-replicated data read-only.
 This avoids mixing local-only rows into the production physical database copy.
 ### Avoid: Bidirectional Litestream
 Do not attempt to use Litestream as a bidirectional sync system between production and Nerves.
 The core problem is that Litestream replicates database pages/WAL, not business operations. It has no understanding of records, notes, assets, chats, Oban jobs, timestamps, or conflict intent.
 ## Application-Specific Concerns
 ### Oban
 The app uses Oban heavily for imports, enrichments, external API calls, periodic refreshes, and maintenance.
 On a device:
 - Running production Oban jobs locally would be risky.
 - Replicating the Oban database would be risky.
 - Cron jobs that mutate the main database should likely be disabled.
 - External API workers may be inappropriate if API secrets are not present.
 The Nerves deployment probably needs a distinct Oban configuration:
 - disabled entirely, or
 - local-only queues, or
 - only maintenance tasks that do not mutate production-backed data.
 ### Secrets
 The main DB contains encrypted secrets via Cloak. If production data is restored to the device, encrypted rows come with it.
 Options:
 - Put the same `CLOAK_ENCRYPTION_KEY` on the device, which increases exposure risk.
 - Do not replicate secrets.
 - Keep replicated DB encrypted-but-unreadable for those features.
 - Split secrets into a production-only database/table in a future architecture.
 ### Assets
 Assets are stored as binary blobs in SQLite and content-addressed by hash. This is favorable for replication because immutable content has simpler conflict semantics.
 However:
 - Large assets affect restore time and storage wear.
 - Cover refresh/image generation on-device may exercise Vix/libvips.
 ### Embeddings and sqlite-vec
 The app uses OpenAI embeddings and `sqlite-vec`.
 Read-only similarity search on the device should be plausible if `vec0` loads. Generating embeddings on-device would require OpenAI credentials and write access, so it should probably stay production-side.
 ### External APIs
 MusicBrainz, Last.fm, Discogs, Wikipedia, Brave Search, OpenAI, and Mailgun are integrated.
 On-device deployment should decide which of these are:
 - disabled,
 - proxied through production,
 - allowed directly from the device.
 For most production-backed data mutations, proxying through production is cleaner.
 ## Proposed Research Spikes
 ### Spike 1: Minimal Nerves Build
 Goal:
 - Determine whether the current dependency tree can build for the intended Nerves target.
 Checks:
 - `exqlite` NIF loads.
 - RustlerPrecompiled dependencies load.
 - Vix loads or fails with a clear path.
 - Firmware size remains acceptable.
 ### Spike 2: SQLite Extension Load Test
 Goal:
 - Verify `unicode.so` and `vec0.so` on target hardware.
 Checks:
 - `MusicLibrary.Repo.ensure_supported_platform!()`
 - `SELECT vec_version()`
 - representative FTS/unicode behavior
 - vector similarity query
 ### Spike 3: Read-Only Litestream Replica
 Goal:
 - Restore the production main DB to `/data`.
 - Start app with the main repo opened read-only if possible.
 - Confirm browsing/search/similarity workflows.
 Checks:
 - cold restore time
 - incremental update lag
 - disk usage
 - behavior during network loss
 - behavior during power loss
 ### Spike 4: Write-Through Prototype
 Goal:
 - Pick one simple mutation, such as creating a note or updating wishlist state.
 - Route it to production API instead of local DB.
 - Observe how the replicated DB catches up.
 Checks:
 - UX during replication lag
 - idempotency
 - authentication
 - error handling
 ### Spike 5: Offline Outbox Design
 Goal:
 - Only if offline writes are a requirement.
 - Design command schema, conflict policy, and reconciliation flow.
 Likely requirements:
 - globally unique command IDs
 - entity version or `updated_at` checks
 - tombstones for deletes
 - conflict UI or deterministic server-side policy
 - separation between read model DB and local outbox DB
 ## Open Questions
 - Which hardware target is intended: Raspberry Pi 4/5 aarch64, Raspberry Pi Zero arm32, or something else?
 - Does the device need to serve the full LiveView UI, or a constrained local UI?
 - Is the device expected to work without internet?
 - Are local writes required, or can all writes go through production?
 - Should sensitive tables such as `secrets`, `chats`, and possibly API-derived metadata be present on the device?
 - Is Vix/libvips actually needed on the device, or can image processing be production-only?
 - Should the device run any Oban workers?
 - Is the production S3-compatible bucket reachable from the device network?
 - Is eventual consistency acceptable for the UI after write-through actions?
 ## Conclusion
 The Nerves architecture is plausible if the device is treated as an edge read replica with carefully selected local behavior. The native dependency work is significant but bounded: Vix/libvips and SQLite extension ABI compatibility are the main technical spikes.
 The larger architectural decision is data ownership. Litestream can get production SQLite data onto the device and keep it fresh, but it should not be used as a bidirectional sync system. If the Nerves app must write production data, those writes should go through production or through an explicit application-level sync protocol.