music_library/docs/production-infrastructure.md

# Production Infrastructure

> **Maintenance rule**: Update this file when infrastructure changes (hosting, databases,
> backups, monitoring, deployment pipeline, external services). Keep it factual and concise.

**CRITICAL: under no circumstances you're allowed to interact directly with
production infrastructure without asking the user first.**

## Overview

Self-hosted Phoenix application running as a Docker container orchestrated by
[Coolify](https://coolify.io). SQLite for all persistence, Litestream for continuous
backup to S3-compatible object storage. Deployed automatically on every `main` branch
push via GitHub Actions.

**Domain**: `music-library.claudio-ortolina.org`

---

## Hosting

| Component | Technology |
|-----------|-----------|
| Orchestration | Coolify (self-hosted) |
| Container runtime | Docker |
| SSL termination | Coolify reverse proxy |
| HTTP redirect | HTTP 301 → HTTPS (enforced by app config) |

The Docker image is a multi-stage build:

1. **Builder** — `hexpm/elixir:1.20.0-rc.4-erlang-28.5-debian-trixie-20260421-slim` with
   Node.js 25, compiles deps, builds assets (`mix assets.deploy`), generates an OTP release.
2. **Runner** — `debian:trixie-20260421-slim` with minimal runtime deps (`libstdc++6`,
   `openssl`, `libncurses6`, `ca-certificates`). Runs as unprivileged `nobody` user.

Fluxon UI (licensed dependency) is fetched during build via Docker build secrets
(`FLUXON_LICENSE_KEY`, `FLUXON_KEY_FINGERPRINT`).

---

## Databases

Three separate SQLite databases, each managed by its own Ecto repo:

| Repo | Purpose | Cache size | Pool size |
|------|---------|------------|-----------|
| `MusicLibrary.Repo` | Application data | 128 MB | `$POOL_SIZE` (default 5) |
| `MusicLibrary.BackgroundRepo` | Oban job queue | 16 MB | `$POOL_SIZE` (default 5) |
| `MusicLibrary.TelemetryRepo` | Persistent telemetry metrics | 4 MB | 2 |

All databases use incremental auto-vacuum. Paths are configured via environment variables
(`DATABASE_PATH`, `BACKGROUND_DATABASE_PATH`, `TELEMETRY_DATABASE_PATH`).

---

## Backups

### Litestream (continuous)

Configured inline in `compose.yaml`. Runs as a separate Docker Compose service
(`litestream/litestream:0.5.11-scratch`) sharing the database volume.

| Setting | Value |
|---------|-------|
| S3 endpoint | `https://nbg1.your-objectstorage.com` |
| Bucket | `ffmusiclibrary` |
| Sync interval | 60 minutes |
| Retention | 672 hours (28 days) |
| Healthcheck | `litestream databases` every 30s (timeout 10s, 3 retries) |

Credentials via environment: `LITESTREAM_ACCESS_KEY_ID`, `LITESTREAM_SECRET_ACCESS_KEY`.

### Manual backup

Two manual options are available:

- `mise run prod:backup` — takes an atomic snapshot of the live database on the
  production server using `sqlite3 .backup`, then rsyncs it locally as
  `data/music_library_prod_<timestamp>.db`. Safe under concurrent writes.
- `mise run prod:litestream-backup` — restores the production database locally from
  the S3 Litestream replica. Requires `LITESTREAM_ACCESS_KEY_ID` and
  `LITESTREAM_SECRET_ACCESS_KEY`.

Old backups can be cleaned with `mise run prod:prune-backups`.

---

## Deployment Pipeline

### CI/CD (GitHub Actions)

Workflow: `.github/workflows/test_and_deploy.yml`

Triggers: push to `main`/tags, pull requests, manual `workflow_dispatch`.
Concurrency control cancels in-progress runs for the same ref.
Uses `mise` (via `jdx/mise-action@v4`) for tool version management.

```
Push to main (or PR / manual dispatch)
  ├── Lint (format, gettext, credo, sobelow, mix_audit, shellcheck, docker image, asset build)
  ├── Test (mix test with partitioning, coverage ≥75%)
  └── Deploy (requires GitHub environment approval, main branch only)
        ├── Trigger deployment via Coolify API (hurl)
        ├── Wait for container health check
        └── Run post-deploy verification (test/prod.hurl)
```

A manual verification workflow (`.github/workflows/verify.yml`) can be triggered via
`workflow_dispatch` to re-run production checks without deploying.

### Dependency management (Dependabot)

Automated dependency updates via GitHub Dependabot (`.github/dependabot.yml`):

- **Docker (Dockerfile)**: daily checks for builder/runner base image updates, max 5 open PRs
- **Docker Compose**: daily checks for service image updates
- **Elixir (mix)**: daily checks, max 10 open PRs
- **NPM**: daily checks, max 10 open PRs, ignores path-based local deps
- **GitHub Actions**: daily checks for action version updates

Fluxon (private dependency) is configured with a dedicated registry entry.

### Post-deploy verification

`test/prod.hurl` checks:

- HTTP → HTTPS redirect
- API authentication enforcement
- Valid API responses with auth token
- Dev routes require login
- Bot scanner paths (`wp-admin`, `.env`, `xmlrpc`) return 404

### Deployment credentials

| Secret/Variable | Purpose |
|-----------------|---------|
| `COOLIFY_TOKEN` | API authentication (GitHub secret) |
| `COOLIFY_HOST` | Coolify server address (GitHub variable) |
| `COOLIFY_APP_UUID` | Application identifier (GitHub variable) |

---

## Environment Variables

### Required

| Variable | Purpose |
|----------|---------|
| `DATABASE_PATH` | Absolute path to main SQLite database |
| `BACKGROUND_DATABASE_PATH` | Absolute path to background jobs database |
| `TELEMETRY_DATABASE_PATH` | Absolute path to telemetry database |
| `SECRET_KEY_BASE` | Phoenix secret (`mix phx.gen.secret`) |
| `CLOAK_ENCRYPTION_KEY` | Base64-encoded 32-byte AES key for encrypted secrets |
| `LOGIN_PASSWORD` | Web login password |
| `API_TOKEN` | Bearer token for API endpoints |
| `MAILGUN_API_KEY` | Mailgun API key |

### Optional

| Variable | Default | Purpose |
|----------|---------|---------|
| `SERVICE_FQDN_WEB` | `example.com` | Application domain |
| `PORT` | `4000` | HTTP listen port |
| `POOL_SIZE` | `5` | Database connection pool size |
| `MAILGUN_DOMAIN` | `mailgun.fullyforged.com` | Mailgun sending domain |
| `DEFAULT_TIMEZONE` | `Europe/London` | Application timezone |
| `OPENAI_KEY` | — | OpenAI API (embeddings, chat) |
| `DISCOGS_PERSONAL_ACCESS_TOKEN` | — | Discogs API |
| `BRAVE_SEARCH_API_KEY` | — | Brave Search API |
| `LAST_FM_API_KEY` | — | Last.fm API |
| `LAST_FM_SHARED_SECRET` | — | Last.fm scrobbling auth |
| `LAST_FM_USER` | — | Last.fm username |

---

## OTP Release

Standard Mix release. Entry point: `rel/overlays/bin/server` (sets `PHX_SERVER=true`).

Migrations are skipped during release boot (`skip_migrations?/0` returns `true` when
`RELEASE_NAME` is set). Instead, Coolify is configured to run migrations after the
Docker image is built and the container is started — this happens via the
`rel/overlays/bin/migrate` script, executed as a post-deployment command in Coolify
before the application begins serving traffic.

A standalone `rel/overlays/bin/migrate` script is also available for manual use.

ERL_FLAGS: `+JPperf true` (JIT performance monitoring).

---

## Monitoring & Observability

### Health check

`GET /health` — queries the main database, returns 200 or 500. Used by Docker health
checks and post-deploy verification.

### Error tracking

`ErrorTracker` with email notifications via Mailgun:

- Listens to `:error_tracker` telemetry events
- Throttles repeated error notifications
- Non-actionable errors (bot scanners, `NoRouteError`) filtered via `ErrorIgnorer`
- Muted errors skip email notifications

### Telemetry

SQLite-backed persistent metrics (`MusicLibraryWeb.Telemetry.Storage`) with 30-second
polling interval. Events are buffered in GenServer state keyed by metric and flushed to
SQLite every 5 seconds inside a single transaction; reads via `metrics_history/1`
force-flush only the requested metric so the dashboard sees fresh data without waiting
for the next tick. Per-metric retention is capped at 32 768 rows
(`:retention_limit`), pruned after each flush. Flush failures are logged at `:warning`
and the offending batch is dropped. Tracks:

- Database query times (total, query, queue)
- External API request latency (Finch)
- Rate limiter throttle durations
- Asset cache hit/miss
- Oban job metrics (enqueue, execute, attempt, discard)
- Scrobble rules processing (duration, exceptions)
- Error tracker counters

### Dashboards (behind auth)

- Phoenix LiveDashboard (`/dev/dashboard`)
- Oban Web (`/dev/oban`)
- ErrorTracker (`/dev/errors`)

---

## Email

Mailgun via Swoosh (`Swoosh.Adapters.Mailgun`).

| Setting | Value |
|---------|-------|
| From | `postmaster@mailgun.fullyforged.com` |
| To | `claudio@fullyforged.com` |

Used for error notifications and the daily "records on this day" digest email.

---

## Encryption

Cloak vault (`MusicLibrary.Vault`) with AES.GCM cipher for at-rest encryption of
secrets stored in the `secrets` table. Key configured via `CLOAK_ENCRYPTION_KEY`
(base64-encoded 32-byte key).

---

## Tool Versions

Defined in `mise.toml`.