ML-173: research and plan
This commit is contained in:
Claudio Ortolina
+137
@@ -0,0 +1,137 @@
|
||||
---
|
||||
id: doc-15
|
||||
title: Research Removing dependencies from s3-browser extension
|
||||
type: specification
|
||||
created_date: "2026-05-09 08:31"
|
||||
updated_date: "2026-05-09 08:48"
|
||||
tags:
|
||||
- pi-extension
|
||||
- s3-browser
|
||||
- research
|
||||
---
|
||||
|
||||
# Research: Removing dependencies from s3-browser extension
|
||||
|
||||
## Current state
|
||||
|
||||
The s3-browser pi extension (`.pi/extensions/s3-browser/`) uses `@aws-sdk/client-s3` v3.787.0 as its sole npm dependency. This single package pulls in ~150+ transitive dependencies across `@aws-sdk/*`, `@aws-crypto/*`, `@smithy/*`, `fast-xml-parser`, `tslib`, and others.
|
||||
|
||||
The extension only performs one API operation: **ListObjectsV2** against an S3-compatible object storage endpoint (`nbg1.your-objectstorage.com` — Hetzner Object Storage). It uses:
|
||||
|
||||
- Static credentials (`LITESTREAM_ACCESS_KEY_ID`, `LITESTREAM_SECRET_ACCESS_KEY`)
|
||||
- Path-style endpoint (`forcePathStyle: true`)
|
||||
- No credential chain resolution, no STS, no SSO, no IMDS
|
||||
|
||||
The S3 ListObjectsV2 REST API is a simple `GET /?list-type=2&prefix=...` with AWS SigV4 authentication. The response is XML.
|
||||
|
||||
## Implementation routes
|
||||
|
||||
### Route A: Full zero-dependency client
|
||||
|
||||
**Implement SigV4 signing + HTTP request + XML parsing using only Node.js built-in modules.**
|
||||
|
||||
Replace the entire `@aws-sdk/client-s3` usage with ~200 lines of custom TypeScript using:
|
||||
|
||||
- `crypto.createHmac("sha256", ...)` — for SigV4 signing key derivation and request signature
|
||||
- `fetch` (built-in since Node 18) — for HTTP requests with AbortSignal support
|
||||
- Regex-based XML extraction — for parsing the simple, well-known ListObjectsV2 response schema
|
||||
|
||||
**SigV4 signing** implementation (~100-120 lines):
|
||||
|
||||
1. Derive signing key: `HMAC-SHA256(region, HMAC-SHA256(service, HMAC-SHA256("aws4_request", HMAC-SHA256(date, "AWS4" + secret))))`
|
||||
2. Build canonical request (HTTP method, URI, query string, headers, signed headers, payload hash)
|
||||
3. Build string to sign (algorithm, timestamp, scope, hash of canonical request)
|
||||
4. Calculate signature and assemble Authorization header
|
||||
5. Set `x-amz-content-sha256` and `x-amz-date` headers
|
||||
|
||||
**XML parsing** (~40-50 lines):
|
||||
|
||||
- The ListObjectsV2 response uses a fixed, simple XML schema
|
||||
- Extract `<Key>`, `<Size>`, `<LastModified>` from each `<Contents>` block via regex
|
||||
- Extract `<IsTruncated>` and `<NextContinuationToken>` for pagination
|
||||
- No full XML DOM needed
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Truly zero dependencies — eliminates the entire supply-chain risk
|
||||
- Total code is small and auditable (~200 lines)
|
||||
- SigV4 is a well-documented, deterministic algorithm; easy to verify correctness
|
||||
- The extension has no XML namespace complexities or edge cases to worry about
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Must implement SigV4 correctly (though straightforward and well-specified)
|
||||
- Must handle edge cases in XML parsing (empty response, missing fields, XML namespaces in the response)
|
||||
|
||||
**Dependency surface after change:** 0 npm packages (only Node.js built-ins)
|
||||
|
||||
---
|
||||
|
||||
### Route B: Minimal dependency — SigV4 signing library (`aws4`)
|
||||
|
||||
Use the `aws4` npm package (single-file, ~200 lines) for signing, and Node.js built-in modules for HTTP and XML parsing.
|
||||
|
||||
The `aws4` package is a well-established, minimal library that only signs requests — no HTTP client, no credential resolution, no XML parsing. It is a single-file package with **0 transitive dependencies** (verified: v1.13.2 has an empty `dependencies` field). It's widely used in production and hasn't needed an update in 2 years.
|
||||
|
||||
**Pros:**
|
||||
|
||||
- No need to implement SigV4 manually (reduces risk of signing bugs)
|
||||
- Drastically reduced dependency surface: 1 package vs ~150
|
||||
- `aws4` is auditable in minutes (single file, ~200 lines)
|
||||
- Equally minimal in supply-chain terms as Route A (0 transitive deps)
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Still introduces an external dependency (though extremely low-risk: single-file, 0-dep, 2-year-stable)
|
||||
- Slightly less educational value (doesn't demonstrate SigV4 internals)
|
||||
- `aws4` is designed for Node.js `http.request` options; must bridge to `fetch` by extracting `headers` from the signed options object and constructing the full URL
|
||||
|
||||
**Dependency surface after change:** 1 npm package (`aws4`, 0 transitive dependencies)
|
||||
|
||||
---
|
||||
|
||||
### Route C: AWS SDK minimal subset
|
||||
|
||||
Replace `@aws-sdk/client-s3` with only the credential + signing packages from the AWS SDK, making HTTP calls manually.
|
||||
|
||||
For example: `@aws-sdk/credential-provider-node` + `@smithy/signature-v4` + Node.js `fetch`.
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Uses battle-tested AWS signing implementation
|
||||
- Keeps credential chain resolution (not needed for this use case)
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Still pulls in ~20-30 packages (many from `@smithy/*`)
|
||||
- Doesn't fully address the supply-chain concern
|
||||
- Over-engineered for static-credential use case
|
||||
- Adds credential resolution features we don't use (IMDS, SSO, STS, profiles)
|
||||
|
||||
**Dependency surface after change:** ~20-30 npm packages
|
||||
|
||||
---
|
||||
|
||||
### Route D: S3 presigned URLs (if supported by Hetzner Object Storage)
|
||||
|
||||
Instead of SigV4 signing at request time, generate a presigned URL for the ListObjectsV2 operation and fetch it with a simple HTTP GET.
|
||||
|
||||
**Pros:**
|
||||
|
||||
- No signing code needed in the extension
|
||||
- Could be pre-generated and stored as an env var
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Presigned URLs expire; need regeneration mechanism
|
||||
- Hetzner Object Storage may not support presigned URLs for ListObjectsV2
|
||||
- Pagination with presigned URLs is awkward (each page needs a new URL)
|
||||
- Doesn't eliminate dependencies — just shifts the signing to another tool
|
||||
|
||||
## Final decision
|
||||
|
||||
**Route B (`aws4` + Node.js built-ins) was chosen for implementation (see task ML-173).**
|
||||
|
||||
Route A (full zero-dependency) was the initial recommendation. However, on closer analysis, `aws4` was confirmed to have **0 transitive dependencies** (not ~3 as initially estimated), making it equally minimal in supply-chain terms as Route A. Given this, Route B was preferred because it eliminates all SigV4 signing implementation risk at zero additional supply-chain cost.
|
||||
|
||||
The `aws4` package is a single-file, 200-line, battle-tested library that hasn't needed an update in 2 years. The implementation bridges `aws4`'s `http.request`-style output to `fetch` by extracting signed headers and constructing the full URL — a thin wrapper of ~10-15 lines.
|
||||
@@ -0,0 +1,308 @@
|
||||
---
|
||||
id: ML-173
|
||||
title: remove deps from s3-browser extension
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: "2026-05-09 08:29"
|
||||
updated_date: "2026-05-09 08:48"
|
||||
labels:
|
||||
- pi-extension
|
||||
- security
|
||||
- s3-browser
|
||||
dependencies: []
|
||||
references:
|
||||
- .pi/extensions/s3-browser/index.ts
|
||||
- .pi/extensions/s3-browser/package.json
|
||||
- doc-15 - Research-Removing-dependencies-from-s3-browser-extension
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
|
||||
The s3-browser pi extension has only one API integration, which is to be able to fetch the list of files from the target S3 bucket. To do that, it pulls `@aws-sdk/client-s3` as its sole dependency, which brings in a large dependency tree (~150+ packages including @aws-sdk/_, @aws-crypto/_, @smithy/\*, fast-xml-parser, tslib, etc.). This unnecessarily exposes the development environment to supply-chain attacks.
|
||||
|
||||
The extension needs to be re-written to use a custom, no-dependencies API client that directly calls the S3-compatible REST API using only Node.js built-in modules. The only API operation needed is ListObjectsV2.
|
||||
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
<!-- AC:BEGIN -->
|
||||
|
||||
- [ ] #1 `@aws-sdk/client-s3` is removed from package.json dependencies
|
||||
- [ ] #2 `aws4` is the only runtime dependency with 0 transitive dependencies
|
||||
- [ ] #3 `/backups` command lists S3 backup files with correct sizes, dates, and sorting
|
||||
- [ ] #4 Pagination handles buckets with >1000 objects (multiple pages via ContinuationToken)
|
||||
- [ ] #5 Abort/cancel during loading (escape key) produces clean cancellation with no unhandled rejections
|
||||
- [ ] #6 Missing or invalid `LITESTREAM_ACCESS_KEY_ID`/`LITESTREAM_SECRET_ACCESS_KEY` shows a clear error notification
|
||||
- [ ] #7 Network errors (unreachable endpoint, timeout) show a clear error, not an unhandled exception
|
||||
- [ ] #8 TypeScript compiles without errors (`npx tsc --noEmit`)
|
||||
- [ ] #9 After clean `npm install`, only `aws4` appears in `npm ls --all` (no transitive or phantom dependencies)
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
|
||||
## Objective alignment
|
||||
|
||||
Replace `@aws-sdk/client-s3` (~150 transitive npm dependencies) with `aws4` (0 transitive dependencies) + Node.js built-in modules (`crypto`, `fetch`) for the single API operation the extension needs: S3 ListObjectsV2 against Hetzner Object Storage. The extension continues to function identically — same command (`/backups`), same UI, same behavior — while reducing the supply-chain attack surface to a single, auditable, single-file dependency.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Route A (full zero-dependency):** Rejected. Implementing SigV4 manually introduces signing bug risk for marginal security gain over `aws4`, which is a single-file, 0-dep, battle-tested package that hasn't needed an update in 2 years. The research document (doc-15) initially recommended Route A; this plan overrides that recommendation because on closer analysis `aws4` has 0 transitive dependencies (not ~3 as initially estimated), making it equally minimal in supply-chain terms while eliminating all signing implementation risk.
|
||||
- **Route C (AWS SDK minimal subset):** Rejected. Still pulls ~20-30 packages from `@smithy/*`; doesn't meaningfully address the supply-chain concern.
|
||||
- **Route D (presigned URLs):** Rejected. Presigned URLs expire, don't work well with pagination, and shift the signing problem rather than solving it.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
### Step 1: Replace dependency in package.json
|
||||
|
||||
Remove `@aws-sdk/client-s3`, add `aws4` at latest version, run `npm install` in `.pi/extensions/s3-browser/`.
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `npm ls aws4 --depth=0` shows `aws4` with 0 dependencies
|
||||
- `npm ls @aws-sdk/client-s3` returns empty/unmet
|
||||
- `du -sh node_modules` shows dramatically reduced size
|
||||
|
||||
### Step 2: Create custom S3 client module
|
||||
|
||||
Create `.pi/extensions/s3-browser/s3-client.ts` with:
|
||||
|
||||
- `signRequest()` — wraps `aws4.sign()` with S3-compatible options (path-style endpoint, unsigned payload with `x-amz-content-sha256: UNSIGNED-PAYLOAD` for simple GETs). **Important**: `aws4.sign()` produces options shaped for Node.js `http.request` (`{host, path, headers}`). The wrapper extracts the signed `headers` object from the returned options, constructs the full URL as `https://${host}${path}`, and returns `{url, headers}` ready to pass to `fetch`.
|
||||
- `listObjectsV2()` — builds the signed GET request via `signRequest()`, calls `fetch`, supports `AbortSignal`. Categorizes errors distinctly so the UI can show targeted messages per acceptance criterion #7: `AuthError` (HTTP 401/403 — bad credentials), `NotFoundError` (HTTP 404 — wrong bucket/endpoint), `NetworkError` (fetch rejection — unreachable endpoint, DNS failure, timeout), `AbortError` (user cancelled), `ServerError` (HTTP 5xx/other unexpected status).
|
||||
- `parseListObjectsV2Response()` — regex-based XML extraction of `<Contents>` blocks (Key, Size, LastModified), `<IsTruncated>`, `<NextContinuationToken>`. **Must handle namespace-qualified XML** (Hetzner may return `<Contents xmlns="http://s3.amazonaws.com/doc/2006-03-01/">`) — use namespace-tolerant patterns such as `/<Contents[^>]*>([\s\S]*?)<\/Contents>/g`. Validate against a captured real Hetzner ListObjectsV2 XML response, not a hand-crafted fixture.
|
||||
- Exported types: `S3ClientConfig`, `ListObjectsV2Params`, `ListObjectsV2Response`, `S3Object`, `S3Error` (union of error categories above)
|
||||
|
||||
**Testing approach**: Use Node.js built-in `node:test` runner (zero-dependency, available since Node 18). Create `.pi/extensions/s3-browser/s3-client.test.ts`. Tests:
|
||||
|
||||
- `signRequest` produces well-formed Authorization header (AWS4-HMAC-SHA256 format with correct credential scope and signed headers)
|
||||
- `parseListObjectsV2Response` correctly parses a captured real Hetzner ListObjectsV2 XML response (store as test fixture)
|
||||
- Empty response (no `<Contents>` blocks), truncated response (`<IsTruncated>true</IsTruncated>` + `<NextContinuationToken>`), and error response (`<Error><Code>...</Code><Message>...</Message></Error>`) all handled
|
||||
- Aborting mid-fetch via `AbortController` produces a clean `AbortError` rejection with no unhandled promise rejections
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `node --test s3-client.test.ts` passes all tests
|
||||
- `npx tsc --noEmit` in extension directory compiles without errors
|
||||
|
||||
### Step 3: Integrate into index.ts
|
||||
|
||||
- Remove `import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3"`
|
||||
- Add `import { listObjectsV2 } from "./s3-client"`
|
||||
- Replace `S3Client` construction + `client.send(command, ...)` with direct `listObjectsV2(config, params, signal)` call
|
||||
- `listAllObjects` function signature changes from `(client: S3Client, signal?)` to `(config: S3ClientConfig, signal?)`
|
||||
- **Sorting logic (size descending, then key descending) remains in `listAllObjects` in `index.ts` unchanged** — the client module is a pure data fetcher with no sorting responsibility
|
||||
- UI code (BorderedLoader, SelectList, Container, etc.) remains untouched
|
||||
- Update error handling in the `doFetch` catch block to inspect the new error categories and show distinct notifications (e.g., "Authentication failed — check LITESTREAM_ACCESS_KEY_ID / LITESTREAM_SECRET_ACCESS_KEY" for AuthError, "Could not reach S3 endpoint — check network" for NetworkError)
|
||||
|
||||
**Verify:**
|
||||
|
||||
- TypeScript compiles without errors: `npx tsc --noEmit` in extension directory
|
||||
- All existing exports remain unchanged (no breaking changes for pi agent loader)
|
||||
|
||||
### Step 4: Integration test with real credentials
|
||||
|
||||
- Run `/backups` command in pi agent
|
||||
- Confirm file list loads correctly (sizes, dates, sorting all match previous behavior)
|
||||
- Navigate/select files in the list — confirm same behavior
|
||||
- Test abort: press escape during loading — confirm clean cancellation with no unhandled rejections
|
||||
- If bucket has >1000 objects, confirm pagination works across multiple pages
|
||||
- Test with invalid credentials: unset env vars → confirm clear error notification
|
||||
- Test with wrong endpoint: set endpoint to unreachable host → confirm NetworkError notification
|
||||
|
||||
**Verify:**
|
||||
|
||||
- Output visually matches the current `@aws-sdk/client-s3` version
|
||||
- No console errors from the extension
|
||||
- `tidewave_get_logs` shows no error-level logs from the extension
|
||||
|
||||
### Step 5: Clean install verification
|
||||
|
||||
- Remove `node_modules/` and `package-lock.json` from `.pi/extensions/s3-browser/`
|
||||
- Run `npm install`
|
||||
- Confirm only `aws4` is installed (no transitive deps)
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `npm ls --all` shows only `aws4`
|
||||
- `/backups` still works after clean install
|
||||
|
||||
## Architecture impact
|
||||
|
||||
| Touchpoint | Impact |
|
||||
| --------------------------------------------- | ------------------------------------------------ |
|
||||
| `.pi/extensions/s3-browser/index.ts` | Modified: replace SDK imports with custom client |
|
||||
| `.pi/extensions/s3-browser/s3-client.ts` | **New file**: custom S3 client (~100 lines) |
|
||||
| `.pi/extensions/s3-browser/s3-client.test.ts` | **New file**: unit tests for s3-client |
|
||||
| `.pi/extensions/s3-browser/package.json` | Modified: dependency swap |
|
||||
| `.pi/extensions/s3-browser/package-lock.json` | Regenerated |
|
||||
| Pi agent extension loader | No impact (extension interface unchanged) |
|
||||
| Other pi extensions | No impact (isolated per-extension node_modules) |
|
||||
| Elixir app, schemas, contexts, routes, PubSub | No impact (pi extension, not server-side) |
|
||||
| Environment variables | No impact (same `LITESTREAM_*` vars) |
|
||||
|
||||
## Performance profile
|
||||
|
||||
- **HTTP:** `fetch` (Node built-in) — equivalent to SDK's `@smithy/fetch-http-handler`
|
||||
- **Signing:** `aws4.sign()` — O(1) per request, single HMAC chain, <1ms overhead
|
||||
- **XML parsing:** Regex-based, O(n) where n = response body size. ListObjectsV2 returns max 1000 objects per page (<100KB XML typically). Parsing is sub-millisecond.
|
||||
- **Pagination:** Same pattern as current — sequential pages via ContinuationToken. No concurrent requests.
|
||||
- **Memory:** No buffering beyond the single response body per page. No streaming needed (XML response is small).
|
||||
- **N+1 risk:** None — single HTTP call per page, no JOIN-style patterns.
|
||||
|
||||
## Benchmarking requirements
|
||||
|
||||
No ongoing benchmarks needed. This is a human-triggered, infrequent operation (someone typing `/backups`). One-off validation: ensure listing ~100 objects completes in <5 seconds (current behavior).
|
||||
|
||||
## Cost profile
|
||||
|
||||
No cost impact. No paid API calls (Hetzner Object Storage already provisioned). No additional services, compute, or storage. The extension runs entirely on the developer's machine.
|
||||
|
||||
## Production Changes
|
||||
|
||||
None. The pi extension runs locally in the developer's pi agent. No server-side deployments, environment variable changes, database migrations, DNS changes, or infrastructure provisioning are needed.
|
||||
|
||||
## Documentation
|
||||
|
||||
After implementation, update these documents:
|
||||
|
||||
1. **doc-15 (research document)**: Update recommendation section to note the final decision (Route B chosen) and correct the dependency count (`aws4` has 0 transitive deps, not ~3 as initially estimated)
|
||||
2. **`docs/architecture.md`**: If it references the s3-browser extension or its dependencies, check for staleness
|
||||
3. **AGENTS.md / `docs/project-conventions.md`**: If pi extension dependency patterns are documented, update to reflect the new minimal-dependency convention
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
|
||||
## Objective alignment
|
||||
|
||||
Replace `@aws-sdk/client-s3` (~150 transitive npm dependencies) with `aws4` (0 transitive dependencies) + Node.js built-in modules (`crypto`, `fetch`) for the single API operation the extension needs: S3 ListObjectsV2 against Hetzner Object Storage. The extension continues to function identically — same command (`/backups`), same UI, same behavior — while reducing the supply-chain attack surface to a single, auditable, single-file dependency.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Route A (full zero-dependency):** Rejected. Implementing SigV4 manually introduces signing bug risk for marginal security gain over `aws4`, which is a single-file, 0-dep, battle-tested package that hasn't needed an update in 2 years. The research document (doc-15) initially recommended Route A; this plan overrides that recommendation because on closer analysis `aws4` has 0 transitive dependencies (not ~3 as initially estimated), making it equally minimal in supply-chain terms while eliminating all signing implementation risk.
|
||||
- **Route C (AWS SDK minimal subset):** Rejected. Still pulls ~20-30 packages from `@smithy/*`; doesn't meaningfully address the supply-chain concern.
|
||||
- **Route D (presigned URLs):** Rejected. Presigned URLs expire, don't work well with pagination, and shift the signing problem rather than solving it.
|
||||
|
||||
## Implementation steps
|
||||
|
||||
### Step 1: Replace dependency in package.json
|
||||
|
||||
Remove `@aws-sdk/client-s3`, add `aws4` at latest version, run `npm install` in `.pi/extensions/s3-browser/`.
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `npm ls aws4 --depth=0` shows `aws4` with 0 dependencies
|
||||
- `npm ls @aws-sdk/client-s3` returns empty/unmet
|
||||
- `du -sh node_modules` shows dramatically reduced size
|
||||
|
||||
### Step 2: Create custom S3 client module
|
||||
|
||||
Create `.pi/extensions/s3-browser/s3-client.ts` with:
|
||||
|
||||
- `signRequest()` — wraps `aws4.sign()` with S3-compatible options (path-style endpoint, unsigned payload with `x-amz-content-sha256: UNSIGNED-PAYLOAD` for simple GETs). **Important**: `aws4.sign()` produces options shaped for Node.js `http.request` (`{host, path, headers}`). The wrapper extracts the signed `headers` object from the returned options, constructs the full URL as `https://${host}${path}`, and returns `{url, headers}` ready to pass to `fetch`.
|
||||
- `listObjectsV2()` — builds the signed GET request via `signRequest()`, calls `fetch`, supports `AbortSignal`. Categorizes errors distinctly so the UI can show targeted messages per acceptance criterion #7: `AuthError` (HTTP 401/403 — bad credentials), `NotFoundError` (HTTP 404 — wrong bucket/endpoint), `NetworkError` (fetch rejection — unreachable endpoint, DNS failure, timeout), `AbortError` (user cancelled), `ServerError` (HTTP 5xx/other unexpected status).
|
||||
- `parseListObjectsV2Response()` — regex-based XML extraction of `<Contents>` blocks (Key, Size, LastModified), `<IsTruncated>`, `<NextContinuationToken>`. **Must handle namespace-qualified XML** (Hetzner may return `<Contents xmlns="http://s3.amazonaws.com/doc/2006-03-01/">`) — use namespace-tolerant patterns such as `/<Contents[^>]*>([\s\S]*?)<\/Contents>/g`. Validate against a captured real Hetzner ListObjectsV2 XML response, not a hand-crafted fixture.
|
||||
- Exported types: `S3ClientConfig`, `ListObjectsV2Params`, `ListObjectsV2Response`, `S3Object`, `S3Error` (union of error categories above)
|
||||
|
||||
**Testing approach**: Use Node.js built-in `node:test` runner (zero-dependency, available since Node 18). Create `.pi/extensions/s3-browser/s3-client.test.ts`. Tests:
|
||||
|
||||
- `signRequest` produces well-formed Authorization header (AWS4-HMAC-SHA256 format with correct credential scope and signed headers)
|
||||
- `parseListObjectsV2Response` correctly parses a captured real Hetzner ListObjectsV2 XML response (store as test fixture)
|
||||
- Empty response (no `<Contents>` blocks), truncated response (`<IsTruncated>true</IsTruncated>` + `<NextContinuationToken>`), and error response (`<Error><Code>...</Code><Message>...</Message></Error>`) all handled
|
||||
- Aborting mid-fetch via `AbortController` produces a clean `AbortError` rejection with no unhandled promise rejections
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `node --test s3-client.test.ts` passes all tests
|
||||
- `npx tsc --noEmit` in extension directory compiles without errors
|
||||
|
||||
### Step 3: Integrate into index.ts
|
||||
|
||||
- Remove `import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3"`
|
||||
- Add `import { listObjectsV2 } from "./s3-client"`
|
||||
- Replace `S3Client` construction + `client.send(command, ...)` with direct `listObjectsV2(config, params, signal)` call
|
||||
- `listAllObjects` function signature changes from `(client: S3Client, signal?)` to `(config: S3ClientConfig, signal?)`
|
||||
- **Sorting logic (size descending, then key descending) remains in `listAllObjects` in `index.ts` unchanged** — the client module is a pure data fetcher with no sorting responsibility
|
||||
- UI code (BorderedLoader, SelectList, Container, etc.) remains untouched
|
||||
- Update error handling in the `doFetch` catch block to inspect the new error categories and show distinct notifications (e.g., "Authentication failed — check LITESTREAM_ACCESS_KEY_ID / LITESTREAM_SECRET_ACCESS_KEY" for AuthError, "Could not reach S3 endpoint — check network" for NetworkError)
|
||||
|
||||
**Verify:**
|
||||
|
||||
- TypeScript compiles without errors: `npx tsc --noEmit` in extension directory
|
||||
- All existing exports remain unchanged (no breaking changes for pi agent loader)
|
||||
|
||||
### Step 4: Integration test with real credentials
|
||||
|
||||
- Run `/backups` command in pi agent
|
||||
- Confirm file list loads correctly (sizes, dates, sorting all match previous behavior)
|
||||
- Navigate/select files in the list — confirm same behavior
|
||||
- Test abort: press escape during loading — confirm clean cancellation with no unhandled rejections
|
||||
- If bucket has >1000 objects, confirm pagination works across multiple pages
|
||||
- Test with invalid credentials: unset env vars → confirm clear error notification
|
||||
- Test with wrong endpoint: set endpoint to unreachable host → confirm NetworkError notification
|
||||
|
||||
**Verify:**
|
||||
|
||||
- Output visually matches the current `@aws-sdk/client-s3` version
|
||||
- No console errors from the extension
|
||||
- `tidewave_get_logs` shows no error-level logs from the extension
|
||||
|
||||
### Step 5: Clean install verification
|
||||
|
||||
- Remove `node_modules/` and `package-lock.json` from `.pi/extensions/s3-browser/`
|
||||
- Run `npm install`
|
||||
- Confirm only `aws4` is installed (no transitive deps)
|
||||
|
||||
**Verify:**
|
||||
|
||||
- `npm ls --all` shows only `aws4`
|
||||
- `/backups` still works after clean install
|
||||
|
||||
## Architecture impact
|
||||
|
||||
| Touchpoint | Impact |
|
||||
| --------------------------------------------- | ------------------------------------------------ |
|
||||
| `.pi/extensions/s3-browser/index.ts` | Modified: replace SDK imports with custom client |
|
||||
| `.pi/extensions/s3-browser/s3-client.ts` | **New file**: custom S3 client (~100 lines) |
|
||||
| `.pi/extensions/s3-browser/s3-client.test.ts` | **New file**: unit tests for s3-client |
|
||||
| `.pi/extensions/s3-browser/package.json` | Modified: dependency swap |
|
||||
| `.pi/extensions/s3-browser/package-lock.json` | Regenerated |
|
||||
| Pi agent extension loader | No impact (extension interface unchanged) |
|
||||
| Other pi extensions | No impact (isolated per-extension node_modules) |
|
||||
| Elixir app, schemas, contexts, routes, PubSub | No impact (pi extension, not server-side) |
|
||||
| Environment variables | No impact (same `LITESTREAM_*` vars) |
|
||||
|
||||
## Performance profile
|
||||
|
||||
- **HTTP:** `fetch` (Node built-in) — equivalent to SDK's `@smithy/fetch-http-handler`
|
||||
- **Signing:** `aws4.sign()` — O(1) per request, single HMAC chain, <1ms overhead
|
||||
- **XML parsing:** Regex-based, O(n) where n = response body size. ListObjectsV2 returns max 1000 objects per page (<100KB XML typically). Parsing is sub-millisecond.
|
||||
- **Pagination:** Same pattern as current — sequential pages via ContinuationToken. No concurrent requests.
|
||||
- **Memory:** No buffering beyond the single response body per page. No streaming needed (XML response is small).
|
||||
- **N+1 risk:** None — single HTTP call per page, no JOIN-style patterns.
|
||||
|
||||
## Benchmarking requirements
|
||||
|
||||
No ongoing benchmarks needed. This is a human-triggered, infrequent operation (someone typing `/backups`). One-off validation: ensure listing ~100 objects completes in <5 seconds (current behavior).
|
||||
|
||||
## Cost profile
|
||||
|
||||
No cost impact. No paid API calls (Hetzner Object Storage already provisioned). No additional services, compute, or storage. The extension runs entirely on the developer's machine.
|
||||
|
||||
## Production Changes
|
||||
|
||||
None. The pi extension runs locally in the developer's pi agent. No server-side deployments, environment variable changes, database migrations, DNS changes, or infrastructure provisioning are needed.
|
||||
|
||||
## Documentation
|
||||
|
||||
After implementation, update these documents:
|
||||
|
||||
1. **doc-15 (research document)**: Update recommendation section to note the final decision (Route B chosen) and correct the dependency count (`aws4` has 0 transitive deps, not ~3 as initially estimated)
|
||||
2. **`docs/architecture.md`**: If it references the s3-browser extension or its dependencies, check for staleness
|
||||
3. **AGENTS.md / `docs/project-conventions.md`**: If pi extension dependency patterns are documented, update to reflect the new minimal-dependency convention
|
||||
<!-- SECTION:NOTES:END -->
|
||||
Reference in New Issue
Block a user