docs: add rated engineering feature matrix for mpiper

references/ENGINEERING_FEATURE_MATRIX.md

@@ -0,0 +1,123 @@
+# MPiper — Engineering Feature Matrix
+
+A rated roadmap of features that would move MPiper from a working demo pipeline to
+a defensible, production-grade engineering project. Each row is scored on the
+engineering value it proves (not just product value), and prioritized.
+
+**Rating scale (1–5):** how much real engineering depth the feature forces —
+correctness, reliability, security, testing, operability. A 5 is a feature that
+cannot be faked and demonstrates serious systems thinking.
+
+**Priority:** P0 = blocks production credibility / fixes a latent defect,
+P1 = high leverage, P2 = nice-to-have.
+
+---
+
+## 0. Foundations — latent defects found in review (fix first)
+
+These are not "features" — they are correctness gaps that undermine everything
+built on top of them. Do these before any feature work.
+
+| # | Item | Evidence | Why it matters |
+|---|------|----------|----------------|
+| 0.1 | **Single migration source of truth** — schema drift | The server/worker auto-run `internal/database/migrations/` (golang-migrate, `migrate.go`), but `db/migrations/001_seed.sql` is the documented/manual schema and they **diverge**. The embedded `variants.image` has PK `(asset_id, role)` and **no** `variant_hash`/`content_hash`/`params` columns; the worker inserts into those columns (`images.py`, `ensure_variant_exists`). Embedded `assets` lacks `width`/`height`/`duration_seconds`, which `images.py` writes. | The pipeline would fail at runtime against the auto-run schema. One canonical, versioned migration set (embed-only) is required; the seed file becomes a generated artifact. |
+| 0.2 | **Integration tests that actually run** | `ci.yml` runs `go test -tags=integration ./...`, but **no** test file carries `//go:build integration` — the job executes zero tests. | A green CI "integration" job proves nothing. Stand up MinIO + Postgres + Redis in CI and run a real upload→process→variant assertion. |
+| 0.3 | **Fix latent CI unit-test failure** | `tests/performance_suite_test.go` has no build tag and `t.Fatal`s when `PERF_TEST_URL` is unset; the unit-tests job does not set it. | `go test ./...` in CI would fail. Tag the perf test (`//go:build perf`) or gate it on the env var. |
+| 0.4 | **Auth is self-made and incomplete** | Token = AES-GCM-encrypted userID string (`pkg/utils/crypt.go`); no users table, no login/register, no expiry/rotation, no roles, no scopes. | There is no way to issue or revoke a token today. Homegrown crypto is a liability. See feature 3.1. |
+
+---
+
+## 1. Reliability & correctness
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **Transactional outbox for job dispatch** | `MarkAssetUploaded` commits the job row, then enqueues to Redis **after** commit — a crash between the two loses the job (recovery only re-scans `pending` rows, which never got inserted). | Proves you understand at-least-once delivery, idempotency, and the dual-write problem. An outbox table + relay is the textbook fix. | M | P0 | 5 |
+| **Dead-letter queue + poison-message handling** | Retry cap exists (`MAX_JOB_ATTEMPTS`), but failed jobs just stop; no DLQ, no quarantine, no operator visibility. | Shows mature failure taxonomy: retryable vs fatal, quarantine, alerting. | M | P1 | 4 |
+| **Variant/orphan GC** | Variants are content-addressed and immutable (good), but nothing ever deletes orphaned storage objects or unlinked variants. | Demonstrates lifecycle management beyond "write once." | M | P2 | 3 |
+| **Webhook delivery engine** | Schema only (`000002_webhooks`); no dispatcher, no HMAC signing, no retry/backoff, no replay, no dead-letter. | A real webhook engine (signing, exponential backoff, idempotency keys, delivery receipts) is a classic hard problem that shows production maturity. | L | P1 | 5 |
+| **Graceful shutdown / drain** | `/healthz` exists; no readiness gating, no worker drain on SIGTERM. | Signals you care about zero-downtime deploys and in-flight job safety. | S | P1 | 3 |
+
+## 2. API & product surface
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **Asset read/list/variants API** | Only create/presign + mark-uploaded exist. There is no way to fetch an asset or its variants. | Forces pagination, filtering, projection, and a real read model — the core of any API. | M | P1 | 4 |
+| **Batch processing API** | Roadmap item, absent. | Batch + async job tracking + status endpoints; demonstrates workflow orchestration. | L | P2 | 4 |
+| **Real-time status (SSE/WebSockets)** | Roadmap item, absent. | Proves you can do long-lived connections, backpressure, and reconnect semantics. | M | P2 | 4 |
+| **Idempotency keys on upload** | None. | Client retries currently create duplicate assets. An `Idempotency-Key` header + dedup is a concrete correctness feature. | S | P2 | 3 |
+| **Asset delete / soft-delete** | None. | Adds lifecycle + storage cleanup + authorization surface. | S | P2 | 2 |
+| **Admin dashboard** | Roadmap item, absent. | Mostly product value; low *engineering* proof unless it adds real ops tooling. | XL | P2 | 2 |
+
+## 3. Security
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **Real identity: users, login, JWT/refresh, RBAC** | Homegrown AES token; no identity model. | Forces you to reason about password hashing (bcrypt exists), token expiry/rotation, refresh flows, and least-privilege authz. Highest-leverage security gap. | L | P0 | 5 |
+| **API keys / scoped service tokens** | None. | Machine-to-machine auth with scopes + revocation; separates human vs service identity. | M | P1 | 4 |
+| **Webhook HMAC signing** | No delivery engine at all (see 1.x). | Signing webhook payloads (HMAC-SHA256) is the standard, testable security feature. | M | P1 | 4 |
+| **Signed GET URLs for serving + CDN** | Presigned PUT exists; no signed GET, no CDN. | Enables private content delivery and cache invalidation — production serving. | M | P2 | 3 |
+| **Rate limiting on all endpoints** | Per-IP limiter on `/presign` only. | Extend to a token-bucket/IP limiter middleware across the API; proves abuse-resistance thinking. | S | P2 | 3 |
+| **Secrets management** | Env-file only; `ENCRYPTION_KEY` in env. | Wire Vault / k8s secrets / cloud KMS references; proves you treat keys as infrastructure. | M | P2 | 3 |
+
+## 4. Media pipeline depth
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **AVIF / advanced image optimization** | WebP variants done (Pillow); AVIF absent. | AVIF needs a real encoder (libvips/cwebp/avif) — shows codec/quality/size tradeoff engineering. | M | P2 | 3 |
+| **HLS/DASH adaptive streaming** | `manifest_url` column exists; transcoding is single 720p MP4 + poster + preview only. | Segmenting, playlists, multiple renditions, and CDN-friendly delivery — a genuinely hard, impressive feature. | XL | P1 | 5 |
+| **Replace Pillow with libvips/Sharp** | Pillow is CPU-bound and slow at scale. | Proves performance engineering (memory, throughput, concurrency) on the hot path. | L | P2 | 4 |
+| **Azure Blob Storage provider** | Roadmap item; GCS + S3 done behind `StorageX`. | Adding a third provider validates the abstraction is truly provider-agnostic. | M | P2 | 3 |
+
+## 5. Observability & operability
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **Unified OTel tracing across Go + worker** | API is OTel; worker uses `prometheus_client` only — no trace propagation across the Redis boundary. | Cross-service trace context (W3C traceparent through the stream) is the gold standard for distributed tracing. | M | P1 | 4 |
+| **SLIs/SLOs + error budgets** | Metrics exist; no SLO definitions or budgets. | Forces you to define what "healthy" means and alert on the budget, not just raw metrics. | M | P1 | 4 |
+| **Structured alerting rules (Prometheus)** | Prometheus/Grafana configs exist; no alert rules. | Alerting on latency/error/throughput with proper thresholds and runbooks. | S | P2 | 3 |
+| **Load/soak testing harness** | `performance_suite_test.go` is a naive 200-request loop, not a real load tool. | k6/locust with SLO assertions gives defensible capacity numbers. | M | P2 | 3 |
+
+## 6. DX, testing & delivery
+
+| Feature | Current state | Engineering value | Effort | Priority | Score |
+|---------|---------------|-------------------|--------|----------|-------|
+| **E2E pipeline test (MinIO+PG+Redis)** | No harness runs upload→process→variant. | The single most valuable test: proves the whole system works together and would have caught 0.1. | M | P0 | 5 |
+| **Contract tests / OpenAPI spec** | No OpenAPI; README-only docs. | Generated, versioned API contract; enables client generation and contract testing. | M | P2 | 3 |
+| **Fuzz/property tests (Go)** | Unit tests only (config, auth, logging, mime, s3). | Fuzzing the JSON parsing / token / hash paths finds real bugs. | S | P2 | 3 |
+| **Worker test coverage expansion** | 4 Python test files (retry, recovery, image, dispatch). | Add DB-backed and storage-mocked tests for dedup + clone paths. | M | P2 | 3 |
+| **Helm chart / Terraform** | Raw k8s manifests (good: pdb, resource-quota, rbac, secrets). | Helm packaging + IaC for environments proves reproducible deploys. | L | P2 | 3 |
+| **ADRs / architecture docs** | README + references only. | Records decisions (why Redis transport-only, why content-addressing) for future engineers. | S | P2 | 2 |
+
+---
+
+## Recommended build order
+
+**Phase 1 — make it trustworthy (P0):**
+1. Single canonical migration set (0.1) — unblocks everything.
+2. E2E pipeline test with MinIO+PG+Redis (6) — catches the schema drift.
+3. Fix CI: tag the perf test (0.3), wire real integration tests (0.2).
+4. Transactional outbox for job dispatch (1).
+
+**Phase 2 — make it a real product (P1):**
+5. Real identity + auth (3.1) and API keys (3.2).
+6. Asset read/list/variants API (2).
+7. Webhook delivery engine + HMAC signing (1 + 3.3).
+8. Cross-service tracing (5) + SLIs/SLOs (5).
+
+**Phase 3 — make it impressive (P2):**
+9. HLS/DASH adaptive streaming (4).
+10. Batch API + real-time status (2).
+11. libvips migration, AVIF, Azure provider, DLQ, GC (4/1).
+
+## What the project already proves (strengths)
+
+- Layered Go service (handler/service/repo/models) with typed error hierarchy mapped to HTTP status.
+- Provider-agnostic storage abstraction (GCS + S3/MinIO) with presigned URLs, metrics, tracing.
+- Content-addressed, immutable variants with cross-asset dedup (`canonical_asset_id`).
+- Redis Streams consumer group with idempotency, retry classification, and periodic stuck-job recovery.
+- Full observability stack (OTel + Prometheus + Grafana/Tempo/Loki) and a real CI/CD + release pipeline (staging/LTS, GHCR).
+- Thoughtful k8s manifests (PDB, resource quotas, RBAC, secrets, migration job).
+
+These are the marks of a solid mid-level pipeline. The features above are what separate
+"a working pipeline" from "an engineering project" — correctness under failure,
+real security, a read model, and tests that actually run.