From 6bf1baad94425375333a7935806befc13fb13a7f Mon Sep 17 00:00:00 2001 From: "usehoplite[bot]" <288093033+usehoplite[bot]@users.noreply.github.com> Date: Wed, 2 Sep 2026 21:58:02 +0000 Subject: [PATCH] docs: add rated engineering feature matrix for mpiper Co-authored-by: Shantanu Mane --- references/ENGINEERING_FEATURE_MATRIX.md | 123 +++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 references/ENGINEERING_FEATURE_MATRIX.md diff --git a/references/ENGINEERING_FEATURE_MATRIX.md b/references/ENGINEERING_FEATURE_MATRIX.md new file mode 100644 index 0000000..e8f162f --- /dev/null +++ b/references/ENGINEERING_FEATURE_MATRIX.md @@ -0,0 +1,123 @@ +# MPiper — Engineering Feature Matrix + +A rated roadmap of features that would move MPiper from a working demo pipeline to +a defensible, production-grade engineering project. Each row is scored on the +engineering value it proves (not just product value), and prioritized. + +**Rating scale (1–5):** how much real engineering depth the feature forces — +correctness, reliability, security, testing, operability. A 5 is a feature that +cannot be faked and demonstrates serious systems thinking. + +**Priority:** P0 = blocks production credibility / fixes a latent defect, +P1 = high leverage, P2 = nice-to-have. + +--- + +## 0. Foundations — latent defects found in review (fix first) + +These are not "features" — they are correctness gaps that undermine everything +built on top of them. Do these before any feature work. + +| # | Item | Evidence | Why it matters | +|---|------|----------|----------------| +| 0.1 | **Single migration source of truth** — schema drift | The server/worker auto-run `internal/database/migrations/` (golang-migrate, `migrate.go`), but `db/migrations/001_seed.sql` is the documented/manual schema and they **diverge**. The embedded `variants.image` has PK `(asset_id, role)` and **no** `variant_hash`/`content_hash`/`params` columns; the worker inserts into those columns (`images.py`, `ensure_variant_exists`). Embedded `assets` lacks `width`/`height`/`duration_seconds`, which `images.py` writes. | The pipeline would fail at runtime against the auto-run schema. One canonical, versioned migration set (embed-only) is required; the seed file becomes a generated artifact. | +| 0.2 | **Integration tests that actually run** | `ci.yml` runs `go test -tags=integration ./...`, but **no** test file carries `//go:build integration` — the job executes zero tests. | A green CI "integration" job proves nothing. Stand up MinIO + Postgres + Redis in CI and run a real upload→process→variant assertion. | +| 0.3 | **Fix latent CI unit-test failure** | `tests/performance_suite_test.go` has no build tag and `t.Fatal`s when `PERF_TEST_URL` is unset; the unit-tests job does not set it. | `go test ./...` in CI would fail. Tag the perf test (`//go:build perf`) or gate it on the env var. | +| 0.4 | **Auth is self-made and incomplete** | Token = AES-GCM-encrypted userID string (`pkg/utils/crypt.go`); no users table, no login/register, no expiry/rotation, no roles, no scopes. | There is no way to issue or revoke a token today. Homegrown crypto is a liability. See feature 3.1. | + +--- + +## 1. Reliability & correctness + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **Transactional outbox for job dispatch** | `MarkAssetUploaded` commits the job row, then enqueues to Redis **after** commit — a crash between the two loses the job (recovery only re-scans `pending` rows, which never got inserted). | Proves you understand at-least-once delivery, idempotency, and the dual-write problem. An outbox table + relay is the textbook fix. | M | P0 | 5 | +| **Dead-letter queue + poison-message handling** | Retry cap exists (`MAX_JOB_ATTEMPTS`), but failed jobs just stop; no DLQ, no quarantine, no operator visibility. | Shows mature failure taxonomy: retryable vs fatal, quarantine, alerting. | M | P1 | 4 | +| **Variant/orphan GC** | Variants are content-addressed and immutable (good), but nothing ever deletes orphaned storage objects or unlinked variants. | Demonstrates lifecycle management beyond "write once." | M | P2 | 3 | +| **Webhook delivery engine** | Schema only (`000002_webhooks`); no dispatcher, no HMAC signing, no retry/backoff, no replay, no dead-letter. | A real webhook engine (signing, exponential backoff, idempotency keys, delivery receipts) is a classic hard problem that shows production maturity. | L | P1 | 5 | +| **Graceful shutdown / drain** | `/healthz` exists; no readiness gating, no worker drain on SIGTERM. | Signals you care about zero-downtime deploys and in-flight job safety. | S | P1 | 3 | + +## 2. API & product surface + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **Asset read/list/variants API** | Only create/presign + mark-uploaded exist. There is no way to fetch an asset or its variants. | Forces pagination, filtering, projection, and a real read model — the core of any API. | M | P1 | 4 | +| **Batch processing API** | Roadmap item, absent. | Batch + async job tracking + status endpoints; demonstrates workflow orchestration. | L | P2 | 4 | +| **Real-time status (SSE/WebSockets)** | Roadmap item, absent. | Proves you can do long-lived connections, backpressure, and reconnect semantics. | M | P2 | 4 | +| **Idempotency keys on upload** | None. | Client retries currently create duplicate assets. An `Idempotency-Key` header + dedup is a concrete correctness feature. | S | P2 | 3 | +| **Asset delete / soft-delete** | None. | Adds lifecycle + storage cleanup + authorization surface. | S | P2 | 2 | +| **Admin dashboard** | Roadmap item, absent. | Mostly product value; low *engineering* proof unless it adds real ops tooling. | XL | P2 | 2 | + +## 3. Security + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **Real identity: users, login, JWT/refresh, RBAC** | Homegrown AES token; no identity model. | Forces you to reason about password hashing (bcrypt exists), token expiry/rotation, refresh flows, and least-privilege authz. Highest-leverage security gap. | L | P0 | 5 | +| **API keys / scoped service tokens** | None. | Machine-to-machine auth with scopes + revocation; separates human vs service identity. | M | P1 | 4 | +| **Webhook HMAC signing** | No delivery engine at all (see 1.x). | Signing webhook payloads (HMAC-SHA256) is the standard, testable security feature. | M | P1 | 4 | +| **Signed GET URLs for serving + CDN** | Presigned PUT exists; no signed GET, no CDN. | Enables private content delivery and cache invalidation — production serving. | M | P2 | 3 | +| **Rate limiting on all endpoints** | Per-IP limiter on `/presign` only. | Extend to a token-bucket/IP limiter middleware across the API; proves abuse-resistance thinking. | S | P2 | 3 | +| **Secrets management** | Env-file only; `ENCRYPTION_KEY` in env. | Wire Vault / k8s secrets / cloud KMS references; proves you treat keys as infrastructure. | M | P2 | 3 | + +## 4. Media pipeline depth + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **AVIF / advanced image optimization** | WebP variants done (Pillow); AVIF absent. | AVIF needs a real encoder (libvips/cwebp/avif) — shows codec/quality/size tradeoff engineering. | M | P2 | 3 | +| **HLS/DASH adaptive streaming** | `manifest_url` column exists; transcoding is single 720p MP4 + poster + preview only. | Segmenting, playlists, multiple renditions, and CDN-friendly delivery — a genuinely hard, impressive feature. | XL | P1 | 5 | +| **Replace Pillow with libvips/Sharp** | Pillow is CPU-bound and slow at scale. | Proves performance engineering (memory, throughput, concurrency) on the hot path. | L | P2 | 4 | +| **Azure Blob Storage provider** | Roadmap item; GCS + S3 done behind `StorageX`. | Adding a third provider validates the abstraction is truly provider-agnostic. | M | P2 | 3 | + +## 5. Observability & operability + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **Unified OTel tracing across Go + worker** | API is OTel; worker uses `prometheus_client` only — no trace propagation across the Redis boundary. | Cross-service trace context (W3C traceparent through the stream) is the gold standard for distributed tracing. | M | P1 | 4 | +| **SLIs/SLOs + error budgets** | Metrics exist; no SLO definitions or budgets. | Forces you to define what "healthy" means and alert on the budget, not just raw metrics. | M | P1 | 4 | +| **Structured alerting rules (Prometheus)** | Prometheus/Grafana configs exist; no alert rules. | Alerting on latency/error/throughput with proper thresholds and runbooks. | S | P2 | 3 | +| **Load/soak testing harness** | `performance_suite_test.go` is a naive 200-request loop, not a real load tool. | k6/locust with SLO assertions gives defensible capacity numbers. | M | P2 | 3 | + +## 6. DX, testing & delivery + +| Feature | Current state | Engineering value | Effort | Priority | Score | +|---------|---------------|-------------------|--------|----------|-------| +| **E2E pipeline test (MinIO+PG+Redis)** | No harness runs upload→process→variant. | The single most valuable test: proves the whole system works together and would have caught 0.1. | M | P0 | 5 | +| **Contract tests / OpenAPI spec** | No OpenAPI; README-only docs. | Generated, versioned API contract; enables client generation and contract testing. | M | P2 | 3 | +| **Fuzz/property tests (Go)** | Unit tests only (config, auth, logging, mime, s3). | Fuzzing the JSON parsing / token / hash paths finds real bugs. | S | P2 | 3 | +| **Worker test coverage expansion** | 4 Python test files (retry, recovery, image, dispatch). | Add DB-backed and storage-mocked tests for dedup + clone paths. | M | P2 | 3 | +| **Helm chart / Terraform** | Raw k8s manifests (good: pdb, resource-quota, rbac, secrets). | Helm packaging + IaC for environments proves reproducible deploys. | L | P2 | 3 | +| **ADRs / architecture docs** | README + references only. | Records decisions (why Redis transport-only, why content-addressing) for future engineers. | S | P2 | 2 | + +--- + +## Recommended build order + +**Phase 1 — make it trustworthy (P0):** +1. Single canonical migration set (0.1) — unblocks everything. +2. E2E pipeline test with MinIO+PG+Redis (6) — catches the schema drift. +3. Fix CI: tag the perf test (0.3), wire real integration tests (0.2). +4. Transactional outbox for job dispatch (1). + +**Phase 2 — make it a real product (P1):** +5. Real identity + auth (3.1) and API keys (3.2). +6. Asset read/list/variants API (2). +7. Webhook delivery engine + HMAC signing (1 + 3.3). +8. Cross-service tracing (5) + SLIs/SLOs (5). + +**Phase 3 — make it impressive (P2):** +9. HLS/DASH adaptive streaming (4). +10. Batch API + real-time status (2). +11. libvips migration, AVIF, Azure provider, DLQ, GC (4/1). + +## What the project already proves (strengths) + +- Layered Go service (handler/service/repo/models) with typed error hierarchy mapped to HTTP status. +- Provider-agnostic storage abstraction (GCS + S3/MinIO) with presigned URLs, metrics, tracing. +- Content-addressed, immutable variants with cross-asset dedup (`canonical_asset_id`). +- Redis Streams consumer group with idempotency, retry classification, and periodic stuck-job recovery. +- Full observability stack (OTel + Prometheus + Grafana/Tempo/Loki) and a real CI/CD + release pipeline (staging/LTS, GHCR). +- Thoughtful k8s manifests (PDB, resource quotas, RBAC, secrets, migration job). + +These are the marks of a solid mid-level pipeline. The features above are what separate +"a working pipeline" from "an engineering project" — correctness under failure, +real security, a read model, and tests that actually run.