feat(submitqueue): read List from receipt lookup

Summary:
Builds on #799 to serve List from immutable request_receipt keys and authoritative request_summary rows, instead of the duplicated queue-summary projection.

### Changes

- Scan receipt keys, then point-read summaries for visible results; keep bounds, string-ID ordering, and page-token format unchanged.
- Use the existing RequestSummary response type and wire mapper.
- Fail the page on missing, mismatched, or hidden summaries using the existing InternalConsistencyError.
- Add controller and gateway integration coverage; update the RFC. Keep legacy projection writes for rollback.

### Rollout

Deploy only after #799 reaches all writers. No backfill or legacy-read fallback: requests without receipt keys are absent from List. Duplicate-write and old-table removal remain follow-up work.

## Issue Links
None — no tracking issue was supplied.

Test Plan:
After CD rollout, confirm new Land requests appear in List with the same summary fields as ID lookup, check continuation pages, and monitor page latency.

Revert Plan:
Revert the read cutover to restore the old List path. Legacy projection writes remain active; this PR makes no schema changes.

API Changes:
No RPC/proto changes. The Go ListResult now contains RequestSummary values.

Author: mnoah1

doc/rfc/submitqueue/status-list-api.md

@@ -87,11 +87,11 @@ The immutable context is queue, change URIs, and receipt time. The mutable respo
 
 The key supports authoritative lookup and conditional status updates for one request without a secondary index, and leads with the queue so the table is shardable by queue.
 
-### Request Summaries by Queue
+### Requests by Receipt Time
 
-The queue projection is logically keyed by `(queue, received_at_ms, sqid)` and must support a bounded descending scan over `(received_at_ms, sqid)`. A backend may satisfy that contract with a reverse range scan or by descending-encoding the ordered key components; cursors always carry the original receipt time and sqid values.
+The immutable `request_receipt` lookup contains only `(queue, received_at_ms, request_id)` as its primary key and supports a bounded descending scan over `(received_at_ms, request_id)`. Request IDs break timestamp ties in string order, not numeric order. A backend may use a reverse range scan or descending-encode the ordered key components; cursors always carry the original receipt time and request ID.
 
-The current `request_summary_by_queue` row duplicates the complete `List` response so a page is served by one bounded range scan. Status updates still propagate to this projection during the receipt-lookup transition below; it is not the target storage model.
+List resolves these keys through `request_summary` rather than storing response fields in the lookup. The old `request_summary_by_queue` projection still receives writes for rollback, but List no longer reads it.
 
 The logical key covers the `List` queue predicate, receipt-time range, newest-first ordering, and complete keyset cursor in one bounded scan.
 
@@ -111,23 +111,23 @@ The URI is stored in the canonical form received from the validated Land request
 
 After synchronous validation, Land generates the sqid and one receipt timestamp. The gateway persists the authoritative summary in the internal `accepting` state, then publishes the request to the orchestrator. It does not create the URI or queue projections while the request remains `accepting`.
 
-After publication succeeds, Land appends the initial `accepted` request log. Materializing `accepted`, `started`, or any later event promotes the authoritative summary out of `accepting` and creates the URI and queue projections. This handles the race where the orchestrator emits `started` before Land finishes persisting `accepted`.
+After publication succeeds, Land appends the initial `accepted` request log. Materializing `accepted`, `started`, or any later lifecycle state promotes the authoritative summary out of `accepting`, creates the URI and legacy queue projections, and ensures the receipt lookup key exists. This handles the race where the orchestrator emits `started` before Land finishes persisting `accepted`.
 
 An `accepted` event is the lowest public lifecycle state. A late `accepted` event is retained in request history but must not replace `started` or any later materialized status.
 
 Pipeline publication is the Land success boundary. If publication fails, Land returns an error and leaves the hidden `accepting` receipt for operational cleanup. If publication succeeds but appending or materializing `accepted` fails, Land still returns the sqid because retrying the RPC would submit a duplicate request that is already in the pipeline. A later pipeline event activates and repairs the public projections.
 
 ### Request-Log Materialization
 
-Every gateway request-log persistence path uses the same materialization component. It appends the audit log, compares the incoming entry with the authoritative summary, conditionally advances the winner, and propagates the authoritative value to the queue projection.
+Every gateway request-log persistence path uses the same materialization component. It appends the audit log, compares the incoming entry with the authoritative summary, and conditionally advances the winner. For public summaries, it completes the URI and legacy queue projection writes before ensuring the immutable receipt key, including on unchanged or stale-log retries. Hidden `accepting` summaries remain excluded; a failed receipt write is returned through the existing projection-failure path.
 
 The winner comparison preserves the existing current-status reconciliation behavior:
 
 1. A terminal request-state entry with a positive request version beats every non-terminal or unversioned winner.
 2. Between versioned terminal entries, the greater request version wins.
 3. Equal terminal versions use the greater log timestamp as a tie-breaker.
 4. When no versioned terminal winner exists, the greater log timestamp wins.
-5. Any retained lifecycle event promotes an `accepting` summary into the public projections.
+5. A retained lifecycle status entry promotes an `accepting` summary into the public projections; audit-only events do not.
 6. `accepted` cannot replace `started` or any later status, even when the accepted log arrives later.
 
 Materialization uses optimistic concurrency so stale or out-of-order consumers cannot replace a newer winner. Version arithmetic and reconciliation decisions belong to the materialization component; stores perform only mechanical creates, reads, conditional updates, and bounded page queries.
@@ -144,15 +144,17 @@ The gateway performs a bounded newest-first scan of the URI reverse mapping and
 
 ### List by Queue and Receipt Time
 
-The gateway performs one bounded range scan of the queue projection using queue, receipt-time bounds, and an optional keyset cursor. Queue projections are created only when the request reaches `accepted` or a later state, so `accepting` receipts are absent by construction. The ordering key is immutable, so later status updates cannot move an item across an issued cursor.
+The gateway scans `request_receipt` using the queue, receipt-time bounds, and optional keyset cursor, fetching one extra key to determine whether another page exists. It point-reads `request_summary` for each visible result, preserves receipt-key order, and forms the next token from the last returned receipt. The extra key does not require a summary read. Later status updates cannot move an item across the immutable cursor.
+
+Receipt keys are created only for public summaries. An invalid key, missing summary, identity or receipt-time mismatch, or `accepting` summary fails the entire page as an internal consistency error rather than returning a user-facing not-found or silently omitting the item. Other backend failures retain their error chains.
 
 ## Consistency
 
-The request log remains the append-only audit record. The authoritative summary and queue projection are eventually consistent views of its winning current state.
+The request log remains the append-only audit record. The authoritative summary is an eventually consistent view of its winning current state.
 
-Because the authoritative summary and queue projection are separate writes, a short interval can exist where request-summary retrieval and `List` show different statuses. Retried materialization repairs the queue projection from the authoritative sqid summary until both converge. Neither API reconciles logs during reads to hide this interval.
+List and request-summary retrieval read the same authoritative projection, so List no longer waits for mutable state to propagate into a second table. Membership may lag until the receipt key is written and retries repair partial writes. Separate calls and per-item reads are not a shared snapshot and may observe different lifecycle versions. Neither API reconciles logs during reads.
 
-Request context has a stronger guarantee than status convergence: the authoritative receipt is persisted before the request is published to the orchestrator. Public URI and queue projections are activated only by `accepted` or a later event. Once a queue projection is visible, its queue, sqid, change URIs, and receipt time are complete.
+Request context has a stronger guarantee than status convergence: the authoritative receipt is persisted before the request is published to the orchestrator. Public lookup keys are activated only by `accepted` or a later lifecycle state. Once a receipt key is visible, its authoritative summary has complete queue, sqid, change URIs, and receipt time.
 
 ## Compatibility and Rollout
 
@@ -164,17 +166,13 @@ The request-summary RPCs replace the former polymorphic Status shape with explic
 
 ### Receipt-Lookup Transition
 
-The target model keeps the full projection only in `request_summary`. An immutable `request_receipt` lookup contains only `(queue, received_at_ms, request_id)` as its primary key. List will scan these keys and point-read the corresponding authoritative summaries, preserving the existing response, required time window, string-ID tie ordering, and page tokens. Receipt time and all response fields are already captured; no additional data collection is needed.
-
-The first phase adds the lookup and its writes, without changing List reads or removing the old projection. After materializing a public summary and completing the existing URI/queue projection writes, the materializer ensures the receipt key exists, including on unchanged or stale-log retries. Hidden `accepting` receipts are excluded. A failed lookup write is returned through the existing projection-failure path.
+The receipt lookup and writer path are implemented, and List now scans receipt keys and point-reads authoritative summaries. The response, required time window, string-ID tie ordering, and page-token format are unchanged. Receipt time and all response fields are already captured; no additional data collection is needed.
 
-Deploy the additive table before enabling these writes. To finish the transition:
+CD must roll out the additive table and receipt writers to all instances before deploying the read cutover. The cutover performs one scan plus one point read per visible result; check page latency after rollout.
 
-1. Upgrade all writers, backfill existing List membership from the old table's keys, validate each key against its public authoritative summary, and verify lookup coverage.
-2. Switch List to the receipt lookup plus authoritative summary reads, keeping old projection writes during the rollback window. This trades one page scan for one scan plus a point read per visible result; verify page latency before cutover.
-3. Stop duplicate projection writes, remove `RequestQueueSummary` and its store/mapper, and retire `request_summary_by_queue` after the rollback window. Only the small immutable lookup key remains duplicated.
+There is no historical backfill or fallback to the legacy queue projection. Requests without receipt keys are absent from List, even if their old queue-summary rows remain. Later log activity can create missing receipt keys.
 
-This backfill preserves existing List history; it does not reconstruct requests predating the original read-model rollout. Backfill tooling, read cutover, and removal are follow-up work.
+Keep legacy projection writes during the rollback window; reverting the read cutover restores the old List path. Once stable, a follow-up stops duplicate projection writes and removes `RequestQueueSummary` and its store. Retire `request_summary_by_queue` after the rollback window. The final model keeps one full projection plus immutable lookup keys.
 
 ## Rejected Alternatives
 

service/submitqueue/gateway/server/mapper/list.go

@@ -32,25 +32,8 @@ func ProtoToListRequest(req *pb.ListRequest) entity.ListRequest {
 
 // ListResultToProto maps the domain result to the wire response.
 func ListResultToProto(result entity.ListResult) *pb.ListResponse {
-	requests := make([]*pb.RequestSummary, 0, len(result.Requests))
-	for _, summary := range result.Requests {
-		requests = append(requests, RequestQueueSummaryToProto(summary))
-	}
 	return &pb.ListResponse{
-		Requests:      requests,
+		Requests:      RequestSummariesToProto(result.Requests),
 		NextPageToken: result.NextPageToken,
 	}
 }
-
-// RequestQueueSummaryToProto maps a queue projection to the wire request summary.
-func RequestQueueSummaryToProto(summary entity.RequestQueueSummary) *pb.RequestSummary {
-	return &pb.RequestSummary{
-		Sqid:         summary.RequestID,
-		Queue:        summary.Queue,
-		ChangeUris:   append([]string{}, summary.ChangeURIs...),
-		ReceivedAtMs: summary.ReceivedAtMs,
-		Status:       string(summary.Status),
-		LastError:    summary.LastError,
-		Metadata:     cloneStringMap(summary.Metadata),
-	}
-}

service/submitqueue/gateway/server/mapper/list_test.go

@@ -42,19 +42,26 @@ func TestProtoToListRequest(t *testing.T) {
 
 func TestListResultToProto(t *testing.T) {
 	result := entity.ListResult{
-		Requests: []entity.RequestQueueSummary{{
+		Requests: []entity.RequestSummary{{
 			RequestID:    "1",
 			Queue:        "q",
 			ChangeURIs:   []string{"github://uber/repo/pull/1/abc"},
 			ReceivedAtMs: 100,
-			Status:       entity.RequestStatusAccepted,
-			Metadata:     map[string]string{},
+			Status:       entity.RequestStatusError,
+			LastError:    "build failed",
+			Metadata:     map[string]string{"build": "url"},
 		}},
 		NextPageToken: "next",
 	}
 
 	response := ListResultToProto(result)
 
-	assert.Equal(t, "1", response.Requests[0].Sqid)
-	assert.Equal(t, "next", response.NextPageToken)
+	assert.Equal(t, &pb.ListResponse{
+		Requests: []*pb.RequestSummary{{
+			Sqid: "1", Queue: "q", ChangeUris: []string{"github://uber/repo/pull/1/abc"},
+			ReceivedAtMs: 100, Status: string(entity.RequestStatusError), LastError: "build failed",
+			Metadata: map[string]string{"build": "url"},
+		}},
+		NextPageToken: "next",
+	}, response)
 }

submitqueue/entity/list.go

@@ -31,7 +31,7 @@ type ListRequest struct {
 // ListResult contains one page of queue receipt history.
 type ListResult struct {
 	// Requests are ordered by receipt time descending, then request ID descending.
-	Requests []RequestQueueSummary
+	Requests []RequestSummary
 	// NextPageToken is an opaque continuation token. Empty means this is the last page.
 	NextPageToken string
 }

submitqueue/entity/request_summary.go

@@ -57,7 +57,7 @@ type RequestSummary struct {
 	Metadata map[string]string
 }
 
-// RequestQueueSummary is the queue-ordered projection returned by List.
+// RequestQueueSummary is a queue-ordered copy of a public request summary.
 type RequestQueueSummary struct {
 	// RequestID is the canonical decimal request identifier, unique within Queue.
 	RequestID string

submitqueue/gateway/controller/BUILD.bazel

@@ -39,6 +39,7 @@ go_test(
     srcs = [
         "cancel_test.go",
         "land_test.go",
+        "list_consistency_test.go",
         "list_queues_test.go",
         "list_test.go",
         "ping_test.go",

submitqueue/gateway/controller/list.go

@@ -99,7 +99,7 @@ func (c *listController) List(ctx context.Context, req entity.ListRequest) (resu
 		return entity.ListResult{}, fmt.Errorf("failed to resolve storage for queue %q: %w", req.Queue, err)
 	}
 
-	query := basestorage.RequestQueueSummaryQuery{
+	query := basestorage.RequestReceiptRange{
 		ReceivedAtOrAfterMs: req.ReceivedAtOrAfterMs,
 		ReceivedBeforeMs:    req.ReceivedBeforeMs,
 		Limit:               pageSize + 1,
@@ -112,19 +112,27 @@ func (c *listController) List(ctx context.Context, req entity.ListRequest) (resu
 		if token.Queue != req.Queue || token.ReceivedAtOrAfterMs != req.ReceivedAtOrAfterMs || token.ReceivedBeforeMs != req.ReceivedBeforeMs {
 			return entity.ListResult{}, fmt.Errorf("page token does not match query: %w", ErrInvalidRequest)
 		}
-		query.HasCursor = true
-		query.Cursor = basestorage.RequestQueueSummaryCursor{ReceivedAtMs: token.LastReceivedAtMs, RequestID: token.LastRequestID}
+		query.Before = basestorage.RequestReceiptCursor{ReceivedAtMs: token.LastReceivedAtMs, RequestID: token.LastRequestID}
 	}
 
-	summaries, err := store.GetRequestQueueSummaryStore().List(ctx, query)
+	receipts, err := store.GetRequestReceiptStore().List(ctx, query)
 	if err != nil {
-		return entity.ListResult{}, fmt.Errorf("failed to list queue=%s: %w", req.Queue, err)
+		return entity.ListResult{}, fmt.Errorf("failed to list request receipts queue=%q: %w", req.Queue, err)
+	}
+
+	visible := receipts[:min(len(receipts), pageSize)]
+	result.Requests = make([]entity.RequestSummary, 0, len(visible))
+	if len(visible) > 0 {
+		summaries := store.GetRequestSummaryStore()
+		for _, receipt := range visible {
+			summary, err := readListSummary(ctx, summaries, req.Queue, receipt)
+			if err != nil {
+				return entity.ListResult{}, err
+			}
+			result.Requests = append(result.Requests, summary)
+		}
 	}
-
-	visible := summaries
-	result = entity.ListResult{Requests: make([]entity.RequestQueueSummary, 0, min(len(visible), pageSize))}
-	if len(visible) > pageSize {
-		visible = visible[:pageSize]
+	if len(receipts) > pageSize {
 		last := visible[len(visible)-1]
 		result.NextPageToken = encodeListPageToken(listPageToken{
 			Queue:               req.Queue,
@@ -134,11 +142,27 @@ func (c *listController) List(ctx context.Context, req entity.ListRequest) (resu
 			LastRequestID:       last.RequestID,
 		})
 	}
-	result.Requests = append(result.Requests, visible...)
 	c.logger.Debugw("queue requests listed", "queue", req.Queue, "request_count", len(result.Requests), "has_next_page", result.NextPageToken != "")
 	return result, nil
 }
 
+func readListSummary(ctx context.Context, summaries basestorage.RequestSummaryStore, queue string, receipt entity.RequestReceipt) (entity.RequestSummary, error) {
+	if receipt.Queue != queue || receipt.ReceivedAtMs <= 0 || receipt.RequestID == "" {
+		return entity.RequestSummary{}, &InternalConsistencyError{Message: fmt.Sprintf("invalid request receipt queue=%q request_id=%q", queue, receipt.RequestID)}
+	}
+	summary, err := summaries.Get(ctx, receipt.RequestID)
+	if err != nil {
+		if basestorage.IsNotFound(err) {
+			return entity.RequestSummary{}, &InternalConsistencyError{Message: fmt.Sprintf("request summary missing for receipt queue=%q request_id=%q", queue, receipt.RequestID)}
+		}
+		return entity.RequestSummary{}, fmt.Errorf("failed to read request summary queue=%q request_id=%q: %w", queue, receipt.RequestID, err)
+	}
+	if summary.Queue != queue || summary.RequestID != receipt.RequestID || summary.ReceivedAtMs != receipt.ReceivedAtMs || summary.Status == entity.RequestStatusAccepting {
+		return entity.RequestSummary{}, &InternalConsistencyError{Message: fmt.Sprintf("request receipt disagrees with public summary queue=%q request_id=%q", queue, receipt.RequestID)}
+	}
+	return summary, nil
+}
+
 func encodeListPageToken(token listPageToken) string {
 	values := url.Values{
 		"queue":                   {token.Queue},
@@ -178,7 +202,7 @@ func decodeListPageToken(encoded string) (listPageToken, error) {
 		LastReceivedAtMs:    lastReceivedAtMs,
 		LastRequestID:       values.Get("last_request_id"),
 	}
-	if token.Queue == "" || token.LastRequestID == "" || token.ReceivedAtOrAfterMs >= token.ReceivedBeforeMs {
+	if token.Queue == "" || token.LastRequestID == "" || token.LastReceivedAtMs <= 0 || token.ReceivedAtOrAfterMs >= token.ReceivedBeforeMs {
 		return listPageToken{}, fmt.Errorf("invalid token fields")
 	}
 	return token, nil