diff --git a/doc/rfc/stovepipe/project-status-api.md b/doc/rfc/stovepipe/project-status-api.md index a92332496..58cba5a9e 100644 --- a/doc/rfc/stovepipe/project-status-api.md +++ b/doc/rfc/stovepipe/project-status-api.md @@ -17,7 +17,7 @@ The request and response fields have the following API-level meaning. The publis | `queue` | Yes | Identifies the queue in which to find the validation request. | | `request_id` | ID lookup | Canonical positive decimal request ID returned by `Ingest`. | | `change_uri` | URI lookup | Identifies the exact commit URI under validation. | -| `projects` | No | Consumer-defined project IDs whose results are requested. Omitted or empty returns all recorded project results. | +| `projects` | No | Consumer-defined project IDs whose results are requested. Omitted or empty lists indexed project results. | | `page_size` | No | Limits one page of results for the requested projects. | | `page_token` | No | Continues a previous page of results for the requested projects. | @@ -29,7 +29,7 @@ The request and response fields have the following API-level meaning. The publis | `updated_at_ms` | Records, in Unix milliseconds, the newest durable lifecycle or result record represented by the response. | | `repository_breakage_degree` | Returns the whole-repository result when it is durably recorded. | | `project_results_complete` | Indicates whether the implementation has durably finished producing results for the supplied project IDs. | -| `projects` | Returns recorded results for requested project IDs, or all recorded project results when none are supplied. | +| `projects` | Returns recorded results for requested project IDs, or indexed project results when none are supplied. | | `next_page_token` | Continues requested-project result pagination when another page exists. | `request_state` is a stable public lifecycle vocabulary: `accepted`, `processing`, `succeeded`, `failed`, `cancelled`, or `superseded`. Clients must tolerate a future value. A terminal request state does not by itself mean that project results are complete. @@ -64,7 +64,9 @@ The caller may supply the project IDs for which it wants recorded results. The A When project IDs are supplied, the response includes only results for those IDs, in the supplied order, and paginates that list. An implementation may record results for every requested project, only requested projects with a non-green result, or another documented subset. -With no project IDs supplied, the response lists all recorded project results in project-ID order, excluding the repository fact. Results are a live view; restart pagination to discover facts recorded behind the cursor. +With no project IDs supplied, the response lists indexed project results by descending breakage degree, then ascending project ID, excluding the repository fact. A dedicated ordering mapping makes this a bounded primary-key range read, not a sort over all facts. The continuation token identifies the last project; its immutable fact supplies the degree needed to resume this ordering. A missing cursor fact returns an error. Results are a live view; restart pagination to discover facts recorded before the cursor. + +The index starts empty at deployment, without historical population or fallback scans. Older requests may therefore have empty project listings, and requests spanning the rollout may have partial listings. Explicit project selections still read their facts directly. This intentionally does not preserve unfiltered project listings for older requests. Each project result is an immutable validation fact keyed by queue, commit URI, and project ID. A result must identify the selected request. `project_results_complete` is false until the implementation has durably finished producing results for the supplied project IDs; a completed empty set is valid. @@ -72,7 +74,9 @@ Callers use `project_results_complete`, rather than the presence or absence of a ## Implementation -Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and recorded facts for requested projects in paginated request order, or all recorded project results in project-ID order when none are requested. `project_results_complete` remains false until a durable completion projection is available. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page. +Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and recorded facts for requested projects in paginated request order, or indexed project results by descending breakage degree then ascending project ID when none are requested. `project_results_complete` remains false until a durable completion projection is available. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page. + +Listing reads at most `page_size + 1` ordering mappings and hydrates only the returned facts, plus the cursor fact on continuations. A mapping that disagrees with its authoritative fact is a consistency error. The [project validation index](project-validation-index.md) owns the exact key encoding, write-path repair and fresh-start rollout. Adopt these readers after index writers are deployed; historical coverage is not required. ## Errors and Authorization diff --git a/stovepipe/controller/project_status.go b/stovepipe/controller/project_status.go index 5f9c63abe..d70bcfae2 100644 --- a/stovepipe/controller/project_status.go +++ b/stovepipe/controller/project_status.go @@ -161,7 +161,11 @@ func (c *projectStatusController) GetProjectStatus(ctx context.Context, req enti } } - page, err := loadProjectStatusPage(ctx, req, requestSummary, factStore) + var indexStore storage.ProjectValidationIndexStore + if len(req.Projects) == 0 { + indexStore = store.GetProjectValidationIndexStore() + } + page, err := loadProjectStatusPage(ctx, req, requestSummary, factStore, indexStore) if err != nil { return entity.GetProjectStatusResult{}, err } @@ -221,6 +225,7 @@ func loadProjectStatusPage( req entity.GetProjectStatusRequest, summary entity.RequestSummary, store storage.ValidationFactStore, + indexStore storage.ProjectValidationIndexStore, ) (projectStatusPage, error) { token, err := parseProjectStatusPageToken(req, summary) if err != nil { @@ -233,7 +238,7 @@ func loadProjectStatusPage( } if len(req.Projects) == 0 { - return listProjectStatusPage(ctx, summary, store, token, pageSize) + return listProjectStatusPage(ctx, summary, store, indexStore, token, pageSize) } return getRequestedProjectStatusPage(ctx, summary, store, token, pageSize) @@ -282,32 +287,61 @@ func listProjectStatusPage( ctx context.Context, summary entity.RequestSummary, store storage.ValidationFactStore, + indexStore storage.ProjectValidationIndexStore, token projectStatusPageToken, pageSize int, ) (projectStatusPage, error) { - facts, err := store.ListProjects(ctx, summary.URI, token.AfterProject, pageSize+1) - if err != nil { - return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to list project facts for request %q: %w", summary.RequestID, err) - } + var after entity.ProjectValidationIndex + if token.AfterProject != "" { + fact, err := store.Get(ctx, summary.URI, token.AfterProject) + if err != nil { + return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to load cursor project %q: %w", token.AfterProject, err) + } - for _, fact := range facts { - if err := validateProjectFact(fact, summary, fact.Project); err != nil { + if err := validateProjectFact(fact, summary, token.AfterProject); err != nil { return projectStatusPage{}, err } - if fact.Project == "" { - return projectStatusPage{}, &ProjectStatusConsistencyError{message: "project listing contains a repository fact"} + after, err = entity.NewProjectValidationIndex(fact) + if err != nil { + return projectStatusPage{}, err } } - page := projectStatusPage{facts: facts} - if len(facts) > pageSize { - page.facts = facts[:pageSize] - token.AfterProject = page.facts[pageSize-1].Project + entries, err := indexStore.List(ctx, summary.URI, after, pageSize+1) + if err != nil { + return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to list project index for request %q: %w", summary.RequestID, err) + } + + var page projectStatusPage + if len(entries) > pageSize { + entries = entries[:pageSize] + token.AfterProject = entries[pageSize-1].Project page.nextToken, err = encodeProjectStatusPageToken(token) + if err != nil { + return projectStatusPage{}, err + } } - return page, err + for _, entry := range entries { + fact, err := store.Get(ctx, summary.URI, entry.Project) + if err != nil { + return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to load indexed project %q: %w", entry.Project, err) + } + + if err := validateProjectFact(fact, summary, entry.Project); err != nil { + return projectStatusPage{}, err + } + + expected, err := entity.NewProjectValidationIndex(fact) + if err != nil || entry != expected { + return projectStatusPage{}, &ProjectStatusConsistencyError{message: "project ordering index disagrees with stored fact"} + } + + page.facts = append(page.facts, fact) + } + + return page, nil } func encodeProjectStatusPageToken(token projectStatusPageToken) (string, error) { diff --git a/stovepipe/controller/project_status_test.go b/stovepipe/controller/project_status_test.go index 6832c7152..ba60d742d 100644 --- a/stovepipe/controller/project_status_test.go +++ b/stovepipe/controller/project_status_test.go @@ -111,6 +111,7 @@ func TestGetProjectStatus(t *testing.T) { uriStore := storagemock.NewMockRequestURIStore(mockCtrl) summaryStore := storagemock.NewMockRequestSummaryStore(mockCtrl) factStore := storagemock.NewMockValidationFactStore(mockCtrl) + indexStore := storagemock.NewMockProjectValidationIndexStore(mockCtrl) if !(tt.wantInvalid && tt.request.PageToken == "") { factory.EXPECT().For(storage.Config{QueueName: projectStatusQueue}).Return(store, nil) store.EXPECT().GetRequestURIStore().Return(uriStore) @@ -123,7 +124,14 @@ func TestGetProjectStatus(t *testing.T) { factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "").Return(tt.fact, tt.factErr) if !tt.noProjectLookup && !(tt.factErr == nil && tt.wantConsistency) { if len(tt.request.Projects) == 0 { - factStore.EXPECT().ListProjects(gomock.Any(), projectStatusURI, "", defaultProjectStatusPageSize+1).Return(tt.projectFacts, tt.wantErr) + store.EXPECT().GetProjectValidationIndexStore().Return(indexStore) + indexStore.EXPECT().List(gomock.Any(), projectStatusURI, entity.ProjectValidationIndex{}, defaultProjectStatusPageSize+1). + Return(projectStatusIndexEntries(t, tt.projectFacts), tt.wantErr) + if tt.wantErr == nil { + for _, fact := range tt.projectFacts { + factStore.EXPECT().Get(gomock.Any(), projectStatusURI, fact.Project).Return(fact, nil) + } + } } for _, project := range tt.request.Projects { factStore.EXPECT().Get(gomock.Any(), projectStatusURI, project).Return(entity.ValidationFact{}, storage.ErrNotFound) @@ -300,9 +308,9 @@ func TestGetProjectStatusReturnsProjectFactsInRequestedOrder(t *testing.T) { summaryStore.EXPECT().Get(gomock.Any(), projectStatusID).Return(requestSummary, nil) store.EXPECT().GetValidationFactStore().Return(factStore) factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "").Return(entity.ValidationFact{}, storage.ErrNotFound) - factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "project-b").Return(entity.ValidationFact{URI: projectStatusURI, RequestID: projectStatusID, Project: "project-b", Degree: entity.DegreeBroken, CreatedAt: 20}, nil) + factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "project-b").Return(entity.ValidationFact{URI: projectStatusURI, RequestID: projectStatusID, Project: "project-b", Degree: entity.DegreeGreen, CreatedAt: 20}, nil) factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "missing").Return(entity.ValidationFact{}, storage.ErrNotFound) - factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "project-a").Return(entity.ValidationFact{URI: projectStatusURI, RequestID: projectStatusID, Project: "project-a", Degree: entity.DegreeGreen, CreatedAt: 30}, nil) + factStore.EXPECT().Get(gomock.Any(), projectStatusURI, "project-a").Return(entity.ValidationFact{URI: projectStatusURI, RequestID: projectStatusID, Project: "project-a", Degree: entity.DegreeBroken, CreatedAt: 30}, nil) controller := NewProjectStatusController(zap.NewNop().Sugar(), tally.NoopScope, factory) got, err := controller.GetProjectStatus(context.Background(), entity.GetProjectStatusRequest{Queue: projectStatusQueue, ChangeURI: projectStatusURI, Projects: []string{"project-b", "missing", "project-a"}}) @@ -452,9 +460,9 @@ func TestParseProjectStatusPageTokenRejectsMismatches(t *testing.T) { func TestGetProjectStatusPaginatesStoredProjects(t *testing.T) { summary := entity.RequestSummary{RequestID: projectStatusID, Queue: projectStatusQueue, URI: projectStatusURI, StateTimestampMs: 10} stored := []entity.ValidationFact{ + {URI: projectStatusURI, RequestID: projectStatusID, Project: "c", Degree: 1, CreatedAt: 60}, {URI: projectStatusURI, RequestID: projectStatusID, Project: "a", Degree: 0.5, CreatedAt: 20}, {URI: projectStatusURI, RequestID: projectStatusID, Project: "b", Degree: 0, CreatedAt: 50}, - {URI: projectStatusURI, RequestID: projectStatusID, Project: "c", Degree: 1, CreatedAt: 60}, } ctrl := gomock.NewController(t) @@ -462,14 +470,20 @@ func TestGetProjectStatusPaginatesStoredProjects(t *testing.T) { store := storagemock.NewMockStorage(ctrl) summaries := storagemock.NewMockRequestSummaryStore(ctrl) facts := storagemock.NewMockValidationFactStore(ctrl) + indexStore := storagemock.NewMockProjectValidationIndexStore(ctrl) + entries := projectStatusIndexEntries(t, stored) factory.EXPECT().For(storage.Config{QueueName: projectStatusQueue}).Return(store, nil).Times(2) store.EXPECT().GetRequestSummaryStore().Return(summaries).Times(2) summaries.EXPECT().Get(gomock.Any(), projectStatusID).Return(summary, nil).Times(2) store.EXPECT().GetValidationFactStore().Return(facts).Times(2) + store.EXPECT().GetProjectValidationIndexStore().Return(indexStore).Times(2) facts.EXPECT().Get(gomock.Any(), projectStatusURI, "").Return(entity.ValidationFact{}, storage.ErrNotFound).Times(2) - facts.EXPECT().ListProjects(gomock.Any(), projectStatusURI, "", 3).Return(stored, nil) - facts.EXPECT().ListProjects(gomock.Any(), projectStatusURI, "b", defaultProjectStatusPageSize+1).Return(stored[2:], nil) + indexStore.EXPECT().List(gomock.Any(), projectStatusURI, entity.ProjectValidationIndex{}, 3).Return(entries, nil) + indexStore.EXPECT().List(gomock.Any(), projectStatusURI, entries[1], defaultProjectStatusPageSize+1).Return(entries[2:], nil) + facts.EXPECT().Get(gomock.Any(), projectStatusURI, "c").Return(stored[0], nil) + facts.EXPECT().Get(gomock.Any(), projectStatusURI, "a").Return(stored[1], nil).Times(2) + facts.EXPECT().Get(gomock.Any(), projectStatusURI, "b").Return(stored[2], nil) controller := NewProjectStatusController(zap.NewNop().Sugar(), tally.NoopScope, factory) req := entity.GetProjectStatusRequest{Queue: projectStatusQueue, RequestID: projectStatusID, PageSize: 2} @@ -477,7 +491,7 @@ func TestGetProjectStatusPaginatesStoredProjects(t *testing.T) { first, err := controller.GetProjectStatus(context.Background(), req) require.NoError(t, err) assert.Equal(t, stored[:2], first.ProjectValidationFacts) - assert.Equal(t, int64(50), first.UpdatedAtMs) + assert.Equal(t, int64(60), first.UpdatedAtMs) require.NotEmpty(t, first.NextPageToken) req.PageToken = first.NextPageToken @@ -487,10 +501,69 @@ func TestGetProjectStatusPaginatesStoredProjects(t *testing.T) { second, err := controller.GetProjectStatus(context.Background(), req) require.NoError(t, err) assert.Equal(t, stored[2:], second.ProjectValidationFacts) - assert.Equal(t, int64(60), second.UpdatedAtMs) + assert.Equal(t, int64(50), second.UpdatedAtMs) assert.Empty(t, second.NextPageToken) } +func projectStatusIndexEntries(t *testing.T, facts []entity.ValidationFact) []entity.ProjectValidationIndex { + t.Helper() + var entries []entity.ProjectValidationIndex + for _, fact := range facts { + entry, err := entity.NewProjectValidationIndex(fact) + require.NoError(t, err) + entries = append(entries, entry) + } + return entries +} + +func TestListProjectStatusPageRejectsMissingCursor(t *testing.T) { + for _, readErr := range []error{storage.ErrNotFound, errors.New("database unavailable")} { + t.Run(readErr.Error(), func(t *testing.T) { + ctrl := gomock.NewController(t) + facts := storagemock.NewMockValidationFactStore(ctrl) + index := storagemock.NewMockProjectValidationIndexStore(ctrl) + facts.EXPECT().Get(gomock.Any(), projectStatusURI, "cursor").Return(entity.ValidationFact{}, readErr) + _, err := listProjectStatusPage(context.Background(), entity.RequestSummary{URI: projectStatusURI}, facts, index, projectStatusPageToken{AfterProject: "cursor"}, 2) + require.ErrorIs(t, err, readErr) + }) + } +} + +func TestListProjectStatusPageRejectsInconsistentIndex(t *testing.T) { + fact := entity.ValidationFact{URI: projectStatusURI, Project: "p", Degree: 0.5, RequestID: projectStatusID} + entry, err := entity.NewProjectValidationIndex(fact) + require.NoError(t, err) + for _, tt := range []struct { + name string + modify func(*entity.ProjectValidationIndex, *entity.ValidationFact) + factErr error + }{ + {name: "missing fact", factErr: storage.ErrNotFound, modify: func(*entity.ProjectValidationIndex, *entity.ValidationFact) {}}, + {name: "read failure", factErr: errors.New("database unavailable"), modify: func(*entity.ProjectValidationIndex, *entity.ValidationFact) {}}, + {name: "wrong order key", modify: func(e *entity.ProjectValidationIndex, _ *entity.ValidationFact) { e.DegreeOrder++ }}, + {name: "wrong URI", modify: func(e *entity.ProjectValidationIndex, _ *entity.ValidationFact) { e.URI = "other" }}, + {name: "wrong request", modify: func(_ *entity.ProjectValidationIndex, f *entity.ValidationFact) { f.RequestID = "other" }}, + {name: "repository fact", modify: func(e *entity.ProjectValidationIndex, f *entity.ValidationFact) { e.Project = ""; f.Project = "" }}, + } { + t.Run(tt.name, func(t *testing.T) { + ctrl := gomock.NewController(t) + facts := storagemock.NewMockValidationFactStore(ctrl) + index := storagemock.NewMockProjectValidationIndexStore(ctrl) + e, f := entry, fact + tt.modify(&e, &f) + index.EXPECT().List(gomock.Any(), projectStatusURI, entity.ProjectValidationIndex{}, 3).Return([]entity.ProjectValidationIndex{e}, nil) + facts.EXPECT().Get(gomock.Any(), projectStatusURI, e.Project).Return(f, tt.factErr) + _, err := listProjectStatusPage(context.Background(), entity.RequestSummary{URI: projectStatusURI, RequestID: projectStatusID}, facts, index, projectStatusPageToken{}, 2) + require.Error(t, err) + if tt.factErr != nil { + assert.ErrorIs(t, err, tt.factErr) + } else { + assert.True(t, IsProjectStatusConsistency(err)) + } + }) + } +} + func TestValidateProjectFact(t *testing.T) { requestSummary := entity.RequestSummary{RequestID: projectStatusID, URI: projectStatusURI} assert.NoError(t, validateProjectFact(entity.ValidationFact{URI: projectStatusURI, RequestID: projectStatusID, Project: "project-a", Degree: entity.DegreeGreen}, requestSummary, "project-a")) diff --git a/stovepipe/entity/project_status.go b/stovepipe/entity/project_status.go index 3cacf53b6..4b31b4aa5 100644 --- a/stovepipe/entity/project_status.go +++ b/stovepipe/entity/project_status.go @@ -22,7 +22,7 @@ type GetProjectStatusRequest struct { RequestID string // ChangeURI identifies the exact commit under validation, empty when selecting by request ID. ChangeURI string - // Projects limits results to the supplied consumer-defined project IDs; empty selects all recorded projects. + // Projects limits results to the supplied consumer-defined project IDs; empty selects indexed project results. Projects []string // PageSize is the maximum number of project results per page. PageSize int32 @@ -40,7 +40,7 @@ type GetProjectStatusResult struct { HasRepositoryValidationFact bool // ProjectResultsComplete reports whether the implementation has finished its project-result set. ProjectResultsComplete bool - // ProjectValidationFacts contains recorded results in request order, or project-ID order when Projects is empty. + // ProjectValidationFacts contains recorded results in request order, or descending degree then ascending project ID when Projects is empty. ProjectValidationFacts []ValidationFact // NextPageToken continues requested-project result pagination when another page exists. NextPageToken string diff --git a/test/integration/stovepipe/BUILD.bazel b/test/integration/stovepipe/BUILD.bazel index 11017b9b2..a3bc59593 100644 --- a/test/integration/stovepipe/BUILD.bazel +++ b/test/integration/stovepipe/BUILD.bazel @@ -16,11 +16,14 @@ go_test( ], deps = [ "//api/stovepipe/protopb:go_default_library", + "//stovepipe/entity:go_default_library", + "//stovepipe/extension/storage/mysql:go_default_library", "//test/testutil:go_default_library", "@com_github_go_sql_driver_mysql//:go_default_library", "@com_github_stretchr_testify//assert:go_default_library", "@com_github_stretchr_testify//require:go_default_library", "@com_github_stretchr_testify//suite:go_default_library", + "@com_github_uber_go_tally//:go_default_library", "@org_golang_google_grpc//:go_default_library", "@org_golang_google_protobuf//proto:go_default_library", ], diff --git a/test/integration/stovepipe/suite_test.go b/test/integration/stovepipe/suite_test.go index e6437b518..62f018bdd 100644 --- a/test/integration/stovepipe/suite_test.go +++ b/test/integration/stovepipe/suite_test.go @@ -36,7 +36,10 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" + "github.com/uber-go/tally" pb "github.com/uber/submitqueue/api/stovepipe/protopb" + "github.com/uber/submitqueue/stovepipe/entity" + mysqlstorage "github.com/uber/submitqueue/stovepipe/extension/storage/mysql" "github.com/uber/submitqueue/test/testutil" "google.golang.org/grpc" "google.golang.org/protobuf/proto" @@ -347,25 +350,35 @@ func (s *StovepipeIntegrationSuite) TestProjectStatusAPI() { VALUES (?, ?, ?, ?, ?, ?), (?, ?, ?, ?, ?, ?)`, queue, uri, "project-green", 0.0, id, 4000, queue, uri, "project-b", 1.0, id, 5000) require.NoError(t, err) + indexStore := mysqlstorage.NewProjectValidationIndexStore(s.db, tally.NoopScope, queue) + for _, fact := range []entity.ValidationFact{ + {URI: uri, Project: "project-a", Degree: 0.5}, + {URI: uri, Project: "project-b", Degree: 1}, + {URI: uri, Project: "project-green", Degree: 0}, + } { + entry, err := entity.NewProjectValidationIndex(fact) + require.NoError(t, err) + require.NoError(t, indexStore.Create(s.ctx, entry)) + } firstPage, err := s.client.GetProjectStatusByURI(s.ctx, &pb.GetProjectStatusByURIRequest{ Queue: queue, ChangeUri: uri, PageSize: 1, }) require.NoError(t, err) require.Len(t, firstPage.Projects, 1) - assert.Equal(t, "project-a", firstPage.Projects[0].GetProject()) - assert.Equal(t, 0.5, firstPage.Projects[0].GetBreakageDegree()) - assert.Equal(t, int64(3000), firstPage.GetUpdatedAtMs()) + assert.Equal(t, "project-b", firstPage.Projects[0].GetProject()) + assert.Equal(t, 1.0, firstPage.Projects[0].GetBreakageDegree()) + assert.Equal(t, int64(5000), firstPage.GetUpdatedAtMs()) require.NotEmpty(t, firstPage.NextPageToken) remaining, err := s.client.GetProjectStatusByID(s.ctx, &pb.GetProjectStatusByIDRequest{ Queue: queue, RequestId: id, PageToken: firstPage.NextPageToken, }) require.NoError(t, err) require.Len(t, remaining.Projects, 2) - assert.Equal(t, "project-b", remaining.Projects[0].GetProject()) - assert.Equal(t, 1.0, remaining.Projects[0].GetBreakageDegree()) + assert.Equal(t, "project-a", remaining.Projects[0].GetProject()) + assert.Equal(t, 0.5, remaining.Projects[0].GetBreakageDegree()) assert.Equal(t, "project-green", remaining.Projects[1].GetProject()) assert.Zero(t, remaining.Projects[1].GetBreakageDegree()) - assert.Equal(t, int64(5000), remaining.GetUpdatedAtMs()) + assert.Equal(t, int64(4000), remaining.GetUpdatedAtMs()) assert.Empty(t, remaining.NextPageToken) assert.False(t, remaining.ProjectResultsComplete) @@ -377,6 +390,33 @@ func (s *StovepipeIntegrationSuite) TestProjectStatusAPI() { } } +func (s *StovepipeIntegrationSuite) TestUnindexedFactsRemainAvailableByExplicitProject() { + t := s.T() + const queue, id, uri = "fresh-index/main", "9", "git://fresh-index/main/older-commit" + _, err := s.db.Exec(`INSERT INTO request_summary + (queue, request_id, uri, base_uri, state, request_version, state_timestamp_ms, version) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, queue, id, uri, "", "succeeded", 1, 1000, 1) + require.NoError(t, err) + _, err = s.db.Exec(`INSERT INTO validation_fact (queue, uri, project, degree, request_id, created_at) + VALUES (?, ?, ?, ?, ?, ?)`, queue, uri, "old-project", 0.5, id, 2000) + require.NoError(t, err) + + listed, err := s.client.GetProjectStatusByID(s.ctx, &pb.GetProjectStatusByIDRequest{ + Queue: queue, RequestId: id, + }) + require.NoError(t, err) + assert.Empty(t, listed.Projects) + assert.Empty(t, listed.NextPageToken) + + explicit, err := s.client.GetProjectStatusByID(s.ctx, &pb.GetProjectStatusByIDRequest{ + Queue: queue, RequestId: id, Projects: []string{"old-project"}, + }) + require.NoError(t, err) + require.Len(t, explicit.Projects, 1) + assert.Equal(t, "old-project", explicit.Projects[0].GetProject()) + assert.Equal(t, 0.5, explicit.Projects[0].GetBreakageDegree()) +} + func assertHistoryEvents(t *testing.T, events []*pb.HistoryEvent) { t.Helper() require.Len(t, events, 3) diff --git a/web/test/e2e/stovepipe/run_e2e.py b/web/test/e2e/stovepipe/run_e2e.py index 44be9b801..642c1f161 100644 --- a/web/test/e2e/stovepipe/run_e2e.py +++ b/web/test/e2e/stovepipe/run_e2e.py @@ -46,7 +46,7 @@ def _pagination_fixture_sql() -> str: f"INSERT INTO validation_fact VALUES ('web/pagination','{uri}','',0.25,'51',2000);" ) for i in range(1, 52): - degree = 0.25 if i == 1 else 0 + degree = 1.0 if i == 50 else 0.25 if i == 1 else 0 statements.append( f"INSERT INTO validation_fact VALUES ('web/pagination','{uri}','project-{i:02d}',{degree},'51',{2000+i});" ) diff --git a/web/test/e2e/stovepipe/stovepipe.spec.ts b/web/test/e2e/stovepipe/stovepipe.spec.ts index 009927d90..6b2f55ae8 100644 --- a/web/test/e2e/stovepipe/stovepipe.spec.ts +++ b/web/test/e2e/stovepipe/stovepipe.spec.ts @@ -164,6 +164,9 @@ test("real ingestion, queue selection, cursors, request summaries, history, proj const section = page.getByRole("region", { name: "Project results", exact: true }); const projects = section.getByRole("table", { name: "Recorded project results", exact: true }); await expect(projects.locator("tbody tr")).toHaveCount(50); + await expect(projects.locator("tbody tr").nth(0).getByRole("cell", { name: "project-50", exact: true })).toBeVisible(); + await expect(projects.locator("tbody tr").nth(1).getByRole("cell", { name: "project-01", exact: true })).toBeVisible(); + await expect(projects.locator("tbody tr").nth(2).getByRole("cell", { name: "project-02", exact: true })).toBeVisible(); await expect(status.getByText("Repository result", { exact: true })).toBeVisible(); await expect(status.getByText("Breakage", { exact: true })).toBeVisible(); await expect(projects.getByText(/Repository|%/)).toHaveCount(0);