Skip to content

ci: restore sub-20-minute validation and make deep functional smoke opt-in #167

Closed

Description

@Proxicon

Completed through normally merged PR #168: actual PR 13m04s, natural merged-main 9m03s, both successful. The requested beta.1 release followthrough continues through #152.

The owner policy amendment of 7 October 2026 requires routine PR/main validation to succeed in strictly less than 1,200 seconds (target 10–15 minutes), with functional acceptance performed explicitly by the owner.

Measured successful comparison:

Run Result Workflow .NET Compile Generic/visual
#539 success 16m27s 14m47s 1m16s 4m10s
#549 success 32m50s 32m39s 1m41s 27m52s
#565 success 37m58s 37m49s 2m22s 32m14s
#572 failure 36m46s 36m37s 1m57s 31m27s
#573 cancelled 40m20s 40m11s 2m36s 34m17s

API integration grew from 20 passing cases/45.638s to 67 passing cases/30m32.759s. Full NetRatel/published-RatelDesk pair fixtures and real rotation/recovery windows dominate this regression; compilation does not. Four rotation rows in #572 summed to 18m32.755s; overlapping test sums are not workflow wall time. Core rose from 2m29.844s to 11m19.730s while concurrent with API. Component tests remain cheap.

Implement a focused PR from current main a284782:

  • Keep disclosure, version/source/PostgreSQL/release static checks, one visibly separate Release restore/build, three prebuilt fast Core/API/component assemblies, nonzero all-passing per-assembly TRX verification, and the existing meaningful aggregate.
  • Explicitly classify the eight full-pair API classes as category=manual-integration and exclude compose, hosted, manual-integration from default selection. Preserve bodies/assertions and cheap protocol, canonical, payload, network, auth/PostgreSQL and provider regressions.
  • Move browser/Chromium, full physical/native client, real-time recovery/rotation, deployment/upgrade and release-image functional smoke to one workflow_dispatch-only selected-suite workflow; remove redundant image/rehearsal work and the redundant provider execution.
  • Apply the same selection to release-build while preserving native/CLI/stdio archives, licenses/SBOM/checksums, authenticated artifact receipts and hosted production.
  • Remove promoter's automatic OIDC/MCP functional smoke calls. Preserve legacy verification.state consumers, source/checksum/digest/immutability/public-availability/recovery controls; label build/distribution integrity complete separately from functional acceptance pending owner testing.
  • Update AGENTS/current CI instructions and existing selector/promotion checks. Normally integrate this policy into feat(flows): durable RatelDesk incident delivery through shared M2M #152 without reintroducing the old scheduler or mandatory evidence workload.

Acceptance: one final successful PR run and the natural successful merged-main run both strictly below 1,200 seconds from run_started_at through last required job completion. Record queue separately; include restore/build/test times, source/test-merge SHA, actual per-assembly selected/executed/pass/fail counts and separate sum of runner-job durations. Use clean standard hosted checkout; report cache honestly. Close only after merged-main success. No timeout, cancellation, skipped required execution, zero-test result, bypass or fake compatibility context counts as success.

Historical manual observations remain unresolved: #572 monitoring-clear timed out at 30s; the later #573 1.202s pass does not establish its cause. #573 physical incident hit its 300s timeout, and API had no final TRX when cancelled. Moving cadence does not claim those failures fixed.

Context: #152, #166, #141. This issue changes CI cadence and release integrity semantics; it does not mark unrelated feature issues complete or start beta.2. The already requested v0.1.1-beta.1 release continues after this fix.


Merged PR #168 implements the 7 October owner CI amendment at e3897c3d9c8362de23c006aa6cb259c07db7f50e. Routine validation now restores/builds once and runs the three prebuilt fast regression assemblies. Deep functional work has a separate, selected manual entry point.

Required workflow Result Elapsed workflow time
Successful baseline #539 Success 16m27s
Successful newer main #565 Success 37m58s
Final CI PR #168 Success 13m04s (784s)
Natural post-merge main Success 9m03s (543s)

The PR tested merge SHA was c958000b23217d775f30c5d6c50c3a311ea605fc. The natural main run is 37587846936, started at 2026-10-07T07:31:26Z, and its tested SHA is e3897c3d9c8362de23c006aa6cb259c07db7f50e. The owner’s acceptance criterion is actual success for both runs, each strictly below 1,200 seconds from workflow start through completion of the last required job. Pre-start queue time is reported separately and does not reduce workflow elapsed time.

Timing Final PR Natural main
Pre-start queue 0s 0s
Sum of runner-job durations 792s 547s
Restore 23s 23s
Release build 127s 85s
Fast regression commands 587s 387s
Assembly PR executed / passed / failed PR command time Main executed / passed / failed Main command time
Core (NetRatel.Tests) 2827 / 2827 / 0 456s 2827 / 2827 / 0 304s
API (NetRatel.API.IntegrationTests) 35 / 35 / 0 123s 35 / 35 / 0 78s
Components 368 / 368 / 0 8s 368 / 368 / 0 5s

The PR executed 3,230 passing tests with zero failures. The workflow retains source/disclosure/version/PostgreSQL/release-script checks and honest nonzero per-assembly TRX validation. Restore, compilation and fast-test outcomes are separately visible. It uses standard GitHub-hosted Ubuntu runners with no Actions build/NuGet cache configured. Optional functional smoke was not run by these automatic validations. Main execution/cache notes: fresh standard-runner checkout; restore and Release build actually executed, with no explicit NuGet/build-output cache or undeclared local prebuilt output.

Default test commands explicitly exclude category=compose, category=hosted and category=manual-integration. The new manual category covers eight full-pair API classes: ServiceLinkLifecycleTests, ServiceLinkOrchestrationRoundTripTests, ServiceLinkRotationHttpPostgresTests, ServiceLinkTerminalConvergencePostgresTests, ServiceLinkDatabaseConflictPostgresTests, ServiceLinkPayloadHttpPostgresTests, ServiceLinkWorkerConsentPostgresTests and ServiceLinkOutboundJournalPostgresTests. It also covers the four measured Core fixtures: LinuxInstallerTransactionTests, LinuxUpdaterRollbackTests, AgentGatewayRenewalTerminalTests and RemoteSupportReplicaSafeClusterPostgresTests. The default lane separately excludes full physical-client proof, sustained gateway/native install/update/PTY execution, real-time recovery/soak, the entire Playwright assembly and deployment/upgrade rehearsals. Cheap component, local API and focused PostgreSQL/security regressions remain selected; the redundant provider invocation and smoke-only image builds are removed.

The actual PR TRXs exclude all 12 newly manual classes and retain, among others, token-cache 70P, canonical JSON 56P, payload normalization 32P, dynamic network policy 8P, profile-token PostgreSQL expiry 13P, PostgreSQL provider 3P and ordinary gateway 9P.

Optional integration validation is workflow_dispatch only, with one selected group and no automatic dependencies or matrix fanout:

  • service-link, gateway, browser, deployment
  • release-images, native-client, physical-incident
  • upgrade-local, upgrade-oidc

For example: gh workflow run integration-validation.yml --repo BostonTechnologies/netratel --ref main -f suite=browser -f source_ref=main. Published-image and upgrade choices require an explicit existing completed release tag. Physical incident testing requires a selected source containing the original connector helper. Each requested group records the actual tested source and relevant peer/artifact identities, preserves existing assertions/timeouts/cleanup, and retains generated TRX, screenshots and receipts alongside its job logs. A requested failure remains a failure.

Release packaging stays on GitHub-hosted Actions. Exact tag/version/source, successful build receipts, five fixed image repositories, scans, immutable digests, anonymous pulls, checksums and partial-publication recovery remain required. Publication metadata now distinguishes completed build/distribution integrity from functional acceptance pending owner testing while retaining legacy consumer compatibility. Actual subsequent beta.1 packaging took 17m40s through its build aggregate; natural publication took 5m28s through its final job (5m29s to terminal metadata).

Historical functional observations remain unresolved: #572 timed out for 30 seconds waiting for monitoring-clear; #573 later passed all 87 browser cases, which does not establish the earlier cause. The dedicated physical incident hit its 300-second timeout, and the cancelled 40-minute .NET run had no completed generic API TRX. Moving these checks to manual selection does not turn those results into passes or establish a product fix.

Both actual natural runs are verified successful below 1,200 seconds, so the focused CI issue is complete. The policy has also been normally merged into connector PR #152 at 979e832fb0b07fafe6ce922f48442bbb64a05e5a, with fresh validation. Merge/release followthrough must use its actual results and preserve its feature fixes before epic #141's requested v0.1.1-beta.1 hosted build/publication. Owner functional acceptance remains pending and non-gating; beta.2 remains separate work.

Release follow-through completed — 7 October 2026

Connector #152 was normally merged at 8cb37bcaf5f5a2e290327fb38efb185eabf44391. Its final natural Main validation passed in 13m32s with 3,156 Core + 35 API + 400 Components passes, zero failed/skipped cases. The exact annotated tag v0.1.1-beta.1 was built and normally published once: six-job build, seven-job publication, public release and owner guide. All 13 public download byte/digest/checksum records, ten strict build-provenance file verifications, authenticated derived Compose binding and five anonymous OCI identities passed. The five prior stable image digests and stable/latest release remain unchanged. Owner functional acceptance is still pending; no optional suite was run as an automatic release gate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions