Environment Progression should consider Verification Status for success rate calculation

[jsbroks]

Problem

When evaluating environment progression policies, the success rate calculation only considers JobStatus (e.g., successful, failure) but does not consider JobVerificationStatus (e.g., passed, failed, running).

This means a job could have JobStatus: successful but JobVerificationStatus: failed, and the environment progression policy would still count it as a successful deployment for calculating the success rate percentage.

Current Behavior

In apps/workspace-engine/pkg/workspace/releasemanager/policy/evaluator/environmentprogression/jobtracker.go:

if t.SuccessStatuses[job.Status] && job.CompletedAt != nil {
    targetKey := rt.Key()
    // Store the oldest successful completion time for this release target
    if existingTime, exists := t.successfulReleaseTargets[targetKey]; !exists || job.CompletedAt.Before(existingTime) {
        t.successfulReleaseTargets[targetKey] = *job.CompletedAt
    }
}

The code only checks job.Status against SuccessStatuses, with no consideration of verification outcomes.

Expected Behavior

The EnvironmentProgressionRule should have an option to require verification to pass before counting a job as successful for progression purposes. Possible approaches:

Option 1: Add requireVerificationPassed flag

Add a boolean flag to EnvironmentProgressionRule:

requireVerificationPassed: {
  type: 'boolean',
  default: false,
  description: 'If true, jobs must also have passed verification to count toward the success percentage',
},

Option 2: Add verification statuses to success criteria

Allow specifying required verification statuses alongside job statuses:

requiredVerificationStatuses: {
  type: 'array',
  items: { $ref: '#/components/schemas/JobVerificationStatus' },
  description: 'Verification statuses that must be met for a job to count as successful',
},

Impact

Without this fix, users cannot safely use environment progression policies when they also have verification rules, as a deployment with failed verification could still trigger progression to subsequent environments.

Related Files

• apps/workspace-engine/pkg/workspace/releasemanager/policy/evaluator/environmentprogression/jobtracker.go
• apps/workspace-engine/pkg/workspace/releasemanager/policy/evaluator/environmentprogression/environmentprogression.go
• apps/workspace-engine/pkg/workspace/releasemanager/policy/evaluator/environmentprogression/passrate.go
• apps/workspace-engine/oapi/spec/schemas/policy.jsonnet (EnvironmentProgressionRule schema)

[adityachoudhari26]

@claude fix this

[claude]

Claude finished @adityachoudhari26's task in 13m 42s —— View job • claude/issue-742-20260410-2146 • Create PR ➔

---

Fixing Environment Progression Verification Status

• Read and analyze relevant files
• Add require_verification_passed column to DB schema and migration
• Update API OpenAPI schema (apps/api/openapi/schemas/policies.jsonnet, openapi.json)
• Update API route handler (apps/api/src/routes/v1/workspaces/policies.ts)
• Update workspace-engine OpenAPI schema (apps/workspace-engine/oapi/spec/schemas/policy.jsonnet)
• Update generated Go oapi struct (oapi.gen.go)
• Update DB queries to include require_verification_passed
• Update generated DB Go code
• Update ReleaseTargetJob struct to include VerificationStatus
• Update SQL query to compute verification status per job
• Update jobtracker.go to check verification status
• Update environmentprogression.go to pass the new flag
• Update convert.go to map the new field
• Update mock and add tests
• Commit and push changes

Summary

Implemented the requireVerificationPassed flag (Option 1 from the issue) for EnvironmentProgressionRule. Here's what was changed:

Database:

• Added require_verification_passed BOOLEAN NOT NULL DEFAULT FALSE column to policy_rule_environment_progression table
• Created migration 0186_add_require_verification_passed.sql

API:

• Added requireVerificationPassed field to EnvironmentProgressionRule schema in OpenAPI specs
• Updated the API route handler to persist and return the new field

Workspace Engine (core logic):

• Updated ListJobsByEnvironmentAndVersion SQL query to compute aggregate verification status per job inline, using the existing job_verification_metric and job_verification_metric_measurement tables
• Added VerificationStatus string to ReleaseTargetJob struct
• Added RequireVerificationPassed bool to ReleaseTargetJobTracker
• When RequireVerificationPassed=true, jobs with VerificationStatus == "failed" or "running" are excluded from the success rate calculation. Jobs with no verification metrics (empty string) or "passed" still count.
• Added 4 new test cases covering the new behavior