Repository navigation
epic: migrate all scanner rules to the evaluate() coverage contract #380
Description
What problem does this solve?
scanner/evaluation.py defines a per-resource coverage contract (#263). A rule exposes evaluate() and reports PASS / FAIL / UNKNOWN / ERROR / NOT_APPLICABLE for every resource it inspected, instead of only reporting violations through scan().
Only 1 of 144 rules (AZ-KV-006) implements it. The engine records the other 143 as UNKNOWN / LEGACY_RULE_NOT_MIGRATED. Since compliance reports became evaluation-derived in #310, most controls in the CIS, NIST CSF, ISO 27001:2022 and SOC 2 reports read UNKNOWN, so OpenShield cannot yet say which controls passed.
This epic tracks migrating every rule to the contract. It supports the October–December 2026 roadmap goal: stabilize the scanner and REST API contracts for a supported 1.0 release.
Describe the solution
Land a small foundation first, then migrate one rule family per issue. Every rule is covered exactly once (144 total).
Foundation (core)
- feat(scanner): evaluate() contract foundation - fallible inventory, shared helpers, single engine path #369 Fallible inventory calls, shared evaluation helpers, single engine path, contract test,
AZ-KV-006fix
Core-team migrations
- feat(scanner): migrate storage rules AZ-STOR-001 to 010 to the evaluate() contract #370 Storage,
AZ-STOR-001to010(10 rules, reference implementation) - feat(scanner): migrate network rules AZ-NET-001 to 027 to the evaluate() contract #371 Network,
AZ-NET-001to027(27 rules, two PRs) - feat(scanner): migrate identity rules AZ-IDN-001 to 025 to the evaluate() contract #372 Identity,
AZ-IDN-001to025(25 rules) - feat(scanner): migrate AKS rules AZ-AKS-001 to 021 to the evaluate() contract #373 AKS,
AZ-AKS-001to021(21 rules) - feat(scanner): migrate compute and PQC rules (AZ-CMP-001 to 007, AZ-PQC-001 to 003) to the evaluate() contract #374 Compute and PQC,
AZ-CMP-001to007andAZ-PQC-001to003(10 rules) - feat(scanner): migrate security operations and supply chain rules (AZ-SECOPS-001 to 010, AZ-SC-001 to 008) to the evaluate() contract #375 Security operations and supply chain,
AZ-SECOPS-001to010andAZ-SC-001to008(18 rules)
Good first issues
- feat(scanner): migrate Key Vault rules AZ-KV-001 to 005 to the evaluate() contract #376 Key Vault,
AZ-KV-001to005(5 rules) - feat(scanner): migrate SQL and PostgreSQL rules AZ-DB-001 to 007 to the evaluate() contract #377 SQL and PostgreSQL,
AZ-DB-001to007(7 rules) - feat(scanner): migrate Cosmos DB, Redis Cache and data link rules to the evaluate() contract #378 Cosmos DB, Redis Cache and data link (5 rules)
- feat(scanner): migrate Function App, Backup and Private Endpoint rules to the evaluate() contract #379 Function App, Backup and Private Endpoint via
_enterprise_resilience_common.py(15 rules)
Related
- Add rule for Azure SQL TLS version enforcement (AZ-SQL-TLS-001) #246 AZ-SQL-TLS-001 (new rule): should be written with
evaluate()from the start. - Implement enterprise application identity and OAuth security rules #254 Enterprise application identity rules: coordinate with feat(scanner): migrate identity rules AZ-IDN-001 to 025 to the evaluate() contract #372.
Definition of done
- Every rule module in
scanner/rules/az_*.pyexposesevaluate(). - No scan result contains
LEGACY_RULE_NOT_MIGRATED. - A failed Azure inventory call never produces
PASSorNOT_APPLICABLEfor any rule (enforced bytests/test_rule_evaluation_contract.py). - Finding output is unchanged for every migrated rule.
- The legacy
scan()-only path inscanner/engine.pyand theLEGACY_RULE_NOT_MIGRATEDplaceholder are removed in a final cleanup PR, anddocs/adding-a-rule.mdrequiresevaluate().
Alternatives considered
- Infer PASS from the absence of findings. Rejected in feat: persist PASS/FAIL/ERROR/NOT_APPLICABLE per rule per resource, fix compliance score #263/fix(compliance): make framework reports evidence-based and non-certifying #310: a crashed, skipped or permission-blocked rule would read as compliant.
- Migrate all 143 rules in one PR. Rejected as unreviewable. Families let contributors work in parallel.
Additional context
The foundation (#369) blocks every migration issue. Good first issues carry the blocked label until it merges.
Activity
SHAURYAKSHARMA24 commentedon Oct 9, 2026
@m-khan-97 @Vishnu2707, a question on scope and ownership for the piece that sits next to this epic.
#381 and #382 get the evaluation layer right. When an inventory call fails (403, 429, expired credential, a pager that dies halfway), migrated rules now report
ERROR/INVENTORY_UNAVAILABLEand show up infailed_rule_ids, where dev reports KV-006 asNOT_APPLICABLE. I checked this by driving the realAzureClientwith the SDK clients patched to fail. Compliance already treats ERROR/UNKNOWN as not passing.What none of the open work covers yet is the scan result itself. In every one of those failure cases the scan still finishes as
completedwith score 100, on dev and on the PR heads, becausescore_findings()only counts findings and the scan has no notion of what it couldn't see./api/scorereturns"status": "OK", and the dashboard shows the 100. Drift has the same problem from the other side: it diffs the last two scans with findings, across subscriptions, so a finding that wasn't observed shows up as REMOVED. That's the gap the #305 gate describes ("failed or incomplete evidence cannot produce PASS, a clean scan, or score 100"), and the evaluate() migration alone doesn't close it.What I'd like to propose, as a separate issue/PR on top of #381:
coverageblock to the scan result and keep three things apart: rule execution (rules loaded vs. rules with usable outcomes), inventory completeness per collector (only for collectors that can report failure, i.e. thelist_*ones from feat(scanner): add the evaluate() contract foundation (#369) #381;NOT_MEASURABLEfor the rest), and resource assessment (PASS/FAIL vs. resources actually observed, migrated rules only). No percentage unless its denominator is something we actually observed.score_status(COMPLETE/INCOMPLETE/NOT_MEASURABLE) next to the score in the scan result,/api/scoreand the dashboard. The score formula stays as it is, so risk and coverage remain separate numbers, but a score is never shown without saying whether it covers everything. Nothing gets a PASS it didn't earn.The tests would be deterministic and offline: empty subscription, compliant, violating, 403, 429, expired credential, failure mid-pagination, one collector failing while others succeed, a previously failing resource becoming unobservable, and a retry restoring coverage.
On #326: lifecycle should read the same signal (evaluations plus
failed_rule_ids) rather than a separate outcome list. I've left details on that PR. The coverage block wouldn't depend on #326 either way.I'm happy to own this if it fits how you want the gate handled. I'd start the design once you're both OK with the direction, and the implementation once #381 merges. If you'd rather it live under #305 or with someone already working in this area, that's fine too.