feat: score merged scanner ground truth per application

benchmarks/README.md

@@ -359,13 +359,30 @@ The HTTP response contains the same JSON that gets persisted to disk.
 
 ## Output format
 
-The output schema is the same for DAST and SAST. The fields inside each
+Schema version 2 is shared by DAST and SAST. Existing aggregate metrics remain;
+`applications` contains the same metrics and item lists for each application.
+The fields inside each
 `missedItems` / `unmatchedItems` entry vary by scan type (unused fields are
 omitted from the JSON).
 
 ```json
 {
+  "schemaVersion": 2,
   "tool": "ZAP",
+  "groundTruthFallback": false,
+  "applications": {
+    "VulnerableApp": {
+      "schemaVersion": 2,
+      "tool": "ZAP",
+      "coverage": 4.29,
+      "totalExpected": 140,
+      "detected": 6,
+      "missed": 134,
+      "unmatched": 147,
+      "missedItems": [{ "application": "VulnerableApp", "url": "/...", "type": "..." }],
+      "unmatchedItems": [{ "application": "VulnerableApp", "url": "/...", "cwe": "..." }]
+    }
+  },
   "coverage": 4.29,
   "totalExpected": 140,
   "detected": 6,
@@ -388,25 +405,57 @@ For SAST runs, items look like:
   `(URL, vulnerabilityType)` pairs across all `UNSECURE` ground-truth entries
   (SECURE entries are intentionally clean and don't count). For SAST, count of
   rows in the CSV.
+- `schemaVersion` — `2` identifies reports with per-application scope. Existing
+  checked-in reports retain their original schema until the next benchmark run.
+- `groundTruthFallback` — `true` when the remote source failed and only local
+  VulnerableApp ground truth was scored. Failure of the local source still fails
+  the run; it does not become a successful empty benchmark.
+- `applications` — metrics grouped by the application keys supplied by the facade.
+  Aggregate coverage is computed from total counts, not an average of percentages.
 - `missedItems` — expected items the scanner did not report.
 - `unmatchedItems` — items the scanner reported that don't line up with any
   expected ground-truth row.
 
+### Finding attribution
+
+Findings may include `"application": "VulnerableApp-jsp"`; the name must match a
+facade response key. This scopes matching and deduplication to that application.
+Legacy findings without this field still work: a single-app run assigns them to
+that app; multi-app DAST uses a URL unique to one app (including secure routes
+when determining ownership), then an application URL prefix for unknown routes.
+SAST uses a file path unique to one app. Ambiguous or unknown locations
+remain in aggregate `unmatchedItems` without an application label and are not
+credited to any app. Supply `application` when apps share paths. An explicit
+unknown application is reported separately with no expected rows.
+
 ## Configuration
 
 | Property                                         | Default                                                                                          | Purpose                                                                                                                                        |
 |--------------------------------------------------|--------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
 | `benchmark.output.dir`                           | `benchmarks`                                                                                     | Directory the JSON report is written to.                                                                                                       |
 | `benchmark.dast.ground-truth.url`                | `http://localhost:${server.port:9090}${server.servlet.context-path:/VulnerableApp}/scanner/dast` | URL the DAST comparator fetches ground truth from. Override when running behind VulnerableApp-facade so coverage spans every backing app.      |
-| `benchmark.dast.ground-truth.connect-timeout-ms` | `5000`                                                                                           | Connect timeout (ms) for the ground-truth fetch. Fail-fast bound so a stalled endpoint can't tie up Tomcat request threads.                    |
-| `benchmark.dast.ground-truth.read-timeout-ms`    | `10000`                                                                                          | Read timeout (ms) for the ground-truth fetch.                                                                                                  |
+| `benchmark.application-name`                     | `VulnerableApp`                                                                                  | Name used for standalone arrays and local ground truth.                                                                                        |
+| `benchmark.dast.ground-truth.fallback-url`       | Local `/scanner/dast` URL, using the configured port and context path                            | Local source used if the configured DAST URL fails or is blank.                                                                                |
+| `benchmark.sast.ground-truth.url`                | Empty                                                                                            | Optional facade `/scanner/sast` URL. Empty uses the local CSV directly; HTTP/parse failures fall back to that CSV.                             |
+| `benchmark.dast.ground-truth.connect-timeout-ms` | `5000`                                                                                           | Connect timeout (ms) for DAST and SAST HTTP ground-truth fetches. Fail-fast bound so a stalled endpoint can't tie up Tomcat request threads.   |
+| `benchmark.dast.ground-truth.read-timeout-ms`    | `10000`                                                                                          | Read timeout (ms) for DAST and SAST HTTP ground-truth fetches.                                                                                 |
 | `benchmark.sast.ground-truth.path`               | `classpath:scanner/sast/expectedIssues.csv`                                                      | CSV the SAST comparator loads expected issues from, and `/scanner/sast` serves. Drop the `classpath:` prefix to read a file from disk instead. |
 
 The default DAST URL is a self-call against the running app, which means
 benchmarking works out of the box in standalone mode. In a facade-composed
-deployment, point this at the facade's aggregated `/scanner/dast` endpoint so
-scanner findings are graded against the union of every backing app's ground
-truth.
+deployment, set `benchmark.dast.ground-truth.url` and
+`benchmark.sast.ground-truth.url` to the facade's `/scanner/dast` and
+`/scanner/sast`. Each benchmark fetches its selected endpoint once, without
+per-app requests. Both standalone arrays and facade objects keyed by application
+are accepted. Unavailable apps omitted by a successful facade response are
+outside that run's scope; the comparator does not discover them separately.
+An empty facade object (`{}`), emitted when every backing endpoint fails,
+triggers local fallback. Named applications with empty arrays remain valid.
+
+```properties
+benchmark.dast.ground-truth.url=http://facade/scanner/dast
+benchmark.sast.ground-truth.url=http://facade/scanner/sast
+```
 
 ## Where the report is written
 

src/main/java/org/sasanlabs/benchmark/model/BenchmarkResult.java

@@ -2,16 +2,49 @@
 
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
 /**
- * Result of comparing a scanner's findings against VulnerableApp's DAST ground truth.
+ * Result of comparing scanner findings against DAST or SAST ground truth.
  *
  * <p>{@code coverage} is detected / totalExpected as a percentage, or 0.0 when the ground truth is
  * empty.
  */
 public class BenchmarkResult {
 
+    @JsonProperty("schemaVersion")
+    private final int schemaVersion = 2;
+
+    @JsonInclude(JsonInclude.Include.NON_EMPTY)
+    @JsonProperty("applications")
+    private Map<String, BenchmarkResult> applications = Collections.emptyMap();
+
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    @JsonProperty("groundTruthFallback")
+    private Boolean groundTruthFallback;
+
+    public int getSchemaVersion() {
+        return schemaVersion;
+    }
+
+    public Map<String, BenchmarkResult> getApplications() {
+        return applications;
+    }
+
+    public void setApplications(Map<String, BenchmarkResult> applications) {
+        this.applications = applications;
+    }
+
+    public Boolean getGroundTruthFallback() {
+        return groundTruthFallback;
+    }
+
+    public void setGroundTruthFallback(Boolean fallback) {
+        this.groundTruthFallback = fallback;
+    }
+
     @JsonProperty("tool")
     private String tool;
 

src/main/java/org/sasanlabs/benchmark/model/Finding.java

@@ -34,6 +34,9 @@ public class Finding {
     @JsonProperty("method")
     private String method;
 
+    @JsonProperty("application")
+    private String application;
+
     @JsonCreator
     public Finding(
             @JsonProperty("url") String url,
@@ -42,14 +45,35 @@ public Finding(
             @JsonProperty("line") Integer line,
             @JsonProperty("cwe") String cwe,
             @JsonProperty("wascId") String wascId,
-            @JsonProperty("method") String method) {
+            @JsonProperty("method") String method,
+            @JsonProperty("application") String application) {
         this.url = url;
         this.type = type;
         this.filePath = filePath;
         this.line = line;
         this.cwe = cwe;
         this.wascId = wascId;
         this.method = method;
+        this.application = application;
+    }
+
+    public Finding(
+            String url,
+            String type,
+            String filePath,
+            Integer line,
+            String cwe,
+            String wascId,
+            String method) {
+        this(url, type, filePath, line, cwe, wascId, method, null);
+    }
+
+    public String getApplication() {
+        return application;
+    }
+
+    public Finding withApplication(String application) {
+        return new Finding(url, type, filePath, line, cwe, wascId, method, application);
     }
 
     public Finding(

src/main/java/org/sasanlabs/benchmark/service/ApplicationBenchmark.java

@@ -0,0 +1,156 @@
+package org.sasanlabs.benchmark.service;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiFunction;
+import org.sasanlabs.benchmark.model.BenchmarkResult;
+import org.sasanlabs.benchmark.model.Finding;
+import org.sasanlabs.benchmark.model.ScanType;
+import org.sasanlabs.benchmark.model.ScannerFindings;
+
+/**
+ * Partitions findings before scoring so identical locations in separate apps cannot cross-match.
+ */
+final class ApplicationBenchmark {
+    private ApplicationBenchmark() {}
+
+    static BenchmarkResult compare(
+            ScannerFindings input,
+            BenchmarkGroundTruthProvider.GroundTruth truth,
+            String contextPath,
+            BiFunction<ScannerFindings, List<Finding>, BenchmarkResult> scorer) {
+        Map<String, List<Finding>> grouped = new LinkedHashMap<>();
+        truth.applications.keySet().forEach(app -> grouped.put(app, new ArrayList<>()));
+        Map<String, Set<String>> owners = new LinkedHashMap<>();
+        for (Map.Entry<String, List<Finding>> app : truth.applications.entrySet()) {
+            for (Finding row : app.getValue()) {
+                String key = location(row, input.getScanType(), contextPath);
+                owners.computeIfAbsent(key, ignored -> new LinkedHashSet<>()).add(app.getKey());
+            }
+        }
+        Map<String, Set<String>> fullUrlOwners = new LinkedHashMap<>();
+        for (Map.Entry<String, Set<String>> app : truth.dastUrls.entrySet()) {
+            for (String url : app.getValue()) {
+                String fullPath = DastBenchmarkStrategy.normalizeUrl(url, "");
+                fullUrlOwners
+                        .computeIfAbsent(fullPath, ignored -> new LinkedHashSet<>())
+                        .add(app.getKey());
+                String key = DastBenchmarkStrategy.normalizeUrl(url, contextPath);
+                owners.computeIfAbsent(key, ignored -> new LinkedHashSet<>()).add(app.getKey());
+            }
+        }
+        List<Finding> unattributed = new ArrayList<>();
+        if (input.getFindings() != null) {
+            for (Finding finding : input.getFindings()) {
+                if (finding == null) {
+                    continue;
+                }
+                String app =
+                        resolveApplication(
+                                finding,
+                                input.getScanType(),
+                                truth.applications,
+                                contextPath,
+                                owners,
+                                fullUrlOwners);
+                if (app == null) {
+                    unattributed.add(finding.withApplication(null));
+                } else {
+                    grouped.computeIfAbsent(app, key -> new ArrayList<>())
+                            .add(finding.withApplication(app));
+                }
+            }
+        }
+        Map<String, BenchmarkResult> results = new LinkedHashMap<>();
+        List<Finding> missed = new ArrayList<>();
+        List<Finding> unmatched = new ArrayList<>();
+        int total = 0;
+        int detected = 0;
+        for (Map.Entry<String, List<Finding>> app : grouped.entrySet()) {
+            BenchmarkResult result =
+                    scorer.apply(
+                            new ScannerFindings(
+                                    input.getTool(), input.getScanType(), app.getValue()),
+                            truth.applications.getOrDefault(app.getKey(), Collections.emptyList()));
+            result.getMissedItems().replaceAll(f -> f.withApplication(app.getKey()));
+            result.getUnmatchedItems().replaceAll(f -> f.withApplication(app.getKey()));
+            results.put(app.getKey(), result);
+            total += result.getTotalExpected();
+            detected += result.getDetected();
+            missed.addAll(result.getMissedItems());
+            unmatched.addAll(result.getUnmatchedItems());
+        }
+        if (!unattributed.isEmpty()) {
+            BenchmarkResult unknown =
+                    scorer.apply(
+                            new ScannerFindings(input.getTool(), input.getScanType(), unattributed),
+                            Collections.emptyList());
+            unmatched.addAll(unknown.getUnmatchedItems());
+        }
+        BenchmarkResult result =
+                new BenchmarkResult(
+                        input.getTool(),
+                        total == 0 ? 0.0 : detected * 100.0 / total,
+                        total,
+                        detected,
+                        missed.size(),
+                        unmatched.size(),
+                        missed,
+                        unmatched);
+        result.setApplications(results);
+        result.setGroundTruthFallback(truth.fallback);
+        return result;
+    }
+
+    private static String resolveApplication(
+            Finding finding,
+            ScanType type,
+            Map<String, List<Finding>> apps,
+            String contextPath,
+            Map<String, Set<String>> owners,
+            Map<String, Set<String>> fullUrlOwners) {
+        if (finding.getApplication() != null && !finding.getApplication().isBlank()) {
+            return finding.getApplication();
+        }
+        if (apps.size() == 1) {
+            return apps.keySet().iterator().next();
+        }
+        String prefixOwner = null;
+        if (type == ScanType.DAST && finding.getUrl() != null) {
+            String path = DastBenchmarkStrategy.normalizeUrl(finding.getUrl(), "");
+            for (String app : apps.keySet()) {
+                if ((path.equals("/" + app) || path.startsWith("/" + app + "/"))
+                        && (prefixOwner == null || app.length() > prefixOwner.length())) {
+                    prefixOwner = app;
+                }
+            }
+            Set<String> fullOwners = fullUrlOwners.get(path);
+            boolean explicitContext =
+                    contextPath != null
+                            && !contextPath.isEmpty()
+                            && (path.equals(contextPath) || path.startsWith(contextPath + "/"));
+            if (fullOwners != null && (prefixOwner != null || explicitContext)) {
+                return fullOwners.size() == 1 ? fullOwners.iterator().next() : null;
+            }
+            if (prefixOwner != null || explicitContext) {
+                return prefixOwner;
+            }
+        }
+        Set<String> candidates = owners.get(location(finding, type, contextPath));
+        if (candidates != null) {
+            return candidates.size() == 1 ? candidates.iterator().next() : null;
+        }
+        return prefixOwner;
+    }
+
+    private static String location(Finding finding, ScanType type, String contextPath) {
+        return type == ScanType.SAST
+                ? SastBenchmarkStrategy.normalizeFilePath(finding.getFilePath())
+                : DastBenchmarkStrategy.normalizeUrl(finding.getUrl(), contextPath);
+    }
+}