feat: publish ShellRisk-Bench v0.1

.github/workflows/test.yml

@@ -0,0 +1,22 @@
+name: test
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: python -m pip install -e '.[test]'
+      - run: python -m pytest -q
+      - run: python -m compileall -q adapters shellrisk_bench tests
+

.gitignore

@@ -0,0 +1,12 @@
+.DS_Store
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+*.egg-info/
+build/
+dist/
+data/raw/
+data/normalized/
+data/splits/
+

CITATION.cff

@@ -0,0 +1,12 @@
+cff-version: 1.2.0
+title: ShellRisk-Bench
+message: If you use ShellRisk-Bench, please cite this repository.
+type: software
+authors:
+  - name: Kontext
+version: 0.1.0
+date-released: 2026-08-18
+repository-code: https://github.com/kontext-security/shellrisk-bench
+url: https://kontext.security
+license: Apache-2.0
+

DATASETS.md

@@ -0,0 +1,36 @@
+# Dataset provenance
+
+ShellRisk-Bench downloads six upstream sources at pinned revisions and writes normalized JSONL locally. No upstream command data is committed to this repository.
+
+The repository's Apache-2.0 license applies only to ShellRisk-Bench code and documentation. It does not relicense upstream datasets or generated data. Users are responsible for complying with upstream terms.
+
+| Key | Class | Pinned revision | Command rows | Upstream license | Transformation and label basis |
+|---|---|---|---:|---|---|
+| `swesmith` | Not risky | `750b2c11239fd5e32f97e6cfb9bf80fb9a9a2983` | 95,825 | [MIT](https://huggingface.co/datasets/Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k/blob/750b2c11239fd5e32f97e6cfb9bf80fb9a9a2983/README.md) | Extract fenced Bash submissions from benign SWE-smith agent trajectories. Exact deduplication, shape cap, and heredoc cap. Label is task-inferred, not human-verified. |
+| `terminalbench` | Not risky | `04e8940f5b6736a7ce8d22224fe2f2af74163ed2` | 53,959 | [Apache-2.0](https://huggingface.co/datasets/yoonholee/terminalbench-trajectories/tree/04e8940f5b6736a7ce8d22224fe2f2af74163ed2) | Keep literal `execute_bash` calls from benign terminal-task trajectories. Label is task-inferred, not human-verified. |
+| `nl2bash` | Not risky | `d6b9f5bdff45621d190134e31ab63b7bf7002190` | 10,624 | [MIT for `data/bash`](https://github.com/TellinaTool/nl2bash/blob/d6b9f5bdff45621d190134e31ab63b7bf7002190/README.md) | Use the command side of human-curated NL→Bash pairs; exact deduplication. |
+| `atomicredteam` | Risky | `5cdeb06642dbdfb3c595d4096cb9eea5f6434d8b` | 265 | [MIT](https://github.com/redcanaryco/atomic-red-team/blob/5cdeb06642dbdfb3c595d4096cb9eea5f6434d8b/LICENSE.txt) | Keep Bash/sh executors, substitute declared defaults, and scrub source-identifying markers. ATT&CK technique is retained as the source label. |
+| `gtfobins` | Risky | `acd524623f9c406acedd2754ebd9c2431f3675ad` | 681 | [GPL-3.0](https://github.com/GTFOBins/GTFOBins.github.io/blob/acd524623f9c406acedd2754ebd9c2431f3675ad/LICENSE) | Extract curated abuse snippets and scrub documentation-only placeholders. Function type is retained as the source label. |
+| `payloads` | Risky | `203bb0c0b290bf7c9158c32d43523b8d66f292c1` | 29 | No license file declared at the pinned revision | Extract Linux/interpreter code fences from InternalAllTheThings, the successor linked from PayloadsAllTheThings, and remove prompts/prose. The repository downloads at build time and does not redistribute the upstream content. |
+
+## Structural routing
+
+Single-line submissions are assigned to the `command` layer and kept whole, including pipelines and `&&`/`;` chains. Multi-line scripts are assigned to `session` and excluded from v0.1 evaluation.
+
+A sequence label is never propagated onto its component commands. A harmless `cd` does not become risky merely because it appeared inside an attacker session.
+
+## Shared normalization
+
+Every source receives the same deterministic substitutions:
+
+- URLs → `http://example.com`
+- IPv4-looking strings → `1.1.1.1`
+- Base64-looking blobs of 40 or more characters → `BASE64`
+
+These transformations abstract incidental hosts and payload bytes without removing the surrounding command structure.
+
+## Global cleaning and split
+
+The split builder operates on `layer == "command"` only. It globally deduplicates exact command strings, removes commands observed with both labels, retains all risky commands, deterministically samples 20,000 benign commands, and performs an 80/20 stratified split with seed 13.
+
+The benchmark contains source-provided and inferred labels, not an independent expert relabeling of every command. This limitation is part of the benchmark and should accompany reported results.

NOTICE

@@ -0,0 +1,8 @@
+ShellRisk-Bench
+Copyright 2026 Kontext
+
+This product includes software developed by Kontext.
+
+Upstream datasets are not included in the distribution and remain subject to
+their respective licenses and terms, documented in DATASETS.md.
+

README.md

@@ -1,2 +1,99 @@
-# shellrisk-bench
-A reproducible benchmark for context-free shell-command risk classification
+# ShellRisk-Bench
+
+ShellRisk-Bench is a reproducible benchmark for **context-free risk classification of individual shell commands**. It tests whether a system can distinguish generally risky commands from ordinary development and operations traffic when it sees only the submitted command.
+
+The benchmark is built from six pinned public sources. Upstream data is downloaded locally and normalized into a common schema; it is not vendored in this repository.
+
+## Scope
+
+ShellRisk-Bench v0.1 evaluates one binary question:
+
+> Does this individual shell-command submission pose meaningful cyber or system risk?
+
+It does **not** infer user intent, inspect surrounding task context, evaluate multi-command sessions, or make a final authorization decision. Multi-line scripts and sessions are excluded rather than flattened into misleading atomic labels.
+
+## Sources
+
+| Class | Source | Command rows | Label basis |
+|---|---|---:|---|
+| Not risky | [SWE-smith trajectories](https://huggingface.co/datasets/Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k) | 95,825 | Inferred from benign software-engineering tasks |
+| Not risky | [Terminal-Bench trajectories](https://huggingface.co/datasets/yoonholee/terminalbench-trajectories) | 53,959 | Inferred from benign terminal tasks |
+| Not risky | [nl2bash](https://github.com/TellinaTool/nl2bash) | 10,624 | Human-curated command corpus |
+| Risky | [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team) | 265 | Executable ATT&CK tests |
+| Risky | [GTFOBins](https://github.com/GTFOBins/GTFOBins.github.io) | 681 | Curated binary-abuse techniques |
+| Risky | [PayloadsAllTheThings / InternalAllTheThings](https://github.com/swisskyrepo/InternalAllTheThings) | 29 | Curated offensive shell payloads |
+
+See [DATASETS.md](DATASETS.md) for pinned revisions, transformations, provenance, and upstream license information.
+
+## Headline split
+
+The published comparison uses a deterministic same-source, in-distribution split:
+
+- 161,383 command rows before global deduplication
+- 160,220 unique command strings
+- 9 cross-label collisions removed
+- All 966 risky commands retained
+- Benign commands deterministically capped at 20,000
+- Stratified 80/20 split with seed 13
+- Test set: 4,194 commands—193 risky and 4,001 not risky
+
+This split measures performance on unseen strings from **known source distributions**. It is not a source-transfer result. Source-grouped evaluation is the appropriate test for novel command dialects; every generated row retains its source so that evaluation can be performed separately.
+
+The approximately 20:1 test mix is a constructed operating point for comparing precision under class imbalance. It is not presented as an empirical measurement of all production shell traffic.
+
+## Published results
+
+| System | Precision | Recall | F1 | Mean latency |
+|---|---:|---:|---:|---:|
+| k-cl-1 | 0.947 | 0.922 | 0.934 | 22 µs |
+| Claude Opus 4.8 | 0.515 | 0.549 | 0.531 | 1.33 s |
+| Claude Sonnet 5 | 0.547 | 0.456 | 0.497 | 2.64 s |
+| Kimi K3 | 0.469 | 0.518 | 0.493 | 8.65 s |
+| GPT-5.5 | 0.393 | 0.456 | 0.422 | 2.74 s |
+| Claude Haiku 4.5 | 0.292 | 0.560 | 0.384 | 0.94 s |
+| GPT-5.6-terra | 0.286 | 0.430 | 0.344 | 1.44 s |
+| GPT-5.6-luna | 0.269 | 0.446 | 0.335 | 1.66 s |
+| Shieldstral 1.0 (3B, local) | 0.406 | 0.269 | 0.324 | 186 ms |
+| Llama Guard 4 (12B) | 0.023 | 0.285 | 0.042 | 1.1 s |
+
+All systems were scored on the same 4,194 commands. Hosted-model latency was measured sequentially and includes the API round trip. Shieldstral used its default 0.5 threshold; Llama Guard counted any unsafe category as risky. Quality results, prompts, and the k-cl-1 per-example verdicts are under [`results/`](results/). The k-cl-1 implementation and weights are not part of this benchmark repository.
+
+## Build
+
+Python 3.11 or newer is recommended.
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install -e '.[test]'
+.venv/bin/python -m shellrisk_bench.build
+.venv/bin/python -m shellrisk_bench.prepare --verify
+```
+
+The first command downloads several gigabytes of upstream trajectories. For a quick adapter smoke test:
+
+```bash
+.venv/bin/python adapters/swesmith.py --limit 50
+.venv/bin/python adapters/terminalbench.py --limit 50
+```
+
+After building the fixed split, score a JSONL prediction file:
+
+```bash
+.venv/bin/python -m shellrisk_bench.score \
+  --gold data/splits/test.jsonl \
+  --predictions results/k-cl-1.predictions.jsonl
+```
+
+Prediction rows contain a stable command hash and a binary verdict:
+
+```json
+{"id":"sha256:…","prediction":"risky"}
+```
+
+## Safety
+
+This repository processes potentially destructive commands as inert text. Nothing in the build or evaluation path executes benchmark commands. Do not pipe dataset contents into a shell.
+
+## License
+
+The benchmark code is licensed under Apache-2.0. Upstream datasets retain their own licenses and terms; see [DATASETS.md](DATASETS.md). The generated dataset is intentionally git-ignored and is not redistributed here.

SCHEMA.md

@@ -0,0 +1,54 @@
+# Schemas
+
+## Normalized adapter row
+
+Adapters write one JSON object per line under `data/normalized/<source>.jsonl`:
+
+```json
+{
+  "id": "atomicredteam-000042",
+  "source": "atomicredteam",
+  "layer": "command",
+  "actions": [{"type": "shell", "content": "example command"}],
+  "context": null,
+  "policy": null,
+  "label": {
+    "verdict": "deny",
+    "source_label": "T1059.004",
+    "rationale": null
+  },
+  "metadata": {
+    "provenance": {
+      "primary_source": "Atomic Red Team (Red Canary)",
+      "edited": "description of transformations"
+    }
+  }
+}
+```
+
+`allow` maps to not risky and `deny` maps to risky for v0.1. The upstream label remains available in `source_label`.
+
+## Prepared split row
+
+`shellrisk_bench.prepare` writes simplified rows to `data/splits/{train,test}.jsonl`:
+
+```json
+{
+  "id": "sha256:<full command digest>",
+  "source": "atomicredteam",
+  "upstream_id": "atomicredteam-000042",
+  "command": "example command",
+  "label": "risky"
+}
+```
+
+The SHA-256 identifier is derived from the normalized command string and is stable across builds.
+
+## Prediction row
+
+```json
+{"id":"sha256:<full command digest>","prediction":"risky"}
+```
+
+Accepted predictions are `risky`/`not_risky`, `deny`/`allow`, or `1`/`0`.
+

SECURITY.md

@@ -0,0 +1,6 @@
+# Security policy
+
+ShellRisk-Bench contains offensive and potentially destructive shell commands as inert text. The adapters and evaluation tools must never execute command contents.
+
+Please report vulnerabilities privately through [GitHub Security Advisories](https://github.com/kontext-security/shellrisk-bench/security/advisories/new). Do not include secrets or live infrastructure details in a public issue.
+