Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
fbc7f5b
feat: render ruleset tracking and hunt provenance fields
vhmartinezm Aug 21, 2026
9847c8e
test: pin the hunt-page formatter legs
vhmartinezm Aug 21, 2026
674d43f
feat: rules list --include-counts
vhmartinezm Aug 21, 2026
ffa5e8a
test: pin the --include-counts wire plumbing
vhmartinezm Aug 21, 2026
ff1dc77
fix: only the flag passes include_counts; pin the legs to real SDK re…
vhmartinezm Aug 21, 2026
9b506fe
feat(rules): favorite command; zero-arg list on the stored-counter co…
vhmartinezm Aug 26, 2026
b430a14
test: re-record hunt cassettes against the stored-counter server
vhmartinezm Aug 26, 2026
871ede1
feat(cli): guard SDK-dependent options behind a signature check
sbneto Aug 28, 2026
ab55ecb
feat(hunt): filter rules list, and scope and bound live feed
sbneto Aug 28, 2026
78b93a5
test(hunt): pin the new options, their forwarding and floor degradation
sbneto Aug 28, 2026
c5f505f
fix(cli): name the SDK floor once, and refuse a negative --max-results
sbneto Aug 28, 2026
f0182f7
fix(tests,specs): guard the fixture tests on the attribute, not the c…
sbneto Aug 28, 2026
33680ea
fix(cli): 0 results is not a new option, and never render None of None
sbneto Aug 28, 2026
236ec78
fix(cli): point --livescan-id at the command that renders the badge
sbneto Aug 28, 2026
11c59c8
fix(cli): fail open on a **kwargs SDK, type the id, document the guards
sbneto Aug 28, 2026
45e4235
fix(tests): guard the re-recorded text cassettes on the floor too
sbneto Aug 28, 2026
e0312b0
docs(specs): keep the --since note out of the command table
sbneto Aug 28, 2026
4c6f247
fix(cli): finish the retractions and use the shared guard everywhere
sbneto Aug 28, 2026
b914239
fix(live): default --since to 86400 seconds
sbneto Aug 28, 2026
7528f36
fix(cli): harden the skip guards and finish two spec edits
sbneto Aug 28, 2026
7cdfc0c
fix(tests): skip the option-passing tests on a floor SDK
sbneto Aug 28, 2026
55defbd
docs(live): stop promising the feed matches the badge
sbneto Aug 28, 2026
cb7e91f
fix(tests): key each floor guard on every parameter its test passes
sbneto Aug 28, 2026
af2a130
fix(tests): keep the favorite command tests off the resource guard
sbneto Aug 28, 2026
65a54ea
fix(tests): guard the favorite cassette renders on the parsed resource
sbneto Aug 28, 2026
b69d902
fix(live): pin that --since 0 reaches the SDK, and trim the help text
sbneto Aug 28, 2026
fcdd02a
test(cli): pin the floor guard's fail-open branch, and fix specs/01
sbneto Aug 28, 2026
6cb030b
refactor: pin the SDK floor instead of probing it at runtime
sbneto Aug 31, 2026
d705bf5
refactor: read the tracking fields directly; the pin makes getattr dead
sbneto Aug 31, 2026
9219e5b
refactor: finish the getattr removal, and stop describing deleted mac…
sbneto Aug 31, 2026
3262e71
docs(specs): state the develop-merge hazard, not just release order
sbneto Aug 31, 2026
a481111
docs(specs): split the floor's merge and release preconditions
sbneto Aug 31, 2026
2108795
docs(specs): one authoritative statement of the floor, and drop stale…
sbneto Aug 31, 2026
c799622
fix(formatters): an unstarred ruleset must not render a Favorited at …
sbneto Aug 31, 2026
9f4a5c8
fix(rules): read the server's refusal off the documented envelope path
sbneto Aug 31, 2026
16f4984
docs(specs): lockstep on develop is the design, not a hazard
sbneto Aug 31, 2026
b591314
fix(cli): stop asserting a badge window the payload does not carry
sbneto Aug 31, 2026
d890543
docs(specs): the badge label names no window, and the specs now agree
sbneto Aug 31, 2026
00b632c
docs: the comment asserted the window the label just stopped naming
sbneto Aug 31, 2026
2e3ba9c
fix(rules): pass the favorite toggle by keyword so autospec can see it
sbneto Aug 31, 2026
f4cc948
docs(specs): resolve the VCR-off invariant against the one test that …
sbneto Aug 31, 2026
1e3d6ac
docs: reconcile AGENTS.md with the spec, and record what the exit cod…
sbneto Aug 31, 2026
f70ab8f
fix(live): give --since the same negative guard as --max-results, and…
sbneto Aug 31, 2026
b5b80c6
test: make the non-limit refusal test assert the fall-through it is n…
sbneto Aug 31, 2026
daa1d33
test: build the non-limit refusal the way production raises it
sbneto Aug 31, 2026
ae57555
Revert "docs(specs): resolve the VCR-off invariant against the one te…
sbneto Aug 31, 2026
41a7b6e
test: close two branches that no test reached, and stop asserting a d…
sbneto Aug 31, 2026
176f3a7
docs(specs): the re-record steps kept a pointer the revert removed, a…
sbneto Aug 31, 2026
0ef7a28
test: assert the two omission arms in the ruleset leg, and pin what -…
sbneto Aug 31, 2026
335b01d
test: drop the favorite-cap cassette, which its own siblings make uns…
sbneto Aug 31, 2026
e1b0830
docs: drop the dual-coverage carve-out for a cassette that no longer …
sbneto Aug 31, 2026
df925cf
fix(rules): guard the json envelope the way the errors mapping alread…
sbneto Aug 31, 2026
ac67341
docs: write down what happens to a ticket-prefixed branch name at merge
sbneto Aug 31, 2026
6869b2d
docs(specs): the favorite re-record recipe described a run that never…
sbneto Aug 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ Details in [`specs/04-testing.md`](./specs/04-testing.md). The shape:
- Conventional commit prefixes (`feat:`, `fix:`, `refactor:`, `chore:`, `docs:`, `test:`).
- Small, scoped commits — each one should be independently reviewable.
- **Don't reference ticket IDs or internal project codes in commit messages, PR titles, or PR descriptions.** This repo is public; published artefacts shouldn't leak internal references. Track tickets in the internal tracker, not the git history.
- **Branch names are the exception, and the merge is where they're contained.** A change spanning this repo and the SDK must use the *identical* branch name in both, because CI resolves the companion SDK by `$CI_COMMIT_BRANCH` (`.gitlab-ci.yml`) — so a shared ticket-prefixed name is often the coordinating key and is deliberately allowed. It does reach public history, but only through the **default merge subject** (`Merge pull request #N from org/TICKET-…`). **Squash-merge with an explicit clean subject**, and it never lands. Prefer a shared descriptive name over a ticket prefix when one reads just as well.
- **Don't name private companion repos in PR descriptions or commit messages on this repo.** Refer to internal services by category, not by repo name.
- No AI-attribution trailers on commits (`Co-Authored-By: Claude …`, "Generated with Claude Code", etc.) — they're noise and they don't belong in project history.
- PRs that depend on an unreleased `polyswarm-api` surface must link the SDK PR under a `## Requires` section (see `specs/05-sdk-contract.md`).
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ classifiers = [
]

dependencies = [
"polyswarm_api>=4.3.0,<5.0.0",
"polyswarm_api>=4.4.0,<5.0.0",
"click>=7.1",
"colorama>=0.4.6",
"click-log>=0.4.0",
Expand Down
11 changes: 10 additions & 1 deletion specs/01-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ The console-script entry (`__main__.py`) calls `polyswarm_cli(prog_name='polyswa

The transport-error branch matches by **ancestry class name** — it intersects `{c.__name__ for c in type(e).__mro__}` with `{'HTTPError', 'RequestException', 'ConnectionError', 'SSLError'}` — because those classes come from the SDK's HTTP dependency (`httpx`; `requests` historically) and shouldn't be imported here directly. `httpx` roots every request/transport/status error at `HTTPError`, so ancestry matching covers all its leaf classes (`ConnectError`, `ReadTimeout`, `RemoteProtocolError`, `ProxyError`, …) without enumerating them.

**The order of the `except` clauses is load-bearing, not stylistic.** The SDK has its own
`api_exceptions.RequestException`, which subclasses `PolyswarmException` but shares the bare
name `requests` used — so it satisfies the ancestry-name test above and would take the
transport branch (exit `1`, "contact support") if it ever reached it. It exits `2` only
because the `PolyswarmException` clause is matched **before** the transport branch. Reordering
those clauses silently changes the exit code of every SDK request refusal, `FAVORITE_LIMIT`
included; `ExitCodeHierarchyTest` pins the subclass relation the ordering rests on.

## The SDK wrapper — `polyswarm.py`

`class Polyswarm(PolyswarmAPI)` subclasses the SDK's sync client to add **CLI-only** behaviour the SDK has no reason to ship:
Expand Down Expand Up @@ -74,7 +82,8 @@ The catalogue of groups and the SDK methods each wraps is in [`02-commands.md`](

## Support — `utils.py`, `exceptions.py`

- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end), `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), and input parsing/validation (`parse_hashes`, hash/IP detection).
- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end) and `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), plus `collect_files` and the detection helpers (`is_valid_id`, `is_ip`, `is_domain`, `is_url`).
- **`client/utils.py`** — `parse_hashes` and the click parameter validators (`validate_id`, `validate_hash(es)`, `validate_key`, …). Note the module is `client/utils.py`, not the top-level `utils.py` above — the two are distinct and easily confused.
- **`exceptions.py`** — the CLI's own hierarchy, **distinct from the SDK's**: `PolyswarmException` → `NoResultsException`, `NotFoundException`, `InternalFailureException`, `PartialResultsException`. `ExceptionHandlingGroup` catches both these and the SDK's `api_exceptions.*`.

## Lifecycle of a command (end to end)
Expand Down
28 changes: 26 additions & 2 deletions specs/02-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,19 +25,43 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
| `report` (`report.py`) | Create/fetch/download reports; `prompt-config` subgroup; LLM reports | `report_create`, `report_wait_for`, `report_download`, `report_get`, `llm_report_{create,get,download}`, `prompt_config_{create,get,update,list}` |
| `report-template` (`report_template.py`) | Manage report templates + logos | `report_template_{create,update,get,list}`, `report_template_logo_{download,upload}` |
| `engine` → `votes` / `assertions` (`engine.py`) | Consolidated votes/assertions bundles per engine | `votes_{create,get,delete,list}`, `assertions_{create,get,delete,list}` |
| `live` (`live.py`) | Live YARA hunts: start/stop, feed, results | `live_start`, `live_stop`, `live_feed`, `live_result`, `live_feed_delete` |
| `live` (`live.py`) | Live YARA hunts: start/stop, feed, results. `feed` takes `--since` in **SECONDS** (default 86400 — 24h; `0` means no time filter at all, and a negative is refused at parse time rather than forwarded). **`0` is the API's contract, not a CLI convention:** the endpoint applies the window only when `since` is truthy, so `0` and an absent parameter behave identically. The CLI's own test can only pin that it forwards `0` rather than dropping it — which is the half that can regress here, plus `--livescan-id` (the drill-down for the per-ruleset new-results badge `rules list` renders — the detail view deliberately does not carry the badge; the badge counts the hunt across **every** community it runs in, public and private, while the feed shows one at a time, so a multi-community hunt lists fewer rows than the badge reports) and `--max-results` (stop after N; unset means every page, as before). Both are guaranteed by the pin (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor) and called directly; they are forwarded only when passed purely so a pre-existing invocation's call shape is unchanged — the request is identical either way | `live_start`, `live_stop`, `live_feed`, `live_result`, `live_feed_delete` |
| `historical` (`historical.py`) | Historical hunts: CRUD + results | `historical_{get,create,update,list}`, `historical_delete_multiple`, `historical_delete_list`, `historical_results_multiple`, `historical_result`, `historical_results_delete` |
| `tag` (`tags.py`) | Tag CRUD | `tag_{create,delete,get,list}` |
| `link` (`links.py`) | Tag/family links on artifacts | `tag_link_multiple`, `tag_link_get`, `tag_link_list` |
| `family` (`families.py`) | Malware-family CRUD | `family_{create,update,delete,get,list}` |
| `rules` (`rules.py`) | YARA ruleset CRUD | `ruleset_{create,delete,update,get,list}` |
| `rules` (`rules.py`) | YARA ruleset CRUD plus `favorite <id> [--unfavorite]` (the star toggle: renders the new state + the server-owned "N of M used" budget, and converts the machine-readable `FAVORITE_LIMIT` refusal into a clean actionable message at exit 2, never 1 — 1 is reserved for no-results/not-found; 2 is the broad bucket `ExceptionHandlingGroup` maps the PolyswarmException hierarchies to. **2 does not identify a server refusal:** click exits 2 for a `UsageError` too, so a scripted caller cannot tell “the favorite budget is full” from “you passed a bad flag” without reading the message). `list` takes the server-side filters `--name` / `--status active` / `--favorites-only` / `--has-new-results` (conjunctive; the list is keyset-paginated, so filtering locally would mean walking every page). `rules favorite` and the `rules list` filters need SDK 4.4.0, which the pin requires (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor), so they are called directly. The formatters read the hunt-page fields directly: the pin guarantees the SDK parses them, so `None` means the *server* had no answer | `ruleset_{create,delete,update,get,list,favorite}` |
| `metadata` (`metadata.py`) | Rerun metadata; scan lookup; IP/URL analysis | `rerun_metadata`, `scan_lookup`, `submit_url` |
| `activity` (`event.py`) | List account activity/events | `event_list` |
| `account` (`account.py`) | Account whois / features | `account_whois`, `account_features` |
| `notification` / webhooks (`notification.py`, `notification_webhook.py`) | Notification webhook CRUD | `notification_webhook_{create,get,update,delete,list}` |
| `bundle` (`bundle.py`) | Sample bundle tasks | `sample_bundle_task_create`, `sample_bundle_task_get`, `sample_bundle_download` |
| `sample` (`sample.py`) | Fetch a consolidated sample view | `sample` |

> **`live feed --since` defaults to 86400 seconds (24h), not 1440.** The old
> default was written as `24 * 60` against an SDK docstring that said the
> parameter was minutes; the server has always read **seconds**, so the real
> default window was 24 minutes while the ruleset badge beside it counts 24
> hours. The fix is here rather than on the wire: the endpoint takes ~197k
> requests per 30 days carrying `since` from clients outside our control, and
> re-basing the server to minutes would widen every one of them 60x with no
> error. `historical list --since` is seconds too — those two agree.
>
> **Migration, for a caller who relied on the old behaviour:** pass
> `--since 1440` to get the 24-minute window back. Two effects compound and
> the second is the sharper one — the default window widens ~60x, and
> `--max-results` is unset by default, so a bare `live feed` pages through all
> of it rather than stopping. This repo has no CHANGELOG, so the note lives
> here and in the command's own `--help` rather than only in a release-time
> reminder.
>
> **A third `--since` is genuinely MINUTES and must stay that way:**
> `download stream --since` (`client/download.py`, `IntRange(1, 2880)`,
> default `1440`) hits a different endpoint that really does read minutes. That
> `1440` is the same literal `live feed` is being corrected away from, and is the
> likeliest origin of the original mistake — check which endpoint you are on
> before copying a default between them.

## Adding to the catalogue

When you add or materially change a group, update its row (and add per-subcommand detail here if the behaviour is non-obvious). The `AGENTS.md` §"When adding a new command family" checklist covers the wiring + formatter + test steps.
31 changes: 29 additions & 2 deletions specs/03-formatters.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ How command output is rendered: the `BaseOutput` interface, the concrete formatt

## The interface — `base.py`

`BaseOutput(output, **kwargs)` holds the output stream and exposes a method per resource type, each raising `NotImplementedError`. The set includes (non-exhaustive): `artifact_instance`, `historical_result`, `hunt`, `hunt_deletion`, `local_artifact`, `ruleset`, `ioc`, `iocs`, `known_host`, `metadata`, `artifact_metadata`, `tag_link`, `family`, `tag`, `known_good`, `sandbox_list`, `sandbox_task`, `sandbox_tasks`, `bundle_task`, `sample`. Concrete formatters add further methods as command families grow (e.g. `report_task`, `webhook`, `llm_prompt_config`, `metadata_field_properties`); keep `text` and `json` in sync.
`BaseOutput(output, **kwargs)` holds the output stream and exposes a method per resource type, each raising `NotImplementedError`. The set includes (non-exhaustive): `artifact_instance`, `historical_result`, `hunt`, `hunt_deletion`, `local_artifact`, `ruleset`, `ruleset_favorite`, `ioc`, `iocs`, `known_host`, `metadata`, `artifact_metadata`, `tag_link`, `family`, `tag`, `known_good`, `sandbox_list`, `sandbox_task`, `sandbox_tasks`, `bundle_task`, `sample`. Concrete formatters add further methods as command families grow (e.g. `report_task`, `webhook`, `llm_prompt_config`, `metadata_field_properties`); keep `text` and `json` in sync.

## Concrete formatters

Expand Down Expand Up @@ -142,7 +142,34 @@ either field) never raises `AttributeError`; an SDK without `.state` simply neve
the known-good branch, which is the safe fallback — the pre-known-good rendering. That
degradation is belt-and-braces, not a supported configuration: `.state` is load-bearing
here with no substitute. Both attributes ship in SDK **4.1.0**, but the dependency floor is
`polyswarm_api>=4.2.0` — set by two *other* behaviours the CLI depends on, both of which
the value in `pyproject.toml` — see [05-sdk-contract.md](./05-sdk-contract.md)
§Current floor, which is authoritative, since repeating the number here is what let this
line go stale before. Its *rationale* is two behaviours that landed in 4.2.0 and still hold
transitively; those two
fail silently on 4.1.0 (see [`05-sdk-contract.md`](./05-sdk-contract.md) §Version pin) — so
every supported install has them. `JSONOutput` needs no change — it dumps the resource's
`.json`, which already carries the raw `state` and `known_good` keys.

## Hunt-page tracking fields (rulesets + historical hunts)

Rendering rules that are deliberate, not incidental. Every one of these fields is
parsed by the pinned SDK, so the attribute always exists and `None` means the
**server** had no answer — never an older SDK (the floor forbids one; see
[`05-sdk-contract.md`](./05-sdk-contract.md) §Current floor). The formatters read
the attributes directly:

- `rule_count` / `historical_hunt_count`: `0` renders as a real zero;
`None` (the server had no answer) omits the line — never shown as 0.
- `favorite` is truthy-only ("Favorite: yes"): False and None both print
nothing, deliberately indistinguishable.
- `new_results_count` is the server's STORED badge (refreshed by its
scheduled job; the window is the server's and the response does not carry
it, so the label deliberately does not name one): a number renders with its
`new_results_counted_at` staleness marker beside it; `None` (never
refreshed / no live hunt) omits both lines.
- `ruleset_favorite` renders the toggle response: `Favorite: yes/no`, the
`favorited_at` timestamp when starred, and the server-owned budget as
"Favorites used: N of M" — the client never counts.
- `source_rule_changed` is tri-state: `None` means UNKNOWN, not "unchanged",
and prints nothing; the label names its reference point — "changed since
this hunt froze it" — so it cannot read as "edited recently".
Loading
Loading