Release 4.5.0

pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "polyswarm"
-version = "4.4.0"
+version = "4.5.0"
 description = "CLI for using the PolySwarm Customer APIs"
 readme = "README.md"
 authors = [{ name = "PolySwarm Developers", email = "info@polyswarm.io" }]
@@ -22,7 +22,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "polyswarm_api>=4.4.0,<5.0.0",
+    "polyswarm_api>=4.5.0,<5.0.0",
     "click>=7.1",
     "colorama>=0.4.6",
     "click-log>=0.4.0",
@@ -51,7 +51,7 @@ include-package-data = true
 where = ["src"]
 
 [tool.bumpversion]
-current_version = "4.4.0"
+current_version = "4.5.0"
 commit = true
 tag = false
 sign_tags = true

specs/02-commands.md

@@ -30,7 +30,7 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
 | `tag` (`tags.py`) | Tag CRUD | `tag_{create,delete,get,list}` |
 | `link` (`links.py`) | Tag/family links on artifacts | `tag_link_multiple`, `tag_link_get`, `tag_link_list` |
 | `family` (`families.py`) | Malware-family CRUD | `family_{create,update,delete,get,list}` |
-| `rules` (`rules.py`) | YARA ruleset CRUD plus `favorite <id> [--unfavorite]` (the star toggle: renders the new state + the server-owned "N of M used" budget, and converts the machine-readable `FAVORITE_LIMIT` refusal into a clean actionable message at exit 2, never 1 — 1 is reserved for no-results/not-found; 2 is the broad bucket `ExceptionHandlingGroup` maps the PolyswarmException hierarchies to. **2 does not identify a server refusal:** click exits 2 for a `UsageError` too, so a scripted caller cannot tell “the favorite budget is full” from “you passed a bad flag” without reading the message). `list` takes the server-side filters `--name` / `--status active` / `--favorites-only` / `--has-new-results` (conjunctive; the list is keyset-paginated, so filtering locally would mean walking every page). `rules favorite` and the `rules list` filters need SDK 4.4.0, which the pin requires (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor), so they are called directly. The formatters read the hunt-page fields directly: the pin guarantees the SDK parses them, so `None` means the *server* had no answer | `ruleset_{create,delete,update,get,list,favorite}` |
+| `rules` (`rules.py`) | YARA ruleset CRUD (`list` takes the server-side filters plus `--sort active-first`, the hunt page's order: rulesets carrying a live hunt link first, deduped by id because the key is mutable — neither the position nor a moved row's own fields are evidence a hunt is running, see [05-sdk-contract.md](./05-sdk-contract.md) §A mutable order makes the walk the caller's problem. Ordering is the server's, never a local re-sort of one keyset page) plus `favorite <id> [--unfavorite]` (the star toggle: renders the new state + the server-owned "N of M used" budget, and converts the machine-readable `FAVORITE_LIMIT` refusal into a clean actionable message at exit 2, never 1 — 1 is reserved for no-results/not-found; 2 is the broad bucket `ExceptionHandlingGroup` maps the PolyswarmException hierarchies to. **2 does not identify a server refusal:** click exits 2 for a `UsageError` too, so a scripted caller cannot tell “the favorite budget is full” from “you passed a bad flag” without reading the message). `list` takes the server-side filters `--name` / `--status active` / `--favorites-only` / `--exclude-favorites` (its inverse, refused together with it — for a client that lists the favorites separately) / `--has-new-results` (conjunctive; the list is keyset-paginated, so filtering locally would mean walking every page). `rules favorite` and the `rules list` filters need SDK 4.4.0 and `rules list --sort` needs 4.5.0, which the pin requires (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor), so they are called directly. The formatters read the hunt-page fields directly: the pin guarantees the SDK parses them, so `None` means the *server* had no answer | `ruleset_{create,delete,update,get,list,favorite}` |
 | `metadata` (`metadata.py`) | Rerun metadata; scan lookup; IP/URL analysis | `rerun_metadata`, `scan_lookup`, `submit_url` |
 | `activity` (`event.py`) | List account activity/events | `event_list` |
 | `account` (`account.py`) | Account whois / features | `account_whois`, `account_features` |

specs/03-formatters.md

@@ -173,3 +173,105 @@ the attributes directly:
 - `source_rule_changed` is tri-state: `None` means UNKNOWN, not "unchanged",
   and prints nothing; the label names its reference point — "changed since
   this hunt froze it" — so it cannot read as "edited recently".
+
+## Matched strings on hunt results
+
+`TextOutput.historical_result` / `TextOutput.live_result` render `result.matched_strings`
+— the yara strings behind a hit — between `Tags:` and `Download Url:`, via the shared
+`TextOutput._matched_strings` helper.
+
+The attribute is **three-state** (the SDK's `05-downstream-contract.md` is authoritative)
+and the three must stay distinguishable in the output — but they do **not** each get a
+line.
+
+| `matched_strings` | Rendered |
+|---|---|
+| `None` | *nothing* — no line is emitted |
+| `[]`, no count | `Matched Strings: none -- the rule matched without byte evidence (a structural or negative match, or private strings)` |
+| `[]`, count > 0 | `Matched Strings: none shown (N withheld, result size limit)` — the count **overrides** the row above, because asserting a structural match while discarding a withheld count would be a confident false statement. Should be unreachable; see below. |
+| `[…]` | `Matched Strings:` followed by one indented `  $ident @ 0xOFFSET (N bytes[, truncated]): DATA` line per entry |
+
+**The silent-`None` branch depends on list routes sending `null`, and that is measured
+rather than assumed.** If a list route ever returned `[]` per row instead, every row of a
+large hunt would carry the loud "matched without byte evidence" line — the permanent
+false alarm this design exists to avoid, arriving through the branch deliberately kept
+loud. The server pins it for **both** hunt pairs in artifact-index's
+`test_list_serializers_render_nulls_not_payloads`, which asserts the key is present-and-null on
+`ScanResultListSerializer` *and* `LiveResultListSerializer`, against fixture rows that do
+carry evidence. This repo cannot verify it; it relies on that test.
+
+Two constraints, both counter-intuitive enough to be worth stating:
+
+- **`None` emits nothing, and `[]` must not follow it into silence.** The instinct is to
+  explain the absence. Resist it: `None` overwhelmingly means "this is a list route",
+  which *can never* carry strings, so a line there is a permanent false alarm on every
+  row rather than information — and `live feed` loops over this same method, with nothing
+  on the resource to tell the routes apart (`live_feed` and `live_result` both yield a
+  `LiveHuntResult`). Route-awareness would mean threading a flag from the command layer
+  into a new parameter on **every** `BaseOutput` implementation (`text`, `json`, all three
+  `hashes` subclasses), which buys too little. `[]` is the opposite case and keeps its
+  line: it only ever reaches a detail route, and "the rule matched with no byte evidence"
+  is a real answer to "why did this hit". Since the analyzer always sends `strings` once
+  this feature ships, `None` on a detail route means a result predating it — nothing to
+  say.
+- **The entry keys are subscripted, deliberately.** `identifier`, `offset`, `length`,
+  `data` and `truncated` are read by subscript, because a partial entry is not version
+  skew but a producer violating its own contract, and rendering half a match as though it
+  were whole is worse than failing. The attribute itself needs no such defence: the floor
+  names an SDK that parses it (§Current floor in [`05-sdk-contract.md`](./05-sdk-contract.md)).
+- **`data` is sanitised before rendering.** It is the only sample-derived field in a hunt
+  result, so it is attacker-controlled end to end. yara escapes non-printables upstream
+  and the analyzer preserves that rendering, so `_safe_data` is a no-op on valid input —
+  it exists because the guarantee lives in another repo, and a raw CSI sequence reaching
+  a terminal would repaint or clear an analyst's screen.
+- **ASCII only, and true of this whole block.** The literals are ASCII; the two
+  server-supplied *string* fields — `data` and `identifier` — go through `_safe_data`;
+  and the three server-supplied *numbers* — `offset`, `length`, `dropped` — carry an
+  integer format spec (`:x` / `:d`), which is what pins them rather than `_safe_data`.
+  Fields *outside* the block (`rule_name`, `tags`) are unfiltered and outside this claim.
+  Stdout under a C/POSIX locale replaces non-ASCII with `?`.
+- **`truncated` is not a byte count.** The stored length is capped server-side, so the
+  marker means "there was more than this" and over-reports at exactly the cap. Never
+  render it as an exact size.
+
+**Read both attributes directly** — `result.matched_strings`, no `getattr`. The floor
+(`polyswarm_api>=4.4.0`, [`05-sdk-contract.md`](./05-sdk-contract.md) §Current floor)
+names an SDK that parses them, so `pip` refuses the install a probe would guard against;
+`specs/05-project-standards.md` §16 has the general rule. `None` then means exactly what
+the table above says — the *server* did not report — which is the reading the silent
+branch depends on.
+
+### The dropped-count line
+
+When `result.matched_strings_dropped` is non-zero, a final line is appended **inside** the
+block, in **yellow** rather than white:
+
+```
+  ... 19 more not shown (result size limit)
+```
+
+It is the one line here reporting something the platform withheld, which is why it is not
+white like the entries above it. Omitted entirely when the count is zero or `None`.
+
+**An empty list with a non-zero count does not claim "no byte evidence".** That
+combination should be unreachable — the analyzer keeps a match's first string, so
+`[] ⇒ dropped == 0` — but the renderer must not *depend* on an invariant owned by another
+repo while making a positive claim about the rule. It reports what is certain instead
+(`none shown (N withheld, result size limit)`), because asserting a structural match and
+discarding the count is the precise wrong inference this line exists to prevent.
+
+This is not cosmetic. Without it a truncated list reads as the whole truth and a user
+concludes their rule hit twice when it hit twenty-one times — the same wrong-inference
+class the three-state contract above exists to prevent, one level down.
+
+`JSONOutput` needs no change — it dumps the resource's `.json`, which already carries the
+raw `matched_strings` and `matched_strings_dropped` keys.
+
+Coverage is two modules. `tests/hunt_matched_strings_test.py` is Style 3 — the formatter
+driven directly with constructed SDK resources — and owns *which line a field value
+produces*. `tests/hunt_matched_strings_cli_test.py` is the Style 1 counterpart
+[`04-testing.md`](./04-testing.md) requires: it drives `live result` through `CliRunner`
+so a broken `output.extend` call is caught, which the Style 3 module cannot see because
+it calls `_matched_strings` itself. The `cli_test.py` cassettes predate the field, so
+every result they render takes the silent `None` branch — they pin that no stray line
+appears, and nothing more.

specs/05-sdk-contract.md

@@ -39,6 +39,32 @@ for item in api.iocs_by_hash(type, value):
 
 Because the generators are **lazy**, calling one does no I/O and raises nothing until iterated. Code that runs SDK calls through a thread pool must consume the generator **inside the worker** so per-item exception handling fires where it's expected — this is why `utils.parallel_executor_iterable_results` materialises each generator inside the submitted callable (see `01-architecture.md`).
 
+### A mutable order makes the walk the caller's problem
+
+Most list endpoints are keyset-paginated on an immutable key, so a walk sees every row once.
+`ruleset_list(sort='active_first')` is the exception in the current surface: its key is the
+live-hunt link, which the hunt itself flips, and the SDK's own docstring puts the consequence
+on the caller — *"This generator streams pages and does not dedupe — dedupe by `id` if you
+consume more than one page"*.
+
+A command that walks every page of such an endpoint must therefore:
+
+- **Dedupe by `id`.** A row whose key changes mid-walk drops below the cursor and is served
+  again. `rules list` keeps a `seen` set (`client/rules.py`); the dedupe is unconditional,
+  because the id is unique under either order and a gate is one more thing to update when the
+  next mutable order appears.
+- **Keep the FIRST copy, and know what that costs.** A streaming printer has already written
+  copy one when copy two arrives, so first-wins is the only option — and copy one carries the
+  PRE-transition values. A ruleset whose hunt stopped mid-walk prints with its old
+  `Live Hunt Id`. Under a mutable order neither the position nor the row's own fields are
+  authoritative for a row that moved; a fresh run shows the settled state.
+- **Say both in the command's help**, not only here. The user reading `--help` is the one who
+  will act on a stale field.
+
+What cannot be repaired client-side: a row whose key changes so that it moves ABOVE the
+cursor is never served at all, so it is missing from that walk entirely. Document it; a
+re-run lists it.
+
 ### No-results signalling
 
 A search that the server answers `204 No Content` raises `NoResultsException` from the SDK **when the generator is iterated**. The CLI's `ExceptionHandlingGroup` maps that (and the CLI's own aggregate `NoResultsException` from `parallel_executor`) to exit code `1`. Don't swallow it in command code.
@@ -79,7 +105,7 @@ When a CLI feature needs an SDK surface that doesn't exist yet:
 
   **Read the declared version off the archive's own tree, and mind pre-release suffixes.** PEP 440 orders `4.2.0.dev1 < 4.2.0`, so a `develop` head carrying a dev suffix (the SDK's `pyproject.toml` has a `[tool.bumpversion.parts.dev]`) would *not* satisfy a `>=4.2.0` floor even though it looks like 4.2.0 — and the archive build would be silently replaced from PyPI. Check the version string in the SDK branch's `pyproject.toml` / `__init__.py`, not the last release tag. When the floor was last verified this way both were read from `origin/develop` as `4.2.0`, no suffix; the pin has since moved on (§Current floor is the one authoritative statement of its value), and every bump should be re-checked the same way.
 
-### Current floor — `polyswarm_api>=4.4.0`
+### Current floor — `polyswarm_api>=4.5.0`
 
 The floor is whatever `pyproject.toml` pins; this header follows it. It lives in ONE authoritative place for a reason — a copy here drifted behind the pin once already. The 4.2.0 rationale below still holds transitively; on 4.1.0 both behaviours fail *silently*, which is why the floor is a hard requirement rather than a preference:
 
@@ -96,8 +122,14 @@ predates this and is documented where it lives: the known-good rendering attribu
 that is not supported rather than against a version the floor permits.) The hunt-page surfaces —
 `ruleset_favorite` and the `YaraRulesetFavorite` resource, the `ruleset_list` filters,
 `live_feed(livescan_id=, max_results=)`, and the tracking/provenance fields the
-formatters render — are what moved the floor to 4.4.0. Code and tests use them
-directly.
+formatters render — are what moved the floor to 4.4.0, together with
+`matched_strings` / `matched_strings_dropped` on the four hunt-result classes
+(the yara evidence behind a hit; see [`03-formatters.md`](./03-formatters.md)
+§Matched strings on hunt results). `rules list --sort active-first` forwards
+`ruleset_list(sort='active_first')`, a keyword 4.5.0 adds, and that is what moved the
+floor to 4.5.0 (the `tests/formatter_hunt_fields_test.py` autospec assertion is the
+signature check: against a 4.4.0 SDK it fails at the mock, not at the server). Code and
+tests use them directly.
 
 **Raising the floor is the whole procedure** when this repo needs something new from
 the SDK:

src/polyswarm/__init__.py

@@ -1 +1 @@
-__version__ = '4.4.0'
+__version__ = '4.5.0'