Release 4.4.0

AGENTS.md

@@ -40,6 +40,7 @@ feature/*  ─┐
 - **`develop → master` PRs are how `master` advances.** They're opened by a maintainer when a release-worthy batch of work is on `develop`. Most contributors never open one of these.
 - **Direct PRs to `master`** are wrong. If you opened one, close it, branch off `develop` instead, and re-open against `develop`.
 - **PyPI release happens automatically** when `pyproject.toml`'s `version` changes on `master`. Don't bump the version inside a feature PR unless the maintainer specifically asks — version bumps belong to the `develop → master` step.
+- **The standing exception: a downstream floor that must point at this change.** (This is the SDK-side instance of a workspace-level standard — *cross-repo dependencies are a version pin, not a runtime probe* — not a rule this repo grants itself.) It applies on one condition: the sibling's CI resolves this repo **from source** by branch name, so it can test against an unpublished version. When that holds, the sibling raises its `polyswarm_api>=` floor to the version introducing the surface — and that floor cannot name a version this repo has not declared yet, so the bump lands **here, in the feature PR**, not at the release step. A consumer that installs only published artifacts has no such need: it adopts after the release, and this repo bumps at its own release step as usual. Bump with `bump-my-version` and **check the emitted string is a clean `X.Y.0`**: the config can serialize a `.devN+sha` form, and PEP 440 orders `4.4.0.dev0 < 4.4.0`, so a dev suffix silently fails the sibling's floor and sends its CI to PyPI for a version that does not exist yet.
 
 **Why this matters:** `master` is the published surface of the SDK. PyPI consumers see whatever shows up there. Skipping `develop` skips the integration soak that protects against accidentally shipping a half-baked change.
 
@@ -87,7 +88,7 @@ No per-symbol codegen carve-outs: every method — including `engines`, a cached
 
 Mirror the existing patterns (`LLMPromptConfig`, `MetadataFieldProperties`, `YaraRuleset`):
 
-1. `class FooBar(BaseJsonResource): RESOURCE_ENDPOINT = '/…'` in [`resources.py`](./src/polyswarm_api/resources.py). If the resource's identifier isn't `id`, set `RESOURCE_ID_KEYS = ['your_key']` so the base class routes it into the query string for `GET` / `DELETE` / `PUT`. Resources are transport-agnostic — only edit `resources.py`.
+1. `class FooBar(BaseJsonResource): RESOURCE_ENDPOINT = '/…'` in [`resources.py`](./src/polyswarm_api/resources.py). `RESOURCE_ID_KEYS` (default `['id']`) names the keys the base class routes into the **query string** for `GET` / `DELETE` / `PUT` — everything else rides the body. Usually that is the identifier, when it isn't called `id`; but it is a routing list, not an identity declaration, so a resource may name a non-identifier key to keep `id` in the body (`YaraRulesetFavorite` does — see [`specs/02-resources.md`](./specs/02-resources.md)). Resources are transport-agnostic — only edit `resources.py`.
 2. Add convenience methods on **[`PolySwarmAsyncAPI`](./src/polyswarm_api/aio/api.py)** (the canonical async source). For a single resource: `return await self._single(resources.FooBar.<builder>(self, …))`. For paginated: `async for item in self._paginate(...): yield item`.
 3. Run `python scripts/regenerate_sync.py` (or rely on the pre-commit hook) to regenerate the sync mirror at `polyswarm_api/api.py`.
 4. Add tests **e2e-first** (see [`specs/04-testing.md`](./specs/04-testing.md)): a live-e2e **VCR lifecycle test** against the real endpoint (sync body in `client_scan_test.py`, async in `async_client_test.py`; record the cassettes against a fresh e2e stack and commit them) plus **pure-unit builder tests** (assert the resulting `PolyswarmRequest`'s shape — body-vs-query routing, None-omission; no httpx fixtures needed). Reach for the respx `ClientTestCase` harness only when the scenario can't reasonably run on the e2e stack.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "polyswarm_api"
-version = "4.3.0"
+version = "4.4.0"
 description = "Client library to simplify interacting with the PolySwarm consumer API"
 readme = "README.md"
 requires-python = ">=3.10,<4"
@@ -55,7 +55,7 @@ package-dir = { "" = "src" }
 where = ["src"]
 
 [tool.bumpversion]
-current_version = "4.3.0"
+current_version = "4.4.0"
 commit = true
 tag = false
 sign_tags = true

specs/01-architecture.md

@@ -42,7 +42,7 @@ Hand-written. No I/O. Three sub-concerns:
 
 **Resource bases**: `BaseResource`, `BaseJsonResource`. The latter has classmethod builders (`create` / `get` / `head` / `update` / `delete` / `list`) that each return a `PolyswarmRequest` descriptor. Per-domain resources in `resources.py` inherit from `BaseJsonResource` and may add custom classmethods (`ArtifactInstance.search_hash`, `IOC.iocs_by_hash`, etc.) — all returning descriptors.
 
-**Helpers**: `Hashable` + `Hash` + `is_valid_sha1/sha256/md5`, `parse_isoformat`, `_normalise_bool_params`, `RequestParamsEncoder`, `_raise_for_status` (the shared non-2xx → typed-exception mapper, used by both `parse_response` and the session's streaming-download path). Downloads stream straight to their destination from the session — there is no response adapter; see [`99-open-questions.md`](./99-open-questions.md) §"Streaming downloads".
+**Helpers**: `Hashable` + `Hash` + `is_valid_sha1/sha256/md5`, `parse_isoformat`, `_normalise_bool_params`, `RequestParamsEncoder`, `_as_result_bound` (the one definition of a caller's result bound — private, like its sibling helpers: nothing outside this package calls it), `_raise_for_status` (the shared non-2xx → typed-exception mapper, used by both `parse_response` and the session's streaming-download path). Downloads stream straight to their destination from the session — there is no response adapter; see [`99-open-questions.md`](./99-open-questions.md) §"Streaming downloads".
 
 ### Layer 2 — transport (`session.py` / `aio/session.py`)
 

specs/02-resources.md

@@ -37,6 +37,7 @@ BaseResource                          # holds .api, ._content, .parse_result cla
     ├── HistoricalHuntResult / List   # /hunt/historical/results (+ /results/list)
     ├── LiveHuntResult / List         # /hunt/live (+ /hunt/live/list)
     ├── YaraRuleset                   # /hunt/rule
+    ├── YaraRulesetFavorite           # /hunt/rule/favorite
     ├── Tag, MalwareFamily, TagLink   # /tags/tag, /tags/family, /tags/link
     ├── AssertionsJob, VotesJob       # /consumer/assertions-job, /consumer/votes-job
     ├── SandboxTask, SandboxProvider  # /sandbox/sandboxtask, /sandbox/provider
@@ -240,7 +241,7 @@ from .core import BaseJsonResource, Hashable, PolyswarmRequest
 
 class FooBar(BaseJsonResource):
     RESOURCE_ENDPOINT = '/foobar'
-    RESOURCE_ID_KEYS = ['foo_id']   # only needed if the identifier isn't 'id'
+    RESOURCE_ID_KEYS = ['foo_id']   # keys routed to the query string, not the body
 
     # Optional: parametrised path
     # RESOURCE_ENDPOINT = '/foobar/{foo_id}'
@@ -343,7 +344,8 @@ Holds `handle`, `artifact_name`, `artifact_type`, `sha256`, `sha1`, `md5`. Also
 Several resources add domain-specific classmethods on top of the standard CRUD set:
 
 - `IOC` — `iocs_by_hash`, `ioc_search`, `check_known_hosts`, `create_known_good`, `create_known_bad`, `update_known_good`, `delete_known_good`.
-- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants.
+- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants. `YaraRuleset` also parses the hunt-page tracking fields (`favorite`, `favorited_at`, `rule_count` — `None` means the server had no answer, distinct from 0 — `historical_hunt_count`, and the stored `new_results_count` with its staleness marker `new_results_counted_at`: the server refreshes the counter on a schedule and the marker says when, so `None` means "not yet refreshed / no live hunt", never 0), and `HistoricalHunt` the source-rule provenance (`rule_id`, `rule_modified`, and the tri-state `source_rule_changed` — `None` is "unknown", never "unchanged"). All additive `.get()` parses; an older server leaves them `None`.
+- `YaraRulesetFavorite` — `RESOURCE_ENDPOINT = '/hunt/rule/favorite'`, **`RESOURCE_ID_KEYS = ['community']`**: a deliberate deviation from the default `['id']`, splitting the request the way the server reads it — the toggle reads BOTH `id` and `favorite` from the PUT **body** (the default would move `id` into the query string and the server would 400 with "A valid rule id must be provided"), while `community` rides the **query string** to match where the ruleset GET/list calls send it. (The server's community middleware accepts the value from either the query or the body — but never both at once — so the placement here is about consistency and the id-400, not about a silently-ignored body value.) `favorite` serialises as `1`/`0` (the `_params` bool→int coercion), which the server's boolean parser accepts. Response carries the star state plus the team's `favorites_used`/`favorites_limit`. Pinned by `test/hunt_tracking_builder_test.py` and both transports of `test/ruleset_favorite_respx_test.py`.
 - `SandboxTask` — `create_file`, `update_file`, `latest`, `my_tasks` for the various sandbox-submission shapes. **No `upload_file` instance method** in 4.0.
 - `Sample` — `create` with `endpoint_fmt={'sha256': sha256}` for the URL-parametrised path.
 - `Webhook` — `test(api, webhook_id)` for the test-payload endpoint.

specs/03-endpoints.md

@@ -86,6 +86,7 @@ Internal-only CRUD for the `/known-good` binary resource (distinct from the IOC
 | `ruleset_create(name, rules, description=None)` | `YaraRuleset.create` |
 | `ruleset_get(ruleset_id=None)` | `YaraRuleset.get` |
 | `ruleset_update(ruleset_id, name=None, rules=None, description=None)` | `YaraRuleset.update` |
+| `ruleset_favorite(ruleset_id, favorite=True)` | `YaraRulesetFavorite.update` — idempotent star/unstar; the response carries the team's `favorites_used`/`favorites_limit`, and an over-budget star is refused with a machine-readable `FAVORITE_LIMIT` error. The id and favorite ride the PUT **body**; `community` rides the query (see the `RESOURCE_ID_KEYS = ['community']` note in specs/02) |
 | `ruleset_delete(ruleset_id)` | `YaraRuleset.delete` |
 | `tag_link_get(sha256)` | `TagLink.get` |
 | `tag_link_update(sha256, tags=None, families=None, emerging=None, remove=False)` | `TagLink.update` |
@@ -186,10 +187,10 @@ refusal.
 | `iocs_by_hash(hash_type, hash_value, hide_known_good=False, beta=False)` | `IOC.iocs_by_hash` |
 | `search_by_ioc(ip=None, domain=None, ttp=None, imphash=None)` | `IOC.ioc_search` |
 | `check_known_hosts(ips=[], domains=[])` | `IOC.check_known_hosts` |
-| `live_feed(since=None, …)` | `LiveHuntResult.list` |
+| `live_feed(since=None, …, livescan_id=None, max_results=None)` | `LiveHuntResult.list` — `livescan_id` scopes the feed to one live hunt (the hunt-page per-ruleset feed); `since` is in **SECONDS** (the server converts with `timedelta(seconds=since)`; the 3.x/4.x docstring said minutes and was wrong), and absent-or-`0` means no time filter at all — the server applies it on a truthiness test; `max_results` bounds how many results the generator yields — `None`/`0`/negative means no bound; it does not alter the request |
 | `historical_list(since=None)` | `HistoricalHunt.list` |
 | `historical_results(hunt=None, …)` | `HistoricalHuntResultList.get` |
-| `ruleset_list()` | `YaraRuleset.list` |
+| `ruleset_list(name=None, status=None, favorites_only=None, has_new_results=None)` | `YaraRuleset.list` — the hunt-page filters, conjunctive and optional; unset filters are omitted from the query so the no-filter request is byte-compatible with the old contract. `has_new_results` selects on the server's STORED counter (no window parameter — the window belongs to the server's scheduled refresh; rows carry `new_results_count` + `new_results_counted_at`) |
 | `tag_list()` | `Tag.list` |
 | `family_list()` | `MalwareFamily.list` |
 | `assertions_list(engine_id)` | `AssertionsJob.list` |

specs/04-testing.md

@@ -26,7 +26,9 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all 
 - `test/async_client_test.py` — async, VCR-backed integration tests (not yet on the parametrised harness — follow-up work).
 - `test/jmespath_test.py` — unit tests for `BaseJsonResource.jmespath`.
 - `test/vcr/*.vcr` — recorded cassettes.
-- `test/eicar.yara`, `test/malicious` — fixture files for upload tests.
+- `test/malicious` — fixture file for upload tests (`test/eicar.yara` was retired when the rules tests moved to per-test `uid_yara` bodies).
+- `test/hunt_tracking_builder_test.py` — pure-unit request-shape and parse tests for the hunt-page tracking builders/resources.
+- `test/ruleset_favorite_respx_test.py` — dual-transport (`ClientTestCase`) respx suite for the favorite toggle: the `FAVORITE_LIMIT` refusal envelope and the query/body split.
 
 ## Three test layers
 
@@ -55,7 +57,7 @@ A respx body for that same arm was written first and deleted once the live cover
 
 ## The parametrised `ClientTestCase` harness
 
-Implemented in `test/_client_harness.py`, importable by any `respx` module that wants one body over both transports; `metadata_field_properties_test.py` is the canonical user, joined by `exists_probe_mapping_test.py` (the `exists()` `404`→`False` arm, which was async-only until the harness existed — the mapping is transport-independent, so one body covers both). The remaining `respx` bodies are the single-transport cases invariant 5 exempts. The shape:
+Implemented in `test/_client_harness.py`, importable by any `respx` module that wants one body over both transports; `metadata_field_properties_test.py` is the canonical user, joined by `exists_probe_mapping_test.py` (the `exists()` `404`→`False` arm, which was async-only until the harness existed — the mapping is transport-independent, so one body covers both) and `ruleset_favorite_respx_test.py` (the favorite toggle's refusal envelope + query/body split). The remaining `respx` bodies are the single-transport cases invariant 5 exempts. The shape:
 
 ```python
 # test/_client_harness.py
@@ -263,6 +265,10 @@ An earlier plan hedged that `-n 8` might need (a) poll windows scaled by `PYTEST
 
 The runner uses `--dist worksteal` (idle workers steal queued tests from busy ones — ≥ static `load` for tail balance; `loadscope` is avoided because it groups by module and would *concentrate* the heavy live tests on fewer workers). `conftest.py`'s `pytest_collection_modifyitems` then front-loads the long-pole live tests so they start at t=0 and the ~100 unit/respx tests backfill the tail. Both are deterministic (keyed on `nodeid`) so every xdist worker collects the same order.
 
+`poll_equals` / `poll_equals_async` (in `_e2e_helpers.py`) is the read-after-write helper: poll a zero-arg `read` until it returns `want`. Use it for any assertion that reads back what the test just wrote through a replica-backed GET — on the e2e stack the replica *is* the primary so the first read usually wins, but a real-replica stack lags and the assertion flakes. The changed-since-freeze one flakes **silently** (a stale source body reads as "unchanged"), which is why it polls rather than sleeps.
+
+**`want` must never be `None`, and the helper refuses it.** Not-found during the lag window is treated as "not yet" and yields `None`, so `want=None` would compare equal and turn a *vanished* resource into a passing assertion. Poll a boolean instead — `read` returning `value is None`, `want=True`. Note the mirror-image limit: a `False` poll cannot ride out a stale `False`, so it only proves the value settled, not that it ever changed.
+
 A few tests submit/dispatch several artifacts in one body; `run_concurrently` / `run_concurrently_async` (in `_e2e_helpers.py`) fan those out **only on the live run** and stay serial on replay — vcrpy patches the transport via `mock.patch`, which isn't thread-safe, and replay no-ops the sleeps anyway, so serial replay is both deterministic and instant (no cassette re-record needed).
 
 **These are correctness/scheduling/cleanup wins, not the wall-clock lever.** The suite's floor is the per-scan settle (`window_closed`), which every settle-bound test on a worker's serial chain pays in full. That floor is **not** the xdist scheduling or the job-phase cadence (lowering the e2e periodicity 10→3 provably didn't move it — a single-scan settle stayed ~32s). It is set deterministically at bounty creation by **`bounty_duration`** (the assertion-window length, ~25s of the ~32s) **+ `ARBITER_VOTE_DELAY`** (~1s). `bounty_duration` is an **artifact-index `ScanConfig` field** — the `default` config the suite submits with — so the lever lives in the e2e/artifact-index config, not in this repo. The full bounty lifecycle and these knobs are documented authoritatively in artifact-index `specs/02-bounty-scan-lifecycle.md`.