feat(rules): list --sort active-first

pyproject.toml

@@ -22,7 +22,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "polyswarm_api>=4.4.0,<5.0.0",
+    "polyswarm_api>=4.5.0,<5.0.0",
     "click>=7.1",
     "colorama>=0.4.6",
     "click-log>=0.4.0",

specs/02-commands.md

@@ -30,7 +30,7 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
 | `tag` (`tags.py`) | Tag CRUD | `tag_{create,delete,get,list}` |
 | `link` (`links.py`) | Tag/family links on artifacts | `tag_link_multiple`, `tag_link_get`, `tag_link_list` |
 | `family` (`families.py`) | Malware-family CRUD | `family_{create,update,delete,get,list}` |
-| `rules` (`rules.py`) | YARA ruleset CRUD plus `favorite <id> [--unfavorite]` (the star toggle: renders the new state + the server-owned "N of M used" budget, and converts the machine-readable `FAVORITE_LIMIT` refusal into a clean actionable message at exit 2, never 1 — 1 is reserved for no-results/not-found; 2 is the broad bucket `ExceptionHandlingGroup` maps the PolyswarmException hierarchies to. **2 does not identify a server refusal:** click exits 2 for a `UsageError` too, so a scripted caller cannot tell “the favorite budget is full” from “you passed a bad flag” without reading the message). `list` takes the server-side filters `--name` / `--status active` / `--favorites-only` / `--has-new-results` (conjunctive; the list is keyset-paginated, so filtering locally would mean walking every page). `rules favorite` and the `rules list` filters need SDK 4.4.0, which the pin requires (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor), so they are called directly. The formatters read the hunt-page fields directly: the pin guarantees the SDK parses them, so `None` means the *server* had no answer | `ruleset_{create,delete,update,get,list,favorite}` |
+| `rules` (`rules.py`) | YARA ruleset CRUD (`list` takes the server-side filters plus `--sort active-first`, the hunt page's order: rulesets carrying a live hunt link first, deduped by id because the key is mutable — neither the position nor a moved row's own fields are evidence a hunt is running, see [05-sdk-contract.md](./05-sdk-contract.md) §A mutable order makes the walk the caller's problem. Ordering is the server's, never a local re-sort of one keyset page) plus `favorite <id> [--unfavorite]` (the star toggle: renders the new state + the server-owned "N of M used" budget, and converts the machine-readable `FAVORITE_LIMIT` refusal into a clean actionable message at exit 2, never 1 — 1 is reserved for no-results/not-found; 2 is the broad bucket `ExceptionHandlingGroup` maps the PolyswarmException hierarchies to. **2 does not identify a server refusal:** click exits 2 for a `UsageError` too, so a scripted caller cannot tell “the favorite budget is full” from “you passed a bad flag” without reading the message). `list` takes the server-side filters `--name` / `--status active` / `--favorites-only` / `--exclude-favorites` (its inverse, refused together with it — for a client that lists the favorites separately) / `--has-new-results` (conjunctive; the list is keyset-paginated, so filtering locally would mean walking every page). `rules favorite` and the `rules list` filters need SDK 4.4.0 and `rules list --sort` needs 4.5.0, which the pin requires (see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor), so they are called directly. The formatters read the hunt-page fields directly: the pin guarantees the SDK parses them, so `None` means the *server* had no answer | `ruleset_{create,delete,update,get,list,favorite}` |
 | `metadata` (`metadata.py`) | Rerun metadata; scan lookup; IP/URL analysis | `rerun_metadata`, `scan_lookup`, `submit_url` |
 | `activity` (`event.py`) | List account activity/events | `event_list` |
 | `account` (`account.py`) | Account whois / features | `account_whois`, `account_features` |

specs/05-sdk-contract.md

@@ -39,6 +39,32 @@ for item in api.iocs_by_hash(type, value):
 
 Because the generators are **lazy**, calling one does no I/O and raises nothing until iterated. Code that runs SDK calls through a thread pool must consume the generator **inside the worker** so per-item exception handling fires where it's expected — this is why `utils.parallel_executor_iterable_results` materialises each generator inside the submitted callable (see `01-architecture.md`).
 
+### A mutable order makes the walk the caller's problem
+
+Most list endpoints are keyset-paginated on an immutable key, so a walk sees every row once.
+`ruleset_list(sort='active_first')` is the exception in the current surface: its key is the
+live-hunt link, which the hunt itself flips, and the SDK's own docstring puts the consequence
+on the caller — *"This generator streams pages and does not dedupe — dedupe by `id` if you
+consume more than one page"*.
+
+A command that walks every page of such an endpoint must therefore:
+
+- **Dedupe by `id`.** A row whose key changes mid-walk drops below the cursor and is served
+  again. `rules list` keeps a `seen` set (`client/rules.py`); the dedupe is unconditional,
+  because the id is unique under either order and a gate is one more thing to update when the
+  next mutable order appears.
+- **Keep the FIRST copy, and know what that costs.** A streaming printer has already written
+  copy one when copy two arrives, so first-wins is the only option — and copy one carries the
+  PRE-transition values. A ruleset whose hunt stopped mid-walk prints with its old
+  `Live Hunt Id`. Under a mutable order neither the position nor the row's own fields are
+  authoritative for a row that moved; a fresh run shows the settled state.
+- **Say both in the command's help**, not only here. The user reading `--help` is the one who
+  will act on a stale field.
+
+What cannot be repaired client-side: a row whose key changes so that it moves ABOVE the
+cursor is never served at all, so it is missing from that walk entirely. Document it; a
+re-run lists it.
+
 ### No-results signalling
 
 A search that the server answers `204 No Content` raises `NoResultsException` from the SDK **when the generator is iterated**. The CLI's `ExceptionHandlingGroup` maps that (and the CLI's own aggregate `NoResultsException` from `parallel_executor`) to exit code `1`. Don't swallow it in command code.
@@ -79,7 +105,7 @@ When a CLI feature needs an SDK surface that doesn't exist yet:
 
   **Read the declared version off the archive's own tree, and mind pre-release suffixes.** PEP 440 orders `4.2.0.dev1 < 4.2.0`, so a `develop` head carrying a dev suffix (the SDK's `pyproject.toml` has a `[tool.bumpversion.parts.dev]`) would *not* satisfy a `>=4.2.0` floor even though it looks like 4.2.0 — and the archive build would be silently replaced from PyPI. Check the version string in the SDK branch's `pyproject.toml` / `__init__.py`, not the last release tag. When the floor was last verified this way both were read from `origin/develop` as `4.2.0`, no suffix; the pin has since moved on (§Current floor is the one authoritative statement of its value), and every bump should be re-checked the same way.
 
-### Current floor — `polyswarm_api>=4.4.0`
+### Current floor — `polyswarm_api>=4.5.0`
 
 The floor is whatever `pyproject.toml` pins; this header follows it. It lives in ONE authoritative place for a reason — a copy here drifted behind the pin once already. The 4.2.0 rationale below still holds transitively; on 4.1.0 both behaviours fail *silently*, which is why the floor is a hard requirement rather than a preference:
 
@@ -99,7 +125,11 @@ that is not supported rather than against a version the floor permits.) The hunt
 formatters render — are what moved the floor to 4.4.0, together with
 `matched_strings` / `matched_strings_dropped` on the four hunt-result classes
 (the yara evidence behind a hit; see [`03-formatters.md`](./03-formatters.md)
-§Matched strings on hunt results). Code and tests use them directly.
+§Matched strings on hunt results). `rules list --sort active-first` forwards
+`ruleset_list(sort='active_first')`, a keyword 4.5.0 adds, and that is what moved the
+floor to 4.5.0 (the `tests/formatter_hunt_fields_test.py` autospec assertion is the
+signature check: against a 4.4.0 SDK it fails at the mock, not at the server). Code and
+tests use them directly.
 
 **Raising the floor is the whole procedure** when this repo needs something new from
 the SDK:

src/polyswarm/client/rules.py

@@ -36,23 +36,66 @@ def delete(ctx, rule_id):
 @click.option('-s', '--status', type=click.Choice(['active']),
               help='Only rulesets whose live hunt is currently running.')
 @click.option('--favorites-only', is_flag=True, help='Only favorited (starred) rulesets.')
+@click.option('--exclude-favorites', is_flag=True,
+              help='Only rulesets that are NOT favorited. The inverse of '
+                   '--favorites-only, and refused together with it. For a '
+                   'client that lists the favorites separately.')
 @click.option('--has-new-results', is_flag=True,
               help='Only rulesets whose stored new-results counter is positive.')
+@click.option('--sort', type=click.Choice(['active-first']),
+              help='Order: rulesets carrying a live hunt link first, newest first '
+                   'within each block. Default is newest first. The rank is the '
+                   'stored link, which is WIDER than what Live Hunt Id renders '
+                   'from: a legacy row whose hunt was stopped without clearing '
+                   'the link leads the list while rendering no Live Hunt Id at '
+                   'all, indistinguishable from an idle one — so the position '
+                   'is not evidence that a hunt is running. Neither is the '
+                   'field for a row that MOVED during the walk: see the note '
+                   'on stale copies below.')
 @click.pass_context
-def list_rules(ctx, name, status, favorites_only, has_new_results):
+def list_rules(ctx, name, status, favorites_only, exclude_favorites, has_new_results, sort):
     """List rulesets, optionally filtered. All filters are conjunctive.
 
-    Filtering is applied SERVER-side: the list is keyset-paginated, so a
-    client filtering locally would have to walk every page to find matches.
+    Filtering and ordering are applied SERVER-side: the list is
+    keyset-paginated, so a client filtering or sorting locally would have to
+    walk every page to get it right.
+
+    This command walks EVERY page, which makes it the consumer the SDK puts the
+    dedupe obligation on: under --sort active-first the ordering key is mutable
+    (it is the live-hunt link the sort ranks on), so a ruleset whose hunt stops
+    between two page fetches drops below the cursor and the server serves it a
+    second time. Rows are therefore emitted at most once per run, keyed on id.
+
+    The copy that survives is the FIRST one, which is the only choice a
+    streaming printer has — it wrote that copy before the second arrived — and
+    it carries the values from BEFORE the transition. So the one row the dedupe
+    acts on prints its old Live Hunt Id, for a hunt that has since stopped.
+    Under this order a moved row is authoritative in neither its position nor
+    its fields; a fresh run shows the settled state.
+
+    The symmetric case cannot be repaired from here and is not hidden: a hunt
+    STARTED mid-walk moves its ruleset above the cursor, so that row never
+    reaches this client at all. A re-run lists it.
     """
     api = ctx.obj['api']
     output = ctx.obj['output']
     # A False flag is not a filter: send only what the caller actually asked for.
+    # The CLI spells the sort with a hyphen; the server token is 'active_first'.
     kwargs = {k: v for k, v in (('name', name), ('status', status),
                                 ('favorites_only', favorites_only or None),
-                                ('has_new_results', has_new_results or None))
+                                ('exclude_favorites', exclude_favorites or None),
+                                ('has_new_results', has_new_results or None),
+                                ('sort', sort.replace('-', '_') if sort else None))
               if v is not None}
+    seen = set()
     for ruleset in api.ruleset_list(**kwargs):
+        # Unconditional rather than gated on `sort`: the id is unique either
+        # way, one set of ids costs nothing next to the rows already rendered,
+        # and a gate would be a second place to update when another mutable
+        # order appears. Under the default order this never drops anything.
+        if ruleset.id in seen:
+            continue
+        seen.add(ruleset.id)
         output.ruleset(ruleset)