feat: search_by_ioc(with_artifacts=) returns artifact metadata rows

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "polyswarm_api"
-version = "4.6.0"
+version = "4.7.0"
 description = "Client library to simplify interacting with the PolySwarm consumer API"
 readme = "README.md"
 requires-python = ">=3.10,<4"
@@ -55,7 +55,7 @@ package-dir = { "" = "src" }
 where = ["src"]
 
 [tool.bumpversion]
-current_version = "4.6.0"
+current_version = "4.7.0"
 commit = true
 tag = false
 sign_tags = true

specs/03-endpoints.md

@@ -185,7 +185,7 @@ refusal.
 | `search_scans(hash_)` | `ArtifactInstance.list_scans` |
 | `search_by_metadata(query, include=None, exclude=None, ips=None, urls=None, domains=None)` | `Metadata.get` |
 | `iocs_by_hash(hash_type, hash_value, hide_known_good=False, beta=False)` | `IOC.iocs_by_hash` |
-| `search_by_ioc(ip=None, domain=None, ttp=None, imphash=None)` | `IOC.ioc_search` |
+| `search_by_ioc(ip=None, domain=None, ttp=None, imphash=None, with_artifacts=False)` | `IOC.ioc_search` — the reverse IOC search. By default each item is an `IOC` whose `json` is a bare sha256 string, and the request carries no `with_artifacts`, so it is byte-compatible with the old contract. `with_artifacts=True` (4.7.0) sends `with_artifacts=1` (an int: the session renders a bool as `'True'`, which the server's boolean parser refuses) and yields `Metadata` resources instead — metadata-search rows trimmed to an include set the SERVER owns (today `artifact.*`; `scan.first_seen`, `scan.first_scan.created`, `scan.last_seen`, `scan.detections`, `scan.mimetype`; `scan.latest_scan.{polyscore,artifact_instance_id,created}`; `scan.filename`, `scan.url`; `polyunite.malware_family`; `hash.ssdeep`, `hash.tlsh`). So `first_seen`, `last_scanned`, the detection counts, mimetypes, filenames, `ssdeep` and `tlsh` are populated; attributes read from outside the set (the `strings.*` IOC lists, for instance) parse as `None` or empty. The SDK does not restate the set: a server-side change to it shows up here without an SDK release. Against a server that predates the parameter the flag is ignored upstream and the sha256 strings fail to parse as `Metadata` with a `TypeError` — so this SDK version must not be released ahead of that server. The SDK does not validate the terms. With `with_artifacts=True` and none of ip/domain/ttp/imphash the server answers 400, which surfaces as the usual typed exception; without the flag a bare call behaves exactly as before. |
 | `check_known_hosts(ips=[], domains=[])` | `IOC.check_known_hosts` |
 | `live_feed(since=None, …, livescan_id=None, max_results=None)` | `LiveHuntResult.list` — `livescan_id` scopes the feed to one live hunt (the hunt-page per-ruleset feed); `since` is in **SECONDS** (the server converts with `timedelta(seconds=since)`; the 3.x/4.x docstring said minutes and was wrong), and absent-or-`0` means no time filter at all — the server applies it on a truthiness test; `max_results` bounds how many results the generator yields — `None`/`0`/negative means no bound; it does not alter the request |
 | `historical_list(since=None)` | `HistoricalHunt.list` |

specs/04-testing.md

@@ -31,6 +31,7 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all 
 - `test/malicious` — fixture file for upload tests (`test/eicar.yara` was retired when the rules tests moved to per-test `uid_yara` bodies).
 - `test/hunt_tracking_builder_test.py` — pure-unit request-shape and parse tests for the hunt-page tracking builders/resources.
 - `test/refang_test.py` — pure-unit tests for `polyswarm_api.refang` (driven by the shared case table `test/fixtures/refang_cases.json`, kept byte-identical with the other clients that implement the same contract) and request-shape tests for every client method that refangs its IoC inputs, captured at the `_paginate` / `_single` boundary for both transports. Pure-unit because this is client-side input normalization: the server contract is unchanged, so there is no new endpoint behaviour for a cassette to pin.
+- `test/ioc_search_test.py` — `search_by_ioc(with_artifacts=)`: pure-unit request shape (the int flag, the unchanged default and bare call), pass-through on both client methods, and a dual-transport (`ClientTestCase`) respx parse of an artifact row — a stand-in until the e2e stack serves the parameter (see `99-open-questions.md`).
 - `test/ruleset_favorite_respx_test.py` — dual-transport (`ClientTestCase`) respx suite for the favorite toggle: the `FAVORITE_LIMIT` refusal envelope and the query/body split.
 
 ## Three test layers

specs/05-downstream-contract.md

@@ -13,7 +13,7 @@ This spec describes the **4.0 surface**. The 3.x → 4.0 migration is covered in
 3. **Exception class names and the inheritance hierarchy are part of the contract.** Callers catch on specific subclasses (`except NotFoundException:`).
 4. **The session classes `PolyswarmSession` / `AsyncPolyswarmSession` are the customization point.** Subclass them, override the methods you want to change, and pass via `PolyswarmAPI(session=...)` / `PolySwarmAsyncAPI(session=...)`. There are no module-level monkey-patch sites.
 5. **The `[async]` extras group is preserved.** Downstream consumers pin `polyswarm-api[async]`. The extra is an empty list (since `httpx` is a core dependency) but the name must remain so old pin specs parse.
-6. **Version bumps go on the `develop → master` step, not feature PRs.** A PyPI release fires automatically when `pyproject.toml` `version` changes on `master`.
+6. **Version bumps go on the `develop → master` step, not feature PRs.** A PyPI release fires automatically when `pyproject.toml` `version` changes on `master`. The one exception is AGENTS.md's standing exception: when a sibling resolves this repo from source by branch name and must raise its `polyswarm_api>=` floor to the version introducing a surface, the feature PR carries the bump, because a floor cannot name a version this repo has not declared. That bump must emit a clean `X.Y.0`: PEP 440 orders a `.devN` suffix below the release, so a suffixed version fails the sibling's floor and sends its CI to PyPI for a version that does not exist yet.
 
 ## Files
 

specs/99-open-questions.md

@@ -210,3 +210,19 @@ Producing an over-budget match on the e2e stack means a rule whose matches excee
 server's per-hunt byte budget across a single artifact — engineering a fixture for that is
 disproportionate to what it would pin. **Recorded rather than tested, deliberately.** If a
 stack fixture ever produces one cheaply, assert both claims there and delete this entry.
+
+## `search_by_ioc(with_artifacts=True)` is not pinned against a live server
+
+**Status:** gap, blocked on the server leg reaching the e2e stack.
+
+The artifact-row shape is asserted only against a fabricated respx envelope
+(`test/ioc_search_test.py`), cut by hand to the server's current include set (`IOC_ARTIFACT_INCLUDES`
+upstream; listed in `03-endpoints.md`), so it drifts silently if the server changes that set.
+The default sha256 path is still covered live by `test_search_by_ioc` /
+`test_async_search_by_ioc`. That is the "asserts what we *think* the server returns" gap
+invariant 1 exists to close.
+
+**Action:** once the e2e stack serves the parameter, add a `with_artifacts=True` pass to
+the live `test_search_by_ioc` pair, record both cassettes against a fresh stack, and delete
+the respx `IocSearchWithArtifactsTestCase` along with this entry. The builder and
+pass-through tests in that module stay: they pin request shape, which a cassette cannot.

src/polyswarm_api/__init__.py

@@ -1,5 +1,5 @@
 # https://www.python.org/dev/peps/pep-0008/#module-level-dunder-names
-__version__ = '4.6.0'
+__version__ = '4.7.0'
 __release_url__ = 'https://api.github.com/repos/polyswarm/polyswarm-api/releases/latest'
 
 from . import api

src/polyswarm_api/aio/api.py

@@ -410,19 +410,28 @@ async def iocs_by_hash(self, hash_type, hash_value, hide_known_good=False, beta=
         async for item in self._paginate(resources.IOC.iocs_by_hash(self, hash_value, hash_type, hide_known_good=hide_known_good, beta=beta)):
             yield item
 
-    async def search_by_ioc(self, ip=None, domain=None, ttp=None, imphash=None):
+    async def search_by_ioc(self, ip=None, domain=None, ttp=None, imphash=None, with_artifacts=False):
         """
         Search artifacts by IOC (ip, domain, ttp, or imphash)
-        
+
         :param ip: ip address to search by
         :param domain: domain address to search by
         :param ttp: ttp to search by
         :param imphash: ImpHash to search by
-        :return: Generator of ArtifactInstance resources
+        :param with_artifacts: True yields a Metadata resource per matching artifact
+            (a metadata-search row trimmed to a field set the server owns: artifact.*,
+            the scan summary, ssdeep/tlsh and the malware family) instead of its bare
+            sha256. With no ip/domain/ttp/imphash the server refuses it with a 400
+            (a typed exception); without it a bare call behaves as it always has.
+            Needs a server that supports the parameter:
+            an older one ignores it and answers bare sha256 strings, which fail to
+            parse as Metadata (TypeError).
+        :return: Generator of IOC resources whose ``json`` is a sha256 string, or of
+            Metadata resources when ``with_artifacts`` is True
         """
         ip, domain = self._refang(ip), self._refang(domain)
-        logger.info('Searching by ioc %s', dict(ip=ip, domain=domain, ttp=ttp, imphash=imphash))
-        async for item in self._paginate(resources.IOC.ioc_search(self, ip=ip, domain=domain, ttp=ttp, imphash=imphash)):
+        logger.info('Searching by ioc %s', dict(ip=ip, domain=domain, ttp=ttp, imphash=imphash, with_artifacts=with_artifacts))
+        async for item in self._paginate(resources.IOC.ioc_search(self, ip=ip, domain=domain, ttp=ttp, imphash=imphash, with_artifacts=with_artifacts)):
             yield item
 
     async def check_known_hosts(self, ips=[], domains=[]):

src/polyswarm_api/api.py

@@ -459,23 +459,46 @@ def iocs_by_hash(self, hash_type, hash_value, hide_known_good=False, beta=False)
         ):
             yield item
 
-    def search_by_ioc(self, ip=None, domain=None, ttp=None, imphash=None):
+    def search_by_ioc(
+        self, ip=None, domain=None, ttp=None, imphash=None, with_artifacts=False
+    ):
         """
         Search artifacts by IOC (ip, domain, ttp, or imphash)
 
         :param ip: ip address to search by
         :param domain: domain address to search by
         :param ttp: ttp to search by
         :param imphash: ImpHash to search by
-        :return: Generator of ArtifactInstance resources
+        :param with_artifacts: True yields a Metadata resource per matching artifact
+            (a metadata-search row trimmed to a field set the server owns: artifact.*,
+            the scan summary, ssdeep/tlsh and the malware family) instead of its bare
+            sha256. With no ip/domain/ttp/imphash the server refuses it with a 400
+            (a typed exception); without it a bare call behaves as it always has.
+            Needs a server that supports the parameter:
+            an older one ignores it and answers bare sha256 strings, which fail to
+            parse as Metadata (TypeError).
+        :return: Generator of IOC resources whose ``json`` is a sha256 string, or of
+            Metadata resources when ``with_artifacts`` is True
         """
         ip, domain = self._refang(ip), self._refang(domain)
         logger.info(
-            "Searching by ioc %s", dict(ip=ip, domain=domain, ttp=ttp, imphash=imphash)
+            "Searching by ioc %s",
+            dict(
+                ip=ip,
+                domain=domain,
+                ttp=ttp,
+                imphash=imphash,
+                with_artifacts=with_artifacts,
+            ),
         )
         for item in self._paginate(
             resources.IOC.ioc_search(
-                self, ip=ip, domain=domain, ttp=ttp, imphash=imphash
+                self,
+                ip=ip,
+                domain=domain,
+                ttp=ttp,
+                imphash=imphash,
+                with_artifacts=with_artifacts,
             )
         ):
             yield item

src/polyswarm_api/resources.py

@@ -169,7 +169,7 @@ def iocs_by_hash(cls, api, hash_value, hash_type, hide_known_good=False, beta=Fa
         )
 
     @classmethod
-    def ioc_search(cls, api, ip=None, domain=None, ttp=None, imphash=None):
+    def ioc_search(cls, api, ip=None, domain=None, ttp=None, imphash=None, with_artifacts=False):
         params = dict(community=api.community)
         if ip is not None:
             params['ip'] = ip
@@ -179,12 +179,17 @@ def ioc_search(cls, api, ip=None, domain=None, ttp=None, imphash=None):
             params['ttp'] = ttp
         if imphash is not None:
             params['imphash'] = imphash
+        if with_artifacts:
+            # An int, not a bool: the session renders bools as 'True', which the
+            # server's boolean parser refuses (it accepts only 0/1/false/true).
+            params['with_artifacts'] = 1
         return core.PolyswarmRequest(
             api=api,
             method='GET',
             url=f'{api.uri}/ioc/search',
             params=params,
-            result_parser=cls,
+            # Opt-in rows are metadata-search documents; the default is bare sha256s.
+            result_parser=Metadata if with_artifacts else cls,
         )
 
     @classmethod