diff --git a/specs/02-resources.md b/specs/02-resources.md index 18c28a20..1eb25abe 100644 --- a/specs/02-resources.md +++ b/specs/02-resources.md @@ -377,7 +377,7 @@ A file-system or in-memory artifact prepared for upload. Constructed via: - `LocalArtifact.from_path(api, path, artifact_type, artifact_name=None)` — opens a file. - `LocalArtifact.from_handle(api, handle, artifact_name, artifact_type)` — wraps an open file-like. -- `LocalArtifact.from_content(api, content, artifact_name, artifact_type)` — wraps an in-memory string (URL submissions). +- `LocalArtifact.from_content(api, content, artifact_name, artifact_type)` — wraps an in-memory string (URL submissions, except QR-code submissions, which name an image file and use `from_path`). Holds `handle`, `artifact_name`, `artifact_type`, `sha256`, `sha1`, `md5`. Also has classmethod builders `download`, `download_id`, `download_archive`, `download_sandbox_artifact` that return `PolyswarmRequest` descriptors (with `result_parser=LocalArtifact` — the non-JSON parsing path). diff --git a/specs/03-endpoints.md b/specs/03-endpoints.md index 12e4f857..3e98ef45 100644 --- a/specs/03-endpoints.md +++ b/specs/03-endpoints.md @@ -264,7 +264,7 @@ async def submit(self, artifact, ...): Generated (`api.py`) is the same with `await`/`async` lowered. (`sandbox_file` / `sandbox_url` follow the same create → `upload_file` → finalize shape, finalizing via `_finalize_sandbox_task`.) -A URL passed as a string to `submit` / `sandbox_file` (with `artifact_type=URL`) or as `sandbox_url(url)` is refanged before `LocalArtifact.from_content` when `refang_iocs` is on, so the uploaded content and the default artifact name both carry the live URL (§"IoC refanging" in [`05-downstream-contract.md`](./05-downstream-contract.md)). A QR-code submission (`preprocessing={'type': 'qrcode'}`) is the exception on both `submit` and `sandbox_file`: its argument names an image file, not a URL, and is passed on unchanged. +A URL passed as a string to `submit` / `sandbox_file` (with `artifact_type=URL`) or as `sandbox_url(url)` is refanged before `LocalArtifact.from_content` when `refang_iocs` is on, so the uploaded content and the default artifact name both carry the live URL (§"IoC refanging" in [`05-downstream-contract.md`](./05-downstream-contract.md)). A QR-code submission (`preprocessing={'type': 'qrcode'}`) is the exception on both `submit` and `sandbox_file`: its argument names an image file, not a URL, so it is read from disk with `LocalArtifact.from_path` (the uploaded body is the image's bytes; the default artifact name is the file's basename) and is never refanged. `upload_file` is a method on the session class (`AsyncPolyswarmSession.upload_file` / `PolyswarmSession.upload_file`). Both strip the session-level `Authorization` header so the PolySwarm API key doesn't leak to the pre-signed S3 origin. Downstream consumers customize behaviour by subclassing the session — see [`05-downstream-contract.md`](./05-downstream-contract.md) §"Customizing transport behaviour". diff --git a/specs/04-testing.md b/specs/04-testing.md index 19d7c6b2..bcf7333b 100644 --- a/specs/04-testing.md +++ b/specs/04-testing.md @@ -32,6 +32,7 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all - `test/hunt_tracking_builder_test.py` — pure-unit request-shape and parse tests for the hunt-page tracking builders/resources. - `test/refang_test.py` — pure-unit tests for `polyswarm_api.refang` (driven by the shared case table `test/fixtures/refang_cases.json`, kept byte-identical with the other clients that implement the same contract) and request-shape tests for every client method that refangs its IoC inputs, captured at the `_paginate` / `_single` boundary for both transports. Pure-unit because this is client-side input normalization: the server contract is unchanged, so there is no new endpoint behaviour for a cassette to pin. - `test/ruleset_favorite_respx_test.py` — dual-transport (`ClientTestCase`) respx suite for the favorite toggle: the `FAVORITE_LIMIT` refusal envelope and the query/body split. +- `test/sandbox_file_qrcode_respx_test.py` — dual-transport (`ClientTestCase`) respx suite pinning that a QR-code `sandbox_file` submission uploads the image's bytes (the S3 PUT body) under its unrefanged basename. Respx rather than e2e because the stack's sandbox providers cannot reasonably process a QR image. ## Three test layers diff --git a/specs/05-downstream-contract.md b/specs/05-downstream-contract.md index 271c6db6..7b3e887c 100644 --- a/specs/05-downstream-contract.md +++ b/specs/05-downstream-contract.md @@ -226,7 +226,7 @@ Threat-intel reports print indicators defanged (`hxxps[:]//evil[.]com`, `127[.]0 | `is_network_ioc(candidate)` | URL (optional http(s)/ftp(s) scheme, userinfo, port, path), domain, IPv4, or bracketed IPv6 host. | | `refang_ioc(value, accept=None)` | The gated form: returns the refanged value only when something was defanged, the result has no whitespace or `"`, the rewrite does not keep the input's scheme and host intact (if it does, only a path would change, so `example.com/a[.]b` and `https://example.com/a[.]b` both survive), the result is a network IoC, and the optional `accept(candidate)` agrees. Otherwise returns `value` unchanged (untrimmed). Non-strings pass through. | -Invariants: the rules, their order, the gate and the case table (`test/fixtures/refang_cases.json`) are a contract other PolySwarm clients implement too — the table is kept byte-identical across them, so changing a rule means changing it everywhere. Out of scope: email `[at]`, bare-word ` dot `, `http__host` / `http:\\host`, bare-bracket stripping (IPv6 syntax), non-ASCII hosts. +Invariants: the rules, their order, the gate and the case table (`test/fixtures/refang_cases.json`) are a contract other PolySwarm clients implement too — the table is kept byte-identical across them (by hand: each client's suite pins only the digest of its own copy), so changing a rule means changing it everywhere. Out of scope: email `[at]`, bare-word ` dot `, `http__host` / `http:\\host`, bare-bracket stripping (IPv6 syntax), non-ASCII hosts. **Where the client applies it** (when constructed with `refang_iocs=True`; the default is `False`): `search_url(url)`; `search_by_metadata(ips=, urls=, domains=)` — **never** the free-form `query`; `search_by_ioc(ip=, domain=)`; `check_known_hosts(ips=, domains=)`; the `host` of `add_known_good_host` / `add_known_bad_host` / `update_known_good_host`; and the URL of every URL submission (`submit(..., artifact_type=URL)` and `sandbox_file(..., artifact_type=URL)` from a string, `sandbox_url(url)`) — both the uploaded content and the default artifact name. An explicit `artifact_name` is kept as given. Hashes and ids are never touched, and neither is the argument of a QR-code submission (`preprocessing={'type': 'qrcode'}` on `submit` or `sandbox_file`): it names an image file, not a URL, so a file called `qr[.]png` stays `qr[.]png`. With the default `refang_iocs=False` every input is sent verbatim, exactly as in 4.5.0. The setting is kept as the public attribute `api.refang_iocs`, so a consumer that handles a value outside the endpoint methods (its own validation, or a request it builds with `_single`) can apply `refang.refang_ioc` under the same switch — the CLI does exactly that. diff --git a/src/polyswarm_api/aio/api.py b/src/polyswarm_api/aio/api.py index 7cbc1ab9..058b007d 100644 --- a/src/polyswarm_api/aio/api.py +++ b/src/polyswarm_api/aio/api.py @@ -1516,13 +1516,17 @@ async def sandbox_file( ) elif artifact_type == resources.ArtifactType.URL: # A QR-code submission's argument is an image path, not a URL - # (same rule as ``submit``), so it is never refanged. - if not (preprocessing and preprocessing.get("type") == "qrcode"): + # (same rule as ``submit``): read the image, never refang it. + if preprocessing and preprocessing.get("type") == "qrcode": + artifact = resources.LocalArtifact.from_path( + self, artifact, artifact_type=artifact_type, artifact_name=artifact_name + ) + else: artifact = self._refang(artifact) - artifact = resources.LocalArtifact.from_content( - self, artifact, artifact_name=artifact_name or artifact, - artifact_type=artifact_type, - ) + artifact = resources.LocalArtifact.from_content( + self, artifact, artifact_name=artifact_name or artifact, + artifact_type=artifact_type, + ) json_params = { "artifact_name": artifact.artifact_name, diff --git a/src/polyswarm_api/api.py b/src/polyswarm_api/api.py index 3ac8beda..7aeb7785 100644 --- a/src/polyswarm_api/api.py +++ b/src/polyswarm_api/api.py @@ -1862,15 +1862,22 @@ def sandbox_file( ) elif artifact_type == resources.ArtifactType.URL: # A QR-code submission's argument is an image path, not a URL - # (same rule as ``submit``), so it is never refanged. - if not (preprocessing and preprocessing.get("type") == "qrcode"): + # (same rule as ``submit``): read the image, never refang it. + if preprocessing and preprocessing.get("type") == "qrcode": + artifact = resources.LocalArtifact.from_path( + self, + artifact, + artifact_type=artifact_type, + artifact_name=artifact_name, + ) + else: artifact = self._refang(artifact) - artifact = resources.LocalArtifact.from_content( - self, - artifact, - artifact_name=artifact_name or artifact, - artifact_type=artifact_type, - ) + artifact = resources.LocalArtifact.from_content( + self, + artifact, + artifact_name=artifact_name or artifact, + artifact_type=artifact_type, + ) json_params = { "artifact_name": artifact.artifact_name, diff --git a/src/polyswarm_api/refang.py b/src/polyswarm_api/refang.py index fb77205b..282cf262 100644 --- a/src/polyswarm_api/refang.py +++ b/src/polyswarm_api/refang.py @@ -1,18 +1,6 @@ """Refang defanged indicators of compromise (IoCs). -Threat-intel reports print every indicator defanged — ``hxxps[:]//evil[.]com``, -``127[.]0[.]0[.]1`` — so that nobody clicks it by accident. Pasted as-is into a -search or a URL submission it never matches anything: the server looks URLs up -by an exact hash of the string and stores a submitted URL verbatim, so a -defanged value silently misses (search) or becomes a new, broken URL artifact -(submission). The server deliberately does not guess, so clients refang at their -own edge, before the request is built. - -This module is the SDK's implementation of a contract other PolySwarm clients -implement too: same rules, same order, same gate, same case table -(``test/fixtures/refang_cases.json``, kept byte-identical across clients). - -Portability is part of that contract, because the same pattern can match +Portability is part of the contract, because the same pattern can match different characters in different regex engines. So: no case-insensitive flag (Python's folds Unicode, e.g. U+212A KELVIN SIGN matches ``k``) — letters are spelled as explicit ``[aA]`` classes; no ``\\b``, ``\\d``, ``\\w``, ``\\s`` @@ -21,9 +9,9 @@ and full matches use ``fullmatch`` (Python's ``$`` also matches before a trailing newline). -Out of scope, everywhere: email ``[at]``, a bare-word `` dot ``, ``http__host`` -and ``http:\\\\host`` variants, stripping bare brackets (they are IPv6 literal -syntax), and non-ASCII (IDN) hosts. +What the contract is, why clients refang at their own edge, where the client +applies it and what is out of scope: ``specs/05-downstream-contract.md`` +§"IoC refanging". """ import re diff --git a/test/_client_harness.py b/test/_client_harness.py index 18be58fe..4d30386c 100644 --- a/test/_client_harness.py +++ b/test/_client_harness.py @@ -63,6 +63,15 @@ def last_request_url(self) -> str: """ return str(self._router.calls[-1].request.url) + @property + def requests(self): + """Every request the mock received, in order (``httpx.Request`` objects). + + For multi-request flows (create, upload, finalize) that need to read a + specific request's raw ``content`` rather than only the last JSON body. + """ + return [call.request for call in self._router.calls] + @property def last_request_body(self): """The JSON body of the most recent request (None if it carried none).""" diff --git a/test/refang_test.py b/test/refang_test.py index bd324017..8dd2fcf2 100644 --- a/test/refang_test.py +++ b/test/refang_test.py @@ -29,13 +29,14 @@ CASES_PATH = pathlib.Path(__file__).parent / 'fixtures' / 'refang_cases.json' CASES = json.loads(CASES_PATH.read_text(encoding='utf-8')) -# Drift guard: the web UI pins the same digest over its copy, so editing the -# table here fails this suite until the other copy -- and both pins -- change -# together. +# Drift guard, for this copy only: the pin catches an edit to this file made +# without updating the pin (a formatter rewrite, say). It cannot see the other +# clients' copies (the web UI pins the same digest over its own), so keeping +# them identical is manual: change every copy and every pin together. SHARED_CASES_SHA256 = '4bda4b2f8f0dacdd3fa80632fe2a6044b9d0ff5194402968cac9698605a72da8' -def test_contract_table_is_byte_identical_to_the_pinned_copy(): +def test_contract_table_matches_its_pinned_digest(): assert hashlib.sha256(CASES_PATH.read_bytes()).hexdigest() == SHARED_CASES_SHA256 @@ -357,10 +358,10 @@ async def test_async_opt_out_writes_raw_host(self, method, args): # ── QR-code submissions: the argument is an image PATH, never a URL ──────── # # ``qr[.]png`` refangs to ``qr.png``, which is shaped like a domain, so these -# fail if the QR branch ever runs the refang. ``submit`` reads the image with -# ``LocalArtifact.from_path``; ``sandbox_file`` hands the string to -# ``from_content`` (its URL branch has no path reader), so each is spied where -# the value actually lands. +# fail if the QR branch ever runs the refang. Both ``submit`` and +# ``sandbox_file`` must READ the image with ``LocalArtifact.from_path``: handing +# the path string to ``from_content`` would upload the text of the path, which +# the server rejects as an unrecognised image. QR_PREPROCESSING = {'type': 'qrcode'} @@ -391,17 +392,21 @@ async def test_async_submit_qrcode_path_is_never_refanged(monkeypatch): assert seen == ['qr[.]png'] -def test_sync_sandbox_file_qrcode_value_is_never_refanged(monkeypatch): - seen = _spy(monkeypatch, 'from_content') +def test_sync_sandbox_file_qrcode_reads_the_image_path_unrefanged(monkeypatch): + uploaded_as_text = _spy(monkeypatch, 'from_content') + seen = _spy(monkeypatch, 'from_path') api, _ = _sync_client() _run_sync(api, 'sandbox_file', 'qr[.]png', 'provider', 'vm', artifact_type='URL', preprocessing=QR_PREPROCESSING) assert seen == ['qr[.]png'] + assert uploaded_as_text == [] -async def test_async_sandbox_file_qrcode_value_is_never_refanged(monkeypatch): - seen = _spy(monkeypatch, 'from_content') +async def test_async_sandbox_file_qrcode_reads_the_image_path_unrefanged(monkeypatch): + uploaded_as_text = _spy(monkeypatch, 'from_content') + seen = _spy(monkeypatch, 'from_path') api, _ = _async_client() await _run_async(api, 'sandbox_file', 'qr[.]png', 'provider', 'vm', artifact_type='URL', preprocessing=QR_PREPROCESSING) assert seen == ['qr[.]png'] + assert uploaded_as_text == [] diff --git a/test/sandbox_file_qrcode_respx_test.py b/test/sandbox_file_qrcode_respx_test.py new file mode 100644 index 00000000..d5feec01 --- /dev/null +++ b/test/sandbox_file_qrcode_respx_test.py @@ -0,0 +1,63 @@ +"""A QR-code ``sandbox_file`` submission uploads the IMAGE, respx-mocked on BOTH +clients (``ClientTestCase`` — specs/04-testing.md invariant 5). + +The respx tier because the e2e stack's sandbox providers cannot reasonably +process a QR image. What is pinned is the wire: the S3 PUT body must be the +file's bytes. A client that hands the path string to ``from_content`` uploads +the text of the path instead, and the server then fails the task as an +unrecognised image — that is the bug this guards. Refanging is switched on so +the same test also proves the path (``qr[.]png``, shaped like a domain once +refanged) is never rewritten. +""" +import json +import os +import tempfile + +from polyswarm_api.aio import PolySwarmAsyncAPI +from polyswarm_api.api import PolyswarmAPI +from test._client_harness import ( + API_KEY, + BASE_URL, + COMMUNITY, + ClientTestCase, + _AsyncToSync, +) + +_TASK_URL = f'{BASE_URL}/sandbox/sandboxtask/instance' +_UPLOAD_URL = 'https://s3.example.test/upload?signature=abc' +_IMAGE_BYTES = b'\x89PNG\r\n\x1a\n not really a png, but bytes all the same' +_TASK = { + 'id': '7', 'community': COMMUNITY, 'sandbox': 'provider', 'created': None, + 'expiration': None, 'status': 'PENDING', 'account_number': None, + 'team_account_number': None, 'instance_id': None, 'sha256': None, + 'report': None, 'upload_url': _UPLOAD_URL, 'config': {}, 'artifact': None, +} + + +class SandboxFileQrCodeTestCase(ClientTestCase): + def setUp(self): + super().setUp() + if self._client_kind == 'sync': + self.api = PolyswarmAPI(API_KEY, uri=BASE_URL, community=COMMUNITY, refang_iocs=True) + else: + self.api = _AsyncToSync( + PolySwarmAsyncAPI(API_KEY, uri=BASE_URL, community=COMMUNITY, refang_iocs=True), + ) + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.path = os.path.join(self.tmp.name, 'qr[.]png') + with open(self.path, 'wb') as f: + f.write(_IMAGE_BYTES) + + def test_uploads_the_image_bytes_under_its_unrefanged_basename(self): + self.mock.add('POST', _TASK_URL, json={'status': 'OK', 'result': _TASK}) + self.mock.add('PUT', _UPLOAD_URL, json={}) + self.mock.add('PUT', f'{_TASK_URL}?id=7', json={'status': 'OK', 'result': _TASK}) + + self.api.sandbox_file(self.path, 'provider', 'vm', artifact_type='URL', + preprocessing={'type': 'qrcode'}) + + create, upload, _finalize = self.mock.requests + assert upload.url == _UPLOAD_URL + assert upload.content == _IMAGE_BYTES + assert json.loads(create.content)['artifact_name'] == 'qr[.]png'