feat: refang defanged IoC inputs, with a global --no-refang opt-out

AGENTS.md

@@ -89,8 +89,9 @@ Mirror the existing patterns (e.g. `field-property`, `prompt-config`, `ruleset`)
 Details in [`specs/04-testing.md`](./specs/04-testing.md). The shape:
 
 - Tests live in `tests/` and drive command behaviour through `click.testing.CliRunner` — no live PolySwarm stack needed.
-- Two styles for that: **SDK-boundary mocks** (`mock.patch('polyswarm_api.api.PolyswarmAPI.<method>')`, e.g. `tests/field_property_test.py`) and **VCR cassettes** (`tests/cli_test.py`) that replay recorded HTTP for end-to-end CLI runs. Cassettes live in `tests/vcr/` (`.vcr` for the HTTP interactions, `.click` for the expected rendered output).
+- Three styles for that: **SDK-boundary mocks** (`mock.patch('polyswarm_api.api.PolyswarmAPI.<method>')`, e.g. `tests/field_property_test.py`), **SDK-transport mocks** (`PolyswarmSession.execute`, e.g. `tests/refang_test.py`, for behaviour that runs inside an SDK method; see the next bullet) and **VCR cassettes** (`tests/cli_test.py`) that replay recorded HTTP for end-to-end CLI runs. Cassettes live in `tests/vcr/` (`.vcr` for the HTTP interactions, `.click` for the expected rendered output).
 - Pure rendering logic — which line a given field set produces, no command-tree behaviour — is instead unit-tested against the formatter directly (e.g. `tests/known_good_field_test.py`); see the spec's *Style 3* for when that's the right choice.
+- Behaviour that runs *inside* an SDK endpoint method (IoC refanging) is tested at the SDK transport instead — `PolyswarmSession.execute` — because an SDK-method mock would replace the code under test (e.g. `tests/refang_test.py`); see the spec's *Style 4*.
 - VCR is an **efficiency cache, not a requirement** — the suite must pass against a live e2e stack with VCR off. Re-record a cassette by deleting it and re-running the test against a live stack; never hand-edit a cassette or `cp` one from a sibling test.
 
 ## Commit + PR hygiene

pyproject.toml

@@ -22,7 +22,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "polyswarm_api>=4.5.0,<5.0.0",
+    "polyswarm_api>=4.6.0,<5.0.0",
     "click>=7.1",
     "colorama>=0.4.6",
     "click-log>=0.4.0",

specs/01-architecture.md

@@ -16,7 +16,7 @@ The components of the CLI and how a command flows from `argv` to rendered output
 
 `polyswarm_cli` is the top-level `click.Group`, constructed with `cls=ExceptionHandlingGroup`. It:
 
-1. Declares the **global options** — `--api-key` (env `POLYSWARM_API_KEY`), `--api-uri` (env `POLYSWARM_API_URI`), the **endpoint shortcuts** `--prod` / `--stage` / `--local` / `--prod-eu` / `--stage-eu`, `--output-file`, `--output-format`/`--fmt` (`text`|`json`|…), `--color/--no-color`, `--verbose`, `--community` (env `POLYSWARM_COMMUNITY`), `--parallel`, `--verify/--no-verify`, plus `--version` / `--api-version`.
+1. Declares the **global options** — `--api-key` (env `POLYSWARM_API_KEY`), `--api-uri` (env `POLYSWARM_API_URI`), the **endpoint shortcuts** `--prod` / `--stage` / `--local` / `--prod-eu` / `--stage-eu`, `--output-file`, `--output-format`/`--fmt` (`text`|`json`|…), `--color/--no-color`, `--verbose`, `--community` (env `POLYSWARM_COMMUNITY`), `--parallel`, `--verify/--no-verify`, `--refang/--no-refang` (default on; see [`02-commands.md`](./02-commands.md) §Global `--refang/--no-refang`), plus `--version` / `--api-version`.
 
    **Endpoint resolution** (`resolve_api_uri`): the shortcuts are convenience aliases for known public endpoints (`API_URI_SHORTCUTS`); `--prod` is an explicit alias for the production endpoint (identical to the no-flag default, offered for symmetry). Precedence is **explicit command-line flag → `POLYSWARM_API_URI` env var → production default** (`PROD_API_URI` = `https://api.polyswarm.network/v3`). Specifically: a shortcut and an explicit *command-line* `--api-uri` are mutually exclusive (conflict → `click.UsageError`, exit 2), as are two shortcuts; a shortcut **wins over** an ambient `POLYSWARM_API_URI` (the env var is consulted only when no shortcut is given) — so `--prod` forces production even when the env var points elsewhere; and a command-line `--api-uri` wins over the env var (click's own source precedence). The command-line-vs-env distinction uses `ctx.get_parameter_source('api_uri') == ParameterSource.COMMANDLINE`.
 2. **Seeds `ctx.obj`** — constructs a `Polyswarm(...)` client (the SDK wrapper) as `ctx.obj['api']` and the selected formatter as `ctx.obj['output']`.
@@ -82,13 +82,13 @@ The catalogue of groups and the SDK methods each wraps is in [`02-commands.md`](
 
 ## Support — `utils.py`, `exceptions.py`
 
-- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end) and `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), plus `collect_files` and the detection helpers (`is_valid_id`, `is_ip`, `is_domain`, `is_url`).
+- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end) and `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), plus `collect_files` and the detection helpers (`is_valid_id`, `is_ip`, `is_domain`, `is_url`), and `refang_input(api, value)` — not a detection helper but the refang one: it applies the SDK's `polyswarm_api.refang.refang_ioc` only when the client's `refang_iocs` is on, for the few places the CLI handles a value before (or instead of) an SDK endpoint method.
 - **`client/utils.py`** — `parse_hashes` and the click parameter validators (`validate_id`, `validate_hash(es)`, `validate_key`, …). Note the module is `client/utils.py`, not the top-level `utils.py` above — the two are distinct and easily confused.
 - **`exceptions.py`** — the CLI's own hierarchy, **distinct from the SDK's**: `PolyswarmException` → `NoResultsException`, `NotFoundException`, `InternalFailureException`, `PartialResultsException`. `ExceptionHandlingGroup` catches both these and the SDK's `api_exceptions.*`.
 
 ## Lifecycle of a command (end to end)
 
-1. `polyswarm_cli` parses global options, builds `Polyswarm(api_key, uri=…, community=…, parallel=…, verify=…)` and the formatter into `ctx.obj`.
+1. `polyswarm_cli` parses global options, builds `Polyswarm(api_key, uri=…, community=…, parallel=…, verify=…, refang_iocs=…)` and the formatter into `ctx.obj`.
 2. The subcommand reads `api`/`output` from `ctx.obj`, parses its own args, and calls one or more SDK methods (directly or via a wrapper fan-out method).
 3. Results are rendered through the formatter; collections are iterated.
 4. Any exception propagates to `ExceptionHandlingGroup.invoke`, which logs it and raises `Exit(<code>)`.

specs/02-commands.md

@@ -62,6 +62,19 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
 > likeliest origin of the original mistake — check which endpoint you are on
 > before copying a default between them.
 
+## Global `--refang/--no-refang` (IoC refanging)
+
+Threat-intel reports print indicators defanged (`hxxps[:]//evil[.]com`, `127[.]0[.]0[.]1`); pasted verbatim they never match a search, and a submitted one becomes a broken URL artifact. The root group's `--refang/--no-refang` (default on) is passed to the client as `Polyswarm(..., refang_iocs=…)`. The SDK's own default is off (its refanging is opt-in, so the SDK release stays a minor bump); the CLI opts in unless `--no-refang` is given, and the SDK refangs the URL / domain / IP inputs of its own endpoint methods (`polyswarm_api.refang`; rules and gate in the SDK's downstream-contract spec). So `search url`, `search metadata -p/-u/-d` (never the free-form query), `search ioc ip|domain`, `search known`, `known add/update`, and the `-r/--url-file` lines of `scan url` (which reach the wire only through the SDK's `submit`) need no CLI code. Positional `scan url` / `sandbox url` URLs are the exception below.
+
+**Write paths change too, not only searches.** With refanging on (the default), `known add` / `known update` store the live host (`known add domain evil[.]com feed` stores `evil.com`), so rows an earlier CLI wrote verbatim as `evil[.]com` are no longer reachable from `search known -d evil[.]com`, which now sends `evil.com`. And `scan url` / `sandbox url` given a defanged URL submit a different artifact (content, `artifact_name`, sha — and the quota it costs) than the same invocation did before. Pass `--no-refang` to keep the verbatim behaviour, e.g. to reach catalogue rows stored defanged.
+
+Two places handle the value in the CLI and call `utils.refang_input(api, value)` — the SDK's `refang_ioc`, gated on `api.refang_iocs`:
+
+- **Validation before the SDK sees the value.** `scan url` and `sandbox url` check positional URLs with `is_url`; they validate the refanged form, so `hxxps[:]//evil[.]com` is accepted instead of rejected as invalid. With `--no-refang` the defanged URL is still rejected, exactly as before.
+- **CLI-owned requests.** `metadata analyze-ip` goes through `Polyswarm.submit_url`, which builds its request with `_single` rather than an SDK endpoint method, so it refangs explicitly.
+
+Hashes and ids are never touched. Neither is a `--qrcode-file` path on `scan url` / `sandbox url`, and that exemption lives in the SDK, not in CLI code: a `preprocessing={'type': 'qrcode'}` submission skips refanging entirely (pinned here by the two qrcode tests in `tests/refang_test.py`). Tests: `tests/refang_test.py`, mocking at the SDK transport so the SDK's own refang is exercised — see [`04-testing.md`](./04-testing.md) §Style 4.
+
 ## Adding to the catalogue
 
 When you add or materially change a group, update its row (and add per-subcommand detail here if the behaviour is non-obvious). The `AGENTS.md` §"When adding a new command family" checklist covers the wiring + formatter + test steps.

specs/04-testing.md

@@ -2,12 +2,12 @@
 
 ## Scope
 
-How the CLI is tested: the `CliRunner` harness, the two mocking styles (SDK-boundary mocks vs VCR cassettes), the formatter-unit style for pure rendering, the cassette layout and record workflow, and how to run the suite. Files: `tests/`, `tests/vcr/`, `src/conftest.py`, `pyproject.toml` (`[project.optional-dependencies].tests`, `[tool.pytest.ini_options]`).
+How the CLI is tested: the `CliRunner` harness, the mocking styles (SDK-boundary mocks, SDK-transport mocks, VCR cassettes), the formatter-unit style for pure rendering, the cassette layout and record workflow, and how to run the suite. Files: `tests/`, `tests/vcr/`, `src/conftest.py`, `pyproject.toml` (`[project.optional-dependencies].tests`, `[tool.pytest.ini_options]`).
 
 ## Invariants
 
 - **Anything that is command behaviour is driven through `click.testing.CliRunner`** — argument parsing, the SDK call, the wiring, the exit code: exercise the real command tree, never an internal function standing in for it. No live PolySwarm stack is required. The one sanctioned exception is pure rendering logic — see [Style 3](#style-3--formatter-unit-tests).
-- **Mock at the SDK boundary, or replay HTTP with VCR — never both for the same path.** A test either patches `polyswarm_api.api.PolyswarmAPI.<method>` (unit-style) or lets VCR replay recorded HTTP (end-to-end). The CLI's own code is exercised either way.
+- **Mock at exactly one point per path — the SDK method, the SDK transport, or VCR.** A test patches `polyswarm_api.api.PolyswarmAPI.<method>` (unit-style, the default), patches the SDK transport `PolyswarmSession.execute` when the behaviour under test runs *inside* the SDK method ([Style 4](#style-4--sdk-transport-mocks)), or lets VCR replay recorded HTTP (end-to-end). Never two of them for the same path. The CLI's own code is exercised either way.
 
 - **VCR is an efficiency cache, not a load-bearing requirement.** The suite must pass against a live e2e stack with VCR off. Don't hardcode `record_mode='none'`; if a test only works against its recorded cassette, that's a bug in the test. Note this is about a test's *logic*, not its fixtures: a `.click` snapshot pins server-generated ids and timestamps, so re-recording needs a stack in a particular state — see [Re-recording a cassette](#re-recording-a-cassette).
 
@@ -84,6 +84,14 @@ For **rendering logic with no command-tree behaviour** — which labelled line a
 
 Use it **only** for that. Argument parsing, SDK calls, generator consumption, `ctx.obj` wiring and exit codes are command behaviour: a formatter unit test can't observe them, so those need Style 1 or Style 2. A command whose rendering is covered by Style 3 still needs at least one `CliRunner` test proving the command reaches the formatter at all.
 
+## Style 4 — SDK-transport mocks
+
+For behaviour that happens **inside** an SDK endpoint method, a Style 1 mock proves nothing: patching `PolyswarmAPI.search_url` replaces the very code under test. IoC refanging is the case today — the SDK refangs `search_url`'s argument (and the other URL / domain / IP inputs) before it builds the request, so the observable effect is the request that reaches the transport.
+
+Patch `polyswarm_api.session.PolyswarmSession.execute` — the SDK's documented session customization point — and assert on the `PolyswarmRequest` it receives: `request.params` (query string) and `request.input_json` (JSON body), read-only. Example: `tests/refang_test.py`.
+
+Use this style only when Style 1 would bypass the behaviour under test. Moving these tests "up" to endpoint-method mocks would keep them green while dropping their coverage. The dependency it adds (the `execute(request)` seam and those two request fields) is recorded in [`05-sdk-contract.md`](./05-sdk-contract.md); an SDK rename there breaks these tests, which is the intended signal.
+
 ## What to test for a new command
 
 1. The command parses its arguments and calls the expected SDK method with the expected arguments.

specs/05-sdk-contract.md

@@ -21,6 +21,7 @@ How the CLI depends on the `polyswarm-api` SDK: which parts of the SDK's public
 | `from polyswarm_api import exceptions as api_exceptions` | Caught in `ExceptionHandlingGroup` and `utils.parallel_executor` (`NoResultsException`, `NotFoundException`, `FailedInstanceException`, `PolyswarmException`). Also `RequestException`, caught by `rules favorite` (`client/rules.py`) to read the machine-readable `FAVORITE_LIMIT` refusal off `exc.request.errors['code']` — and, when the envelope carries no counters, `exc.request.json['result']` as the server's own message. Note the spelling: the request object exposes the response envelope as `.json` and keeps only a private `._result`, so `exc.request.result` is not a thing — reading it yields `None` silently. The SDK does not raise a typed exception for that refusal by design: `.request.errors` is a plain dict the server's error envelope populates. It is pinned without a recording — the SDK's stubbed-transport suite fixes the wire shape, and both CLI branches are unit-pinned against a real `PolyswarmRequest` (not a hand-built mock, which fabricates whatever attribute it is asked for and so cannot detect a rename). It cannot be cassette-pinned: the server sends the counters on every `FAVORITE_LIMIT`, so the envelope the fallback exists for is one no recording can produce. The fallback is defensive against a server that omits them, and the unit test is what fixes the spelling it reads. |
 | `from polyswarm_api.core import parse_isoformat` | Date rendering in `formatters/text.py`. |
 | `import polyswarm_api` (`__version__`) | `--api-version`. |
+| `from polyswarm_api import refang` | `utils.refang_input` — the SDK's `refang_ioc`, for the values the CLI handles itself (pre-SDK URL validation, the CLI-owned `submit_url` request); gated on the client's public `refang_iocs` attribute, which `--refang/--no-refang` sets through the constructor. See [`02-commands.md`](./02-commands.md) §Global `--refang/--no-refang`. |
 
 All of the above are part of the SDK's documented public surface. If a future change needs something not on that list, that's a signal to add a method/export to the SDK rather than reach into internals.
 
@@ -80,7 +81,9 @@ return self._single(
 )
 ```
 
-`_single` builds the request descriptor, executes it via `self.session`, and returns the parsed resource. Do **not** import `PolyswarmRequest` and call `.execute()`/`.result()` — those are not part of the supported surface.
+`_single` builds the request descriptor, executes it via `self.session`, and returns the parsed resource. Production code must **not** import `PolyswarmRequest` and call `.execute()`/`.result()` itself — those are not part of the supported surface.
+
+**Test-only dependency:** `tests/refang_test.py` (testing Style 4) patches `PolyswarmSession.execute(request)` and reads `request.params` / `request.input_json`. That is the SDK's session customization point, used here only as a test seam, never called from production code. An SDK rename of the seam or those fields breaks those tests, which is the intended signal.
 
 ## Coordinated changes (paired PRs)
 
@@ -105,7 +108,7 @@ When a CLI feature needs an SDK surface that doesn't exist yet:
 
   **Read the declared version off the archive's own tree, and mind pre-release suffixes.** PEP 440 orders `4.2.0.dev1 < 4.2.0`, so a `develop` head carrying a dev suffix (the SDK's `pyproject.toml` has a `[tool.bumpversion.parts.dev]`) would *not* satisfy a `>=4.2.0` floor even though it looks like 4.2.0 — and the archive build would be silently replaced from PyPI. Check the version string in the SDK branch's `pyproject.toml` / `__init__.py`, not the last release tag. When the floor was last verified this way both were read from `origin/develop` as `4.2.0`, no suffix; the pin has since moved on (§Current floor is the one authoritative statement of its value), and every bump should be re-checked the same way.
 
-### Current floor — `polyswarm_api>=4.5.0`
+### Current floor — `polyswarm_api>=4.6.0`
 
 The floor is whatever `pyproject.toml` pins; this header follows it. It lives in ONE authoritative place for a reason — a copy here drifted behind the pin once already. The 4.2.0 rationale below still holds transitively; on 4.1.0 both behaviours fail *silently*, which is why the floor is a hard requirement rather than a preference:
 
@@ -128,8 +131,12 @@ formatters render — are what moved the floor to 4.4.0, together with
 §Matched strings on hunt results). `rules list --sort active-first` forwards
 `ruleset_list(sort='active_first')`, a keyword 4.5.0 adds, and that is what moved the
 floor to 4.5.0 (the `tests/formatter_hunt_fields_test.py` autospec assertion is the
-signature check: against a 4.4.0 SDK it fails at the mock, not at the server). Code and
-tests use them directly.
+signature check: against a 4.4.0 SDK it fails at the mock, not at the server). IoC
+refanging — the `polyswarm_api.refang` module, the `refang_iocs=` constructor keyword and
+the refang inside the SDK's endpoint methods, which `--refang/--no-refang` relies on — is
+4.6.0, and that is what moved the floor to 4.6.0 (on 4.5.0 the constructor rejects
+`refang_iocs=` and every command fails). This is a paired change: the SDK branch of the
+same name declares 4.6.0. Code and tests use them directly.
 
 **Raising the floor is the whole procedure** when this repo needs something new from
 the SDK:

src/polyswarm/client/metadata.py

@@ -49,6 +49,10 @@ def analyze_ip(ctx, url):
     Creates an ArtifactInstance immediately (no S3 upload), triggers only
     the IP analyzer, and consumes no quota.
 
+    \b
+    A defanged URL or IP (127[.]0[.]0[.]1) is submitted in its live form;
+    pass --no-refang to the root command to submit it verbatim.
+
     URL is the URL or IP address to submit for analysis.
     """
     api = ctx.obj['api']