feat: refang defanged IoC inputs, with a global --no-refang opt-out - #274
Conversation
Pasting an indicator straight out of a threat-intel report (hxxps[:]//evil[.]com, 127[.]0[.]0[.]1) never matched a search, and scan url / sandbox url rejected it as an invalid URL. The SDK now refangs the URL/domain/IP inputs of its endpoint methods; the new root --refang/--no-refang flag (default on) is passed through as refang_iocs. Where the CLI handles the value itself - the is_url validation in scan url and sandbox url, and the CLI-owned submit_url request behind metadata analyze-ip - it applies the SDK's refang_ioc via utils.refang_input, so a defanged URL is validated in the form that is actually submitted. Raise the SDK floor to 4.6.0, which introduces the surface.
Capture the built request at PolyswarmSession.execute, the documented session customization point, instead of patching _paginate/_single and rebuilding the request with _to_request. Add the --no-refang rejection case for sandbox url and the known add/update host cases.
|
Reviewed against 1.
2.
Everything else matches the documented conventions: |
|
Addressed in 65b10ce:
The SDK side of this pair also tightened its gate (a rewrite that only changes a URL's path is dropped); the CLI suite passes against it: 219 passed. |
|
Reviewed against Correctness, downstream contract, testing and gitflow all look right:
One thing to fix — That bullet enumerates the module surface: " Nothing else blocking. |
…o the SDK's opt-in refang
|
Addressed in 8b4fb24: |
|
Review — checked against Verified clean:
One thing to fix — The new section frames refanging entirely as a read-path fix: "pasted verbatim they never match a search, and a submitted one becomes a broken URL artifact". But the SDK's own contract spec is explicit that turning it on changes what the write paths persist, and this PR turns it on by default for every CLI user:
Concretely, with no flag given:
The PR body's "Behaviour change (for the release notes)" section has the same gap — it lists the |
|
Addressed in 465064c. |
|
Reviewed against One nit — stale doc line the PR half-updated:
Two things I checked and am satisfied with, noted only so the next reader does not re-derive them:
|
|
Fixed in 8c41887: |
|
Reviewed against Verified:
One non-blocking maintainability note, positional = [refang_input(api, u) for u in url]
urls = list(positional)The |
|
Review Correctness, SDK contract, gitflow and test coverage all check out. One spec-drift item, one nit. 1. Behaviour change is missing from the affected commands' own
...and This PR changes what four unchanged invocations do — Suggest a line in the (The release-notes half is handled — the PR body flags it for the 2. Nit: validation errors now quote a string the user never typed
Checked and clean
🤖 Generated with Claude Code |
…yped URL in validation errors
|
Addressed in 0878582:
221 passed. |
|
Reviewed against Three things to action, none blocking: 1. The read side of the catalogue change has no 2. 3. Four commit subjects are truncated mid-word ( One observation, no action needed: 🤖 Generated with Claude Code |
Also narrow the spec's no-CLI-code list to the --url-file lines of scan url; positional scan/sandbox URLs are refanged in the CLI before validation.
|
Addressed in 60a85b9:
This is the last round of changes on this PR. Anything non-blocking that comes up from here on will be answered with a disposition rather than another push. |
|
Reviewed against Checks that mattered:
One minor note, no action required before merge:
|
|
Thanks. The converse read-path case (a defanged URL artifact stored by an earlier |
SummaryLets an analyst paste an indicator straight out of a threat-intel report. A defanged URL, domain or IP ( Severity: 0 HIGH · 1 MODERATE · 1 LOW — none of them in this repo. Fixes are proposed, not applied; nothing was run. Cross-repo coordination
Merge order: Contracts crossing the set: none carries a finding. Coherence: fixes are independent; no cross-repo adjustment needed, and nothing proposed for the SDK changes a surface this repo consumes. Worth knowing: a third client — the web UI — implements the same refang contract on its own branch, with a byte-identical Findings (round 1)No defects found in this repo's diff. Every refang entry point is accounted for: the SDK refangs the inputs of its own endpoint methods, and the two places this repo handles a value itself — the pre-SDK elsewhere: F1, F2 → polyswarm/polyswarm-api#327 Standards conformityChange-level: no violations introduced by this diff. Project-level, non-clean rows:
Set-level: §14 clean — one externally-facing capability rather than a layer, and no UI change merging ahead of it. Branch-name identity clean: |
mjbradford89
left a comment
mjbradford89
left a comment
There was a problem hiding this comment.
Reviewed. No findings in this repo. Merge after polyswarm/polyswarm-api#327, which the 4.6.0 floor here depends on.
SummaryLets an analyst paste an indicator straight from a threat-intel report ( Severity: 0 HIGH · 2 MODERATE · 4 LOW — none of them in this repo. Prior feedback: 23 checked · 2 open.
Fixes are proposed, not applied; nothing was run. Cross-repo coordinationSet: polyswarm-api#327 merged · SDK (reference) — polyswarm-cli#274 merged · CLI ← you are here — the web UI's PR (private repo) open · UI Merge order:
Coherence: the fixes are independent across repos, and none touches a surface this repo consumes. Findings (round 2)No defects found in this repo's diff. This CLI still refangs elsewhere: F2, F3 → polyswarm-api#327 · F1, F5, F6 → the web UI's PR (private repo) · F4 lands in both Standards conformityThe round 1 audit stands. There are no new change-level violations. |
3 participants
What
The CLI now refangs defanged indicators of compromise. A URL, domain or IP pasted straight out of a threat-intel report (
hxxps[:]//evil[.]com,127[.]0[.]0[.]1,evil[dot]com) is sent in its live form. A new root option,--refang/--no-refang(on by default), controls it.Why
Reports print indicators defanged so that nobody clicks one by accident. Pasted as-is, a defanged value never matched a search.
scan urlandsandbox urlrejected it as an invalid URL.How
The SDK can now refang the URL, domain and IP inputs of its own endpoint methods. On the SDK that is opt-in (
refang_iocsdefaults toFalse, which keeps its release a minor bump), and the CLI opts in:--refang/--no-refangis passed to it asrefang_iocs, on unless--no-refangis given. So these commands need no CLI code:search urlsearch metadata -p/-u/-d(the free-form query is never changed)search ioc ip|domainsearch knownknown add/known updatescan url, including-r/--url-filelinesThe CLI handles the value itself in two places, and applies the SDK's
refang_iocthere throughutils.refang_input, with the same switch:scan urlandsandbox urlnow validate the refanged form, so a defanged URL is judged as it will actually be submitted.metadata analyze-ipgoes throughPolyswarm.submit_url, which uses_singlerather than an SDK endpoint method.Hashes and ids are never touched, and neither is a
--qrcode-filepath: that exemption lives in the SDK (a QR-code submission skips refanging entirely), and two tests here pin it forscan urlandsandbox url, including theurl=Nonethatsandbox url --qrcode-filehands the SDK. With--no-refang, every input is sent verbatim and the old validation applies unchanged.Behaviour change (for the release notes)
scan urlandsandbox urlnow accept a defanged URL such ashxxps[:]//evil[.]comand submithttps://evil.com, where they used to reject it with "URL … is not valid". Searches that used to find nothing for a defanged value now find the live one. The write paths change too:known add/known updatenow store the live host, so known-host rows an earlier CLI wrote verbatim asevil[.]comare no longer matched bysearch known -d evil[.]com(which now sendsevil.com), and a defangedscan url/sandbox urlsubmits a different artifact (content, name, sha, quota) than the same invocation did before. The read side cuts both ways:scan url -r urls.txtnever validated its file lines, so an earlier CLI could store a genuinely defanged URL artifact, andsearch url hxxps[:]//evil[.]comnow refangs and no longer reaches it. Pass--no-refangfor the previous behaviour, including to reach catalogue rows or URL artifacts stored defanged. There is no CHANGELOG in this repo, so thedevelop → masterrelease PR should list this.SDK floor: 4.6.0
This change raises the floor to
polyswarm_api>=4.6.0, the SDK version that introducespolyswarm_api.refangand therefang_iocs=keyword. On 4.5.0 the constructor rejects that keyword. The SDK branch of the same name declares 4.6.0, and CI installs it from that branch. Merge the SDK PR first, and release the SDK before this repo's release.Tests
tests/refang_test.py(25 tests) mocks at the SDK transport,PolyswarmSession.execute(the documented session customization point; new testing Style 4 inspecs/04-testing.md), and reads the built request's params and JSON body. A mock onsearch_urland the other endpoint methods would prove nothing, because the refang runs inside them. The tests cover each command above with the flag on and off, live input left unchanged, the--no-refangrejection inscan urlandsandbox url, the validation error quoting the typed (not the rewritten) URL, and the refang note in the--helpofscan url,sandbox url,known addandknown update.Full suite: 221 passed.
Requires