diff --git a/docs/system-specs/modules/security.md b/docs/system-specs/modules/security.md index a2fdcda3f70..2574d2ed9fe 100644 --- a/docs/system-specs/modules/security.md +++ b/docs/system-specs/modules/security.md @@ -618,7 +618,7 @@ Two mechanisms make "the split changed nothing for a caller" a tested claim rath - `diagnostics.py` — what a refusal says about ITSELF: the refusal-diagnostic record, the character-class census behind it, and the one appender every tier uses. A refusal that names only a matcher cannot be acted on — the agent receiving it cannot tell which tier decided, cannot tell where in its own command the decision landed, and so cannot tell a true positive from a matcher firing on text position. Three properties make it safe to put in front of a model and in an audit record. It carries no payload bytes: the matched region is reported as offsets plus a character-class census, because a refusal is the one message guaranteed to concern content the policy judged sensitive, so echoing the match would make the explanation the leak. Its identifiers are closed: the rule id and the component name are screened against an identifier pattern and replaced by a placeholder otherwise, so the format is structurally incapable of quoting a command rather than merely careful not to. And its cost is bounded: the census reads a fixed maximum of the span while the reported length stays true, because the one tier that refuses WITHOUT scanning refuses precisely because its subject is too large to walk on the event loop. It imports nothing from the package, which is what lets both the keystone and the catalog tier reach it without an import-time cycle; it and `vocabulary.py` and `helpers.py` are peers at the bottom of the order. - `vocabulary.py` — the bottom of the dependency order: the product's own name in the two spellings the matchers need, one that matches the name anywhere in a token and one that matches it only as a whole program name, plus the kill programs that select their target by name. Pure vocabulary — no predicate and no verdict, so nothing here can decide anything. It sits below the reader because two tiers read it and neither owns it: the reader consults it to work out what a computed word expands to, and the argv-structural floor above the reader matches the same name in command position. Holding the spellings here is what keeps that dependency one-way instead of the reader depending on the floor and the floor on the reader, an import-time cycle. It imports nothing from the package. - `helpers.py` — the foundation: the public prompt-injection screen over the shared vocabulary, and the resource-limit policy reader with the `preexec_fn` builder it feeds (see [resource-protection.md](../../architecture/resource-protection.md)). It imports nothing from the package, which is what makes it the bottom of the dependency order and keeps the split acyclic. The resource limits sit here rather than with a matcher because they bound a *spawn*, not a path or a command. -- `shell_normalizer.py` — the shell text reader every tier that judges a command line goes through, and its attribution layer: argv attribution, the glob reading of a program name, the expansion shapes whose value cannot be known without running the line, and the top of the reader — the public path normalizer, the tokenizer and the quote-literal decoder behind it, the nested-payload extractor and the payload walk that descends through every literal wrapper, and the local-assignment resolver that substitutes a variable with a literal assigned on the same line, together with the per-line assignment-resolved views the gate re-scans and the argument-position rule that keeps an argument shaped like an assignment from being read as one. It also holds the word and shape layer that reads a native-shell line into words and each word into a path shape and the change-directory vocabulary. The rest of the reader is four owners below it, which never import it: `shell_normalizer_walk.py`, the lexical layer (statement and word splitting, the quote and escape state machine, line continuations, the structural matches a substitution, a backtick or a `case` arm needs, substitution bodies, and the word views that peel brackets, parameter defaults and empty substitutions off a word); `shell_normalizer_redirects.py`, where program text rides outside argv (redirections, here-strings, heredocs, stdin carriers and the `-c` carrier forms); `shell_normalizer_escapes.py`, printer and ANSI-C escape decoding; and `shell_normalizer_programs.py`, the program vocabulary attribution reads. `vocabulary.py` keeps the product's own name and the by-name kill verbs; a new program-name set the reader attributes goes to `shell_normalizer_programs.py`. A name a test patches stays in the module with every reader of it, which is what keeps the payload walk and its helpers, the data-consumer exemption, the stdin-program reading, the quote-literal decoder and the glob family here. `shell_normalizer.py` re-imports every name a caller imports from it. It decides nothing, and imports only `vocabulary.py`, the layer below, and its four owners — the tiers depend on the reader, never the reverse. The reader deliberately **over**-approximates: one that under-approximates blinds every tier at once, and no OS sandbox sees shell text. Narrowing therefore belongs at the site that decides, never here. The whole-program recognizer and the mint-verb predicate sit here rather than with the argv floor that also reads them: each is a thin test over this module's own basename and operand readers, so a module below the reader cannot hold either without taking those readers down with it, and placing either in the floor would make the reader depend on the floor and the floor on the reader, an import-time cycle. They recognize a word rather than decide anything about it. The name spellings they match are one layer further down, in `vocabulary.py`. +- `shell_normalizer.py` — the shell text reader every tier that judges a command line goes through, and its attribution layer: argv attribution, the glob reading of a program name, the expansion shapes whose value cannot be known without running the line, and the top of the reader — the public path normalizer, the tokenizer and the quote-literal decoder behind it, the nested-payload extractor and the payload walk that descends through every literal wrapper, and the local-assignment resolver that substitutes a variable with a literal assigned on the same line, together with the per-line assignment-resolved views the gate re-scans and the argument-position rule that keeps an argument shaped like an assignment from being read as one. It also holds the word and shape layer that reads a native-shell line into words and each word into a path shape and the change-directory vocabulary. The rest of the reader is five owners below it, which never import it: `shell_normalizer_walk.py`, the lexical layer (statement and word splitting, the quote and escape state machine, line continuations, comments and command position, and the word views that peel brackets, parameter defaults and empty substitutions off a word); `shell_normalizer_span.py`, the span layer above it (where a substitution, a backtick, a `${...}` expansion or a `case` arm ends, and the substitution bodies); `shell_normalizer_redirects.py`, where program text rides outside argv (redirections, here-strings, heredocs, stdin carriers and the `-c` carrier forms); `shell_normalizer_escapes.py`, printer and ANSI-C escape decoding; and `shell_normalizer_programs.py`, the program vocabulary attribution reads. `vocabulary.py` keeps the product's own name and the by-name kill verbs; a new program-name set the reader attributes goes to `shell_normalizer_programs.py`. A name a test patches stays in the module with every reader of it, which is what keeps the payload walk and its helpers, the data-consumer exemption, the stdin-program reading, the quote-literal decoder and the glob family here. `shell_normalizer.py` re-imports every name a caller imports from it. It decides nothing, and imports only `vocabulary.py`, the layer below, and its five owners — the tiers depend on the reader, never the reverse. The reader deliberately **over**-approximates: one that under-approximates blinds every tier at once, and no OS sandbox sees shell text. Narrowing therefore belongs at the site that decides, never here. The whole-program recognizer and the mint-verb predicate sit here rather than with the argv floor that also reads them: each is a thin test over this module's own basename and operand readers, so a module below the reader cannot hold either without taking those readers down with it, and placing either in the floor would make the reader depend on the floor and the floor on the reader, an import-time cycle. They recognize a word rather than decide anything about it. The name spellings they match are one layer further down, in `vocabulary.py`. - `paths.py` — the keystone sensitive-path floor, as a mirror facade (`kiro_crew._facade_mirror`) that keeps the read/write path gates, their matchers (`_path_in_home_dirs`, the keystone publish-artifact check), the scan-size ceiling and diff --git a/src/kiro_crew/security/__init__.py b/src/kiro_crew/security/__init__.py index caa88122e25..da055347a7d 100644 --- a/src/kiro_crew/security/__init__.py +++ b/src/kiro_crew/security/__init__.py @@ -621,7 +621,7 @@ "_ShellChar": "shell_normalizer_walk", "_ShellWalk": "shell_normalizer_walk", "_VAR_USE_RE": "shell_normalizer_walk", - "_backtick_closer": "shell_normalizer_walk", + "_backtick_closer": "shell_normalizer_span", "_continuation_width": "shell_normalizer_walk", "_cut_at_operator": "shell_normalizer_walk", "_debracket": "shell_normalizer_walk", @@ -631,7 +631,7 @@ "_is_computed_value": "shell_normalizer_walk", "_is_shell_variable_reference": "shell_normalizer_walk", "_iter_shell_chars": "shell_normalizer_walk", - "_matching_close_paren": "shell_normalizer_walk", + "_matching_close_paren": "shell_normalizer_span", "_push_option_matches": "shell_normalizer_walk", "_push_token_redirection": "shell_normalizer_walk", "_push_token_shell_read": "shell_normalizer_walk", @@ -643,7 +643,7 @@ "_split_push_command_segments": "shell_normalizer_walk", "_split_shell_words": "shell_normalizer_walk", "_strip_redirect": "shell_normalizer_walk", - "_substitution_bodies": "shell_normalizer_walk", + "_substitution_bodies": "shell_normalizer_span", "_substitution_depth_delta": "shell_normalizer_walk", "_substitution_program": "shell_normalizer_walk", # text_redaction @@ -1274,6 +1274,11 @@ def __delattr__(self, name: str) -> None: _stdin_program_text, _stdin_redirect_carriers, ) + from kiro_crew.security.shell_normalizer_span import ( # noqa: F401 + _backtick_closer, + _matching_close_paren, + _substitution_bodies, + ) from kiro_crew.security.shell_normalizer_walk import ( # noqa: F401 _AMBIGUOUS_EXPANSION_RE, _CMD_SEPARATOR_RE, @@ -1297,7 +1302,6 @@ def __delattr__(self, name: str) -> None: _SHELL_VAR_RE, _SHELL_WRAPPER_CHARS, _VAR_USE_RE, - _backtick_closer, _continuation_width, _cut_at_operator, _debracket, @@ -1307,7 +1311,6 @@ def __delattr__(self, name: str) -> None: _is_computed_value, _is_shell_variable_reference, _iter_shell_chars, - _matching_close_paren, _push_option_matches, _push_token_redirection, _push_token_shell_read, @@ -1321,7 +1324,6 @@ def __delattr__(self, name: str) -> None: _split_push_command_segments, _split_shell_words, _strip_redirect, - _substitution_bodies, _substitution_depth_delta, _substitution_program, ) diff --git a/src/kiro_crew/security/argv_floor.py b/src/kiro_crew/security/argv_floor.py index 3c13d38f278..3178cda5ce1 100644 --- a/src/kiro_crew/security/argv_floor.py +++ b/src/kiro_crew/security/argv_floor.py @@ -172,6 +172,7 @@ ) from kiro_crew.security.argv_floor_self import ( # noqa: F401 _DEV_MODE_CONFIRM_FLAG, + _KILL_VERBS, _RAW_ASSIGNMENT_RE, _SELF_CLOUD_DESTRUCTIVE_VERBS, _SELF_FLOOR_MACHINERY_RE, @@ -180,6 +181,7 @@ _SELF_IMPORT_RE, _SELF_MODULE_SPELLINGS, _bare_kill_raw_bodies, + _frames_name_a_kill, _is_credential_mint, _is_dev_mode_out_of_root_confirm, _is_kill_by_name_program, @@ -282,6 +284,7 @@ _SHELL_WRAPPER_CHARS, _argv_programs, _backtick_closer, + _capped_span_possible, _cut_at_operator, _data_consumer_exempt, _debracket, @@ -289,6 +292,7 @@ _decode_shell_quoted_literals, _dequote_token, _ends_argv, + _fold_line_continuations, _glob_could_expand_to, _is_mint_verb, _is_self_program, @@ -308,6 +312,7 @@ _split_shell_words, _strip_redirect, _substitution_bodies, + _substitution_body_entries, _substitution_depth_delta, _xargs_here_string_rebuild, ) diff --git a/src/kiro_crew/security/argv_floor_self.py b/src/kiro_crew/security/argv_floor_self.py index 5e564241a47..e3cc79ffca4 100644 --- a/src/kiro_crew/security/argv_floor_self.py +++ b/src/kiro_crew/security/argv_floor_self.py @@ -41,10 +41,13 @@ _SHELL_WRAPPER_CHARS, _argv_programs, _backtick_closer, + _capped_span_possible, _data_consumer_exempt, _debracket, _decode_printf_escapes, + _decode_shell_quoted_literals, _ends_argv, + _fold_line_continuations, _glob_could_expand_to, _is_mint_verb, _is_self_program, @@ -55,6 +58,7 @@ _shell_join_continuations, _shell_payload_walk, _substitution_bodies, + _substitution_body_entries, _substitution_depth_delta, ) from .vocabulary import _KILL_BY_NAME_PROGRAMS, _SELF_NAME_RE @@ -255,6 +259,10 @@ def _shell_payload_sources(text_lower: str) -> "list[str]": _SELF_FLOOR_QUOTE_JUNK_RE = re.compile(r"[\"'\\\\]") +#: The programs a kill verb names, for the capped-span fail-closed in +#: :func:`_is_self_kill`: ``kill`` takes PIDs, the rest take names. +_KILL_VERBS = frozenset({"kill"}) | _KILL_BY_NAME_PROGRAMS + def _self_floor_can_fire(text_lower: str) -> bool: """Cheap O(n) necessary condition for the self-protection floor predicates. @@ -586,7 +594,7 @@ def record_body(body: str) -> None: # bash parses the body in a fresh context, so the span is # proven from the slice at NEUTRAL state -- and the walk # resumes with the OUTER state carried across the jump. - rel, proven = _matching_close_paren(source[off + 2 :], 0) + rel, proven = _matching_close_paren(source[off + 2 :]) body = source[off + 2 : off + 1 + rel] if proven else source[off + 2 :] # An EMPTY substitution expands to NOTHING, so the word # CONTINUES across it -- ``kill$()`` runs ``kill`` (the same @@ -649,7 +657,7 @@ def record_body(body: str) -> None: break if step.active: if ch in "<>" and source.startswith("(", off + 1): - rel, proven = _matching_close_paren(source[off + 2 :], 0) + rel, proven = _matching_close_paren(source[off + 2 :]) fresh_word = not word and not open_word if not fresh_word and not _kill_prefix_keeps_anchor(words, word): glued = True @@ -708,6 +716,22 @@ def record_body(body: str) -> None: return bodies +def _frames_name_a_kill(text_lower: str) -> bool: + """True if a token in some payload frame runs ``kill``, ``pkill`` or ``killall``. + + The token resolves as a program does (wrappers, parameter defaults, empty + substitutions), then quote junk comes off (``k''ill``). Not ``killed`` or + ``--kill-after=5``: a whole resolved word, not a substring. + """ + for view in (text_lower, _decode_shell_quoted_literals(text_lower).lower()): + for tokens in _self_token_frames(view): + for token in tokens: + base = _shell_normalizer._program_basename(token) + if _SELF_FLOOR_QUOTE_JUNK_RE.sub("", base) in _KILL_VERBS: + return True + return False + + def _is_self_kill(text_lower: str) -> bool: """True if *text_lower* terminates a Kiro Crew process. @@ -730,6 +754,27 @@ def _is_self_kill(text_lower: str) -> bool: # holds, so a miss skips both descents. if not _self_floor_can_fire(text_lower): return False + # A span past the nesting cap is read in shallow pieces, so the frame holding + # a ``kill`` and the one holding its ``$(pgrep -f kiro${x:-crew})`` can part + # and neither names the target. Refuse a kill once the span reader really + # capped a span in some frame (Opus). The opener count only gates that walk: + # 32 flat ``${A}`` nest nothing and are no reason to refuse (Opus, Security + # Scope). The verb is a frame token read as the frame loop reads a program, so + # ``${k:-kill}`` and ``ki${x:-ll}`` count (Opus), on the raw text and on its + # ANSI-C decoding, so ``$'\x6b'ill`` and ``$'\153'ill`` count too (bolichen97). + # Openers are counted on the text the span reader reads, with line + # continuations folded, so ``$\(`` counts as a ``$(`` (Opus). + if _capped_span_possible(_fold_line_continuations(text_lower)) and _frames_name_a_kill( + text_lower + ): + for source in _shell_payload_sources(text_lower): + folded = _fold_line_continuations(source) + if not _capped_span_possible(folded): + continue + seen: list[bool] = [] + _substitution_body_entries(folded, capped_seen=seen) + if seen: + return True for tokens in _self_token_frames(text_lower): programs = _argv_programs(tokens) # Once per FRAME, not once per trigger token: see ``_is_credential_mint``. diff --git a/src/kiro_crew/security/shell_commands.py b/src/kiro_crew/security/shell_commands.py index c0e218783ab..8f9eabb4558 100644 --- a/src/kiro_crew/security/shell_commands.py +++ b/src/kiro_crew/security/shell_commands.py @@ -185,7 +185,7 @@ def cut(separator: str) -> None: if step.state == 2 and text.startswith("$(", step.offset): # bash reads "$(…)" from an unquoted start, then the double quote resumes body = text[step.offset + 2 :] - close, proven = _shell._matching_close_paren(body, 0) + close, proven = _shell._matching_close_paren(body) out.extend(read_commands(body[: close - 1 if proven else close], track, at + 2)) end = pos + step.offset + 2 + close steps.append(ShellStep(at, _base + end, SUBST, SUBST, 2, False, False)) diff --git a/src/kiro_crew/security/shell_normalizer.py b/src/kiro_crew/security/shell_normalizer.py index af7e497e7f9..68a9ec2d4e8 100644 --- a/src/kiro_crew/security/shell_normalizer.py +++ b/src/kiro_crew/security/shell_normalizer.py @@ -12,18 +12,19 @@ over-approximates on purpose and the tiers above it are where precision belongs. Narrowing a matcher means narrowing the site that DECIDES, never this one. -The reader is four modules below this one, and this one keeps attribution -- which +The reader is five modules below this one, and this one keeps attribution -- which program a token reaches, nested shell payloads, local assignments and function aliases, the data-consumer exemption, the glob reading of a program name and the normalized token view. ``shell_normalizer_walk`` reads characters, quotes, words and -structure; ``shell_normalizer_redirects`` reads the carriers that bring program text +structure; ``shell_normalizer_span`` matches where a substitution's body ends and +returns the bodies; ``shell_normalizer_redirects`` reads the carriers that bring program text outside argv; ``shell_normalizer_escapes`` decodes escapes; ``shell_normalizer_programs`` holds the program vocabulary. Every name a caller imports from this module still resolves here, and a name a test patches lives in the same module as every reader of it. Imports only ``vocabulary``, the layer below, for the product's own name, and its own -four owners, which never import it: the tiers depend on the reader, never the reverse. +five owners, which never import it: the tiers depend on the reader, never the reverse. """ from __future__ import annotations @@ -68,6 +69,13 @@ _stdin_program_text, _xargs_here_string_rebuild, ) +from .shell_normalizer_span import ( # noqa: F401 (also re-exported for importers) + _backtick_closer, + _capped_span_possible, + _matching_close_paren, + _substitution_bodies, + _substitution_body_entries, +) from .shell_normalizer_walk import ( # noqa: F401 (also re-exported for importers) _AMBIGUOUS_EXPANSION_RE, _CMD_SPLIT_RE, @@ -84,7 +92,6 @@ _SHELL_WRAPPER_CHARS, _VAR_USE_RE, _arms_case_context, - _backtick_closer, _cut_at_operator, _debracket, _dequote_token, @@ -93,7 +100,6 @@ _is_computed_value, _is_shell_variable_reference, _iter_shell_chars, - _matching_close_paren, _opens_comment, _prev_word, _push_option_matches, @@ -108,7 +114,6 @@ _split_push_command_segments, _split_shell_words, _strip_redirect, - _substitution_bodies, _substitution_depth_delta, _substitution_program, ) @@ -1437,13 +1442,13 @@ class rather than one spelling of it. finite. """ out: list[tuple[str, list[str]]] = [] - seen: set[str] = set() + seen: dict[str, bool] = {} # The third field is whether this frame may perform the ``eval`` argument join. # A frame PRODUCED by a join may not, which is what bounds the walk: see # ``_nested_shell_payloads``. - pending: list[tuple[str, int, bool]] = [(text_lower, len(text_lower) + 1, True)] + pending: list[tuple[str, int, bool, bool]] = [(text_lower, len(text_lower) + 1, True, True)] while pending: - source, parent_len, allow_join = pending.pop() + source, parent_len, allow_join, extract_subs = pending.pop() tokens = _self_tokens(source) if not tokens: continue @@ -1466,15 +1471,21 @@ class rather than one spelling of it. # continuation that bash keeps literal stays literal in the body too. joined_here: set[str] = set() nested = _nested_shell_payloads(tokens, allow_join=allow_join, joined_out=joined_here) - for payload in list(nested) + _substitution_bodies(_fold_line_continuations(source)): + sub_entries = ( + _substitution_body_entries(_fold_line_continuations(source)) if extract_subs else [] + ) + # A ``suppress`` over-read is scanned whole with its nested openers emitted + # separately; re-extracting only re-walks a heredoc run (O(n^2)), so skip it. + subs = [(b, not sup) for b, sup in sub_entries] + for payload, child_extract in [(p, True) for p in nested] + subs: # Descend through EVERY literal payload, to any depth. Termination is - # structural, not a cap: a payload is carried inside one token of its - # parent, so it is strictly shorter than the parent's source text. + # structural: a payload is carried inside one token of its parent, so + # it is strictly shorter than the parent's source text. payload = _decode_printf_escapes(payload) - if len(payload) >= parent_len or payload in seen: + if len(payload) >= parent_len or seen.get(payload, -1) >= child_extract: continue - seen.add(payload) - pending.append((payload, len(source), payload not in joined_here)) + seen[payload] = child_extract + pending.append((payload, len(source), payload not in joined_here, child_extract)) return out diff --git a/src/kiro_crew/security/shell_normalizer_span.py b/src/kiro_crew/security/shell_normalizer_span.py new file mode 100644 index 00000000000..d427c36bb3f --- /dev/null +++ b/src/kiro_crew/security/shell_normalizer_span.py @@ -0,0 +1,841 @@ +"""Shell text reader, span layer: where a substitution's body ends. + +The part of :mod:`kiro_crew.security.shell_normalizer` that matches the closing +``)``, ``}`` or backtick of a command substitution, process substitution or +``${...}`` expansion, reading nested constructs, quotes, comments and ``case`` +arms the way bash does, and the substitution BODIES the gate scans. A heredoc +is not parsed: a span a heredoc body would start inside is read as ``main`` +reads every span (:func:`_flat_close`), and the extractor scans its whole +remainder. It reads the lexical layer in +:mod:`kiro_crew.security.shell_normalizer_walk` and decides nothing. +``kiro_crew.security.shell_normalizer`` re-exports every name a caller imports +from it, so that import path holds. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from .shell_normalizer_walk import ( + _arms_case_context, + _in_command_position, + _iter_shell_chars, + _opens_comment, + _ShellChar, + _word_at, +) + +if TYPE_CHECKING: + from collections.abc import Iterable + + +def _substitution_bodies(text: str) -> "list[str]": + """The body of each command substitution in *text* (see + :func:`_substitution_body_entries` for the per-body suppress flag).""" + return [body for body, _suppress in _substitution_body_entries(text)] + + +def _substitution_body_entries( + text: str, capped_seen: "list[bool] | None" = None +) -> "list[tuple[str, bool]]": + """The body of each command substitution in *text*, each paired with a + ``suppress`` flag for the payload walk. A span read at the nesting cap + appends to *capped_seen* when it is given. + + ``suppress`` marks an OVER-READ: a reading of an UNPROVEN (heredoc / + unbalanced) span -- its full remainder, its glued tails, a later span's + body -- or a capped span's post-newline tail. This walk continues past such + a span and + emits every later opener as its own entry, so the payload walk need not + re-extract an over-read that is a suffix of its source, and re-extracting + them is what makes a run of sequential heredocs O(n^2) at the 20 KB scan + cap. The flag marks the ENTRY, never a value: a proven body, a capped + segment, a data or unproven-backtick remainder is the only reading of its + span and is always re-extracted, even when its text equals a suffix of the + source, or a nested protected-branch push behind it would never be scanned. + + ``$(...)`` is scanned with paren nesting so a nested substitution closing + first does not truncate the outer body; backticks are taken pairwise. Only + the BODY is returned -- a bare ``kill`` must not be attributed a name that + merely appears in a LATER, unrelated command of the same line. + + The nesting walk is QUOTE-AWARE, through the same + :func:`_matching_close_paren` span the git-publish boundary walk uses. A + private, quote-unaware copy of it truncates the body at a QUOTED ``)``, and + that loses the nested command entirely rather than merely mis-sizing the span: + ``git push origin my-feature > >(X=')' git push origin main)`` extracts the + body ``X='``, so the nested publish of a protected branch is never scanned + and ``is_denied`` returns None for a command bash executes. An UNPROVEN span + yields the whole remainder, which is the fail-closed direction -- scanning + text that is not really in the body can only add findings. + + Quoting was not the whole of it. That span helper counted a ``)`` that shell + COMMAND GRAMMAR also puts there as an ordinary character, and two such + spellings were measured allowing a payload this module still refuses: + a ``#`` comment (``$(: # )`` closes on a later line) and a ``case`` pattern + (``$(case x in x) printf token;; esac)``). Both truncated the body before the + verb, so the value assembled from it was never recognised. The closer for the + BACKTICK form reads the same state machine now too, so the two spellings + cannot disagree. + """ + entries: list[tuple[str, bool]] = [] + # Openers are matched byte by byte, so one may sit inside a single quote, + # where bash reads it as DATA. Such a bogus opener's fresh-context span can + # swallow a REAL opener after it (``echo '$(' "$(echo ')'; bash -c + # '')"``), so a single-quoted opener must be stepped PAST rather than + # trusted -- every later opener still gets its own reading. The quote state + # at each opener is read from the text AROUND the spans only: bash parses a + # span in a fresh context, so a ``'`` inside one (``"$(echo "'")"``) quotes + # nothing after its ``)``. A flat walk through the span would carry that + # ``'`` out and mark a later REAL opener as single-quoted. The walk advances + # from the last span end (``walked``) to each opener, with the state the + # opener had (a span leaves it as it was). Only a ``'`` opens a single quote + # (``$'...'`` included), so text without one skips the walk -- unless it + # holds a ``"`` and a process-sub opener, which bash reads as DATA inside + # double quotes (``"<(echo '"``) while a ``$(`` there still opens. + quoted = "'" in text or ('"' in text and ("<(" in text or ">(" in text)) + # That flat walk does not know a ``#`` comment or a heredoc body, where a + # ``'`` is DATA too: ``# don't`` would leave it inside a single quote at a + # REAL opener after it. Nor does it know that bash closes a backtick at the + # next backtick whatever the quotes, so in ``echo `: '` $() `: '` `` the + # ``'`` quotes nothing and the ``$(`` runs (Opus, bolichen97). With any of the + # three present, a single-quoted reading is not trusted to skip an opener, + # and its unproven remainder is scanned instead. + flat_quotes = ( + quoted + and "<<" not in text + and "`" not in text + and not any(_opens_comment(text, k) for k, char in enumerate(text) if char == "#") + ) + state, ansi, walked = 0, False, 0 + # The glued-tail over-reads of an UNPROVEN heredoc span are emitted ONCE, at + # the first such opener; see that branch below for why. + over_read = True + # The past-the-close scan of a CLOSING capped span runs once per walk; + # see that branch below. + past_close = True + # The flat whole-body frame of a capped span runs once per walk. + whole_capped = True + # So does the post-newline over-read of a capped span. + post_newline = True + i = 0 + while i < len(text): + # PROCESS substitutions run their body as a command just as a command + # substitution does -- ``cat <(kirocrew token)`` executes the inner command and + # feeds its output through a pipe. Same paren-nesting walk. + if text.startswith(("<(", ">(", "$("), i): + if quoted: + for step in _iter_shell_chars(text[walked:i], state, ansi): + state, ansi = step.state, step.ansi + walked = i + # A process sub opens only unquoted; a ``$(`` opens in any state + # but a single quote. + data = state == 1 or (state != 0 and text[i] != "$") + if data and flat_quotes: + # Data to bash: skip the two opener bytes so the REAL opener + # after this one is still reached, and do NOT walk a bogus span. + i += 2 + continue + if data: + # The reading may be wrong, so take the whole remainder as the + # body, as an unproven span does: it holds every later opener. + entries.append((text[i + 2 :], False)) + break + # bash parses the body in a fresh quoting context, so the span is + # proven from the slice at NEUTRAL state, as the argv_floor walk + # does. The matcher skips a NESTED substitution and a ``${...}`` + # expansion as units, so a ``"`` or ``)`` inside one of those stays + # sealed inside it and the enclosing span closes at its OWN matching + # ``)``. A span a heredoc body would start inside is unproven and + # read as ``main`` reads it (:func:`_span_walk`). + rel, proven, capped = _span_walk(text[i + 2 :]) + end = i + 2 + rel + if proven: + # A proven span's nested openers are reached when the payload + # walk recurses into this body, so skip PAST the span instead of + # re-walking from every opener inside it. + entries.append((text[i + 2 : end - 1], False)) + i = walked = end + continue + if capped: + if capped_seen is not None: + capped_seen.append(True) + # A span capped by ``_MAX_SPAN_NESTING``: its deep interior can + # not be proven. Its whole-remainder reading is not appended: it + # would cost an O(n) slice at every opener -- O(n^2) on a RUN of + # capped spans at the 20 KB scan cap -- and the capped segments + # below plus the structural walk cover what it holds. + # + # A capped span that spans lines glues the command after a + # newline to a ``)`` past it, so the post-newline commands are + # scanned from the span's first newline to the remainder's last + # ``)``. The newline search is BOUNDED to the span ``[i + 2, + # end)`` -- it GATES the costlier last-``)`` scan -- so a clean + # capped RUN, whose spans hold no newline, pays one bounded miss + # per opener instead of an O(n) whole-remainder ``rfind``. A later + # capped span's slice lies inside the first one's, so it is + # emitted ONCE per walk; a run of them stays linear (GPT). + newline = ( + text.find("\n", i + 2, end if end > i + 2 else None) if post_newline else -1 + ) + if newline != -1: + post_newline = False + last = text.rfind(")", i + 2) + if last > newline: + entries.append((text[newline + 1 : last], True)) + if text[end - 1 : end] == ")": + # CLOSES: but a data ``)`` consumed at the capped depth can + # make it close LATE -- at the ``)`` of a sibling opener bash + # runs after its own close (``... ; cat <(bash -c '')``). + # That sibling's body is inside our span, so scan the span + # ONCE for its substitution segments, as the never-close + # branch does, bounded at the close so a run stays linear. + # The openers AFTER the close still matter too, so CONTINUE + # the one linear walk from the close. + entries.extend(_capped_tail_entries(text, i + 2, end - 1)) + # The span's body is ALSO one frame, read flat as ``main`` + # reads it and not re-extracted: shallow segments part + # ``t=$(printf token)`` from ``kirocrew $t`` and stop at the + # ``)`` in ``${x:-)}`` (Opus, bolichen97). The guessed close + # can be early, so the frame runs to the text's LAST ``)``. + # It holds every later capped span too, so it is emitted + # once per walk and a run stays linear. + if whole_capped: + whole_capped = False + last = text.rfind(")", end - 1) + entries.append((text[i + 2 : last], True)) + # The close is a guess in the OTHER direction too: a data + # ``)`` read at the cap (``${x:-)}``) closes the span + # EARLY, so the span's own later text -- a ``; bash -c + # ''`` before bash's real close -- sits past the + # guessed close, outside every opener the walk reaches. So + # the text from the close to the end is scanned ONCE per + # walk, every ``)`` in it a segment boundary that closes + # nothing (however many closes the guess was early by), + # and a launcher there is a segment of its own. One pass + # to the end covers every later capped span as well, so a + # run of them stays linear. + if past_close: + past_close = False + entries.extend(_capped_tail_entries(text, end, reopen=True)) + i = walked = max(end, i + 2) + continue + # NEVER closes: a data ``)`` consumed at the capped depth (a + # ``${x:-(}`` default value, say) left the span open past where + # bash closes it, so the later TOP-LEVEL opener bash runs as a + # sibling -- a ``$(bash -c '')`` -- is hidden inside our + # over-long span and no payload source is a publish, so the + # git-publish floor never runs. Scan the remainder ONCE, flat and + # quote-aware, emitting each substitution segment as its own + # shallow entry so that hidden push IS read. Force-extracting the + # remainder, or stepping back into it opener by opener, would + # re-walk the deep interior at every level -- the O(n^2) an opener + # run cannot afford at the 20 KB scan cap. + entries.extend(_capped_tail_entries(text, i + 2)) + # Its body is ALSO one flat frame, as in the closing branch, so + # ``t=$(printf token)`` and ``kirocrew $t`` share a frame (Opus). + if whole_capped: + whole_capped = False + last = text.rfind(")") + entries.append((text[i + 2 : last] if last >= i + 2 else text[i + 2 :], True)) + i = len(text) + continue + # The span is UNPROVEN (a heredoc or a genuine unbalance). The whole + # remainder from the opener is the body over-read, read in the OUTER + # quoting context -- scanning text not really in the body only adds + # findings (fail-closed). It and the readings below are over-reads. + entries.append((text[i + 2 :], True)) + # A span unproven because of a HEREDOC (or a genuine unbalance) has a + # real close with text glued after it: a ``bash -c 'git push'`` there + # can read as ``git push)`` in the remainder. So also scan the span up + # to the LAST ``)``, and to the OUTER close, in the outer quoting + # context with no unit skipping. These glued-tail over-reads span from + # the opener to near the end, so on a RUN of sequential unproven + # heredoc spans (``"$(cat < i + 2 else text[i + 2 :] + entries[-1] = (replaced, True) + # The readings above are SUPPRESSED over-reads whose nested openers + # this walk does not emit (it jumps to ``end``); re-extracting those + # near-full remainders is the O(n^2) a heredoc run cannot afford. But + # a span unproven only by a HEREDOC still has a real close, and a REAL + # opener bash RUNS can sit after the body inside it -- ``echo $(cat + # <'))`` runs the ``<(...)``. The + # suppressed over-reads scan that text but never EMIT the ``<(...)``, + # so its push is no payload source and the git-publish floor never + # runs (measured ALLOWED, push and self-restart twin). Emit the + # span's proven-close body ``text[i+2:end-1]`` NON-suppressed so the + # payload walk reaches those nested openers. Bounded to this one span + # (the walk still jumps to ``end``), so a run stays linear. + if text[end - 1 : end] == ")" and end - 1 > i + 2: + entries.append((text[i + 2 : end - 1], False)) + elif end >= len(text) and any(op in text[i + 2 :] for op in ("$(", "<(", ">(", "`")): + # It never closes, so there is no close to bound a body by and + # every reading above is suppressed. When an opener sits in that + # remainder, scan it ONCE, flat, as the capped never-close branch + # does, so each opener is its own entry; the walk ends here, so + # a run stays linear. A heredoc in that remainder can hold a body + # line with an unbalanced quote (``it's``), which the quote-aware + # pass reads as live and so swallows a later ``<(...)``; the + # quote-blind pass reads past it, as ``main``'s extractor does. + entries.extend(_capped_tail_entries(text, i + 2)) + entries.extend(_capped_tail_entries(text, i + 2, quote_blind=True)) + i = len(text) + continue + i = walked = max(end, i + 2) + continue + if text[i] == "`": + if quoted: + for step in _iter_shell_chars(text[walked:i], state, ansi): + state, ansi = step.state, step.ansi + walked = i + # A backtick opens command substitution in any quoting state but a + # single quote, where bash reads it as a LITERAL (``echo '`'``). The + # ``state`` was ignored here, so a single-quoted backtick opened a + # bogus span whose body then carried that ``'`` over the REAL ``$(`` + # after it and hid a push. Step PAST a single-quoted backtick so the + # opener after it is still reached; only when the flat-quote reading + # is not trusted (a ``#`` comment or heredoc present) take the whole + # remainder instead (fail-closed), as the ``$(`` data branch does. + if state == 1: + if flat_quotes: + i += 1 + continue + entries.append((text[i + 1 :], False)) + break + end, proven = _matching_close_backtick(text, i + 1) + # A proven backtick body ends before the close; an UNPROVEN backtick + # takes the whole remainder. Neither is a ``suppress`` entry, so the + # opener hidden inside the remainder is always re-extracted. + if proven: + entries.append((text[i + 1 : end - 1], False)) + else: + entries.append((text[i + 1 :], False)) + i = walked = end + else: + i += 1 + return entries + + +def _capped_tail_entries( + text: str, + start: int, + stop: "int | None" = None, + reopen: bool = False, + quote_blind: bool = False, +) -> "list[tuple[str, bool]]": + """Shallow substitution-body entries for a capped span. + + One linear, quote-aware forward pass over ``text[start:stop]`` (the body of a + ``$(`` / ``<(`` / ``>(`` span whose deep interior consumed a data ``)`` at + :data:`_MAX_SPAN_NESTING`, so the span never proved a close or closed only + LATE, at a sibling's ``)``). bash closes that span at its OWN matching ``)`` + and runs the later opener as a SIBLING, but our over-long span swallows it, + so without this the push in a hidden ``$(bash -c '')`` or + ``<(bash -c '')`` is never a payload source and the git-publish + floor never runs. *stop* bounds the pass at a late close. + + Each maximal run of text that lies INSIDE a substitution -- between an opener + and the next structural ``$(`` / ``<(`` / ``>(`` / ``)`` / backtick -- is + emitted ONCE as its own entry, never a ``suppress`` one. The pass starts + INSIDE the capped span itself (depth 1, a run open at *start*), so the span's + OWN top-level text is a run too: ``"$($($(...:...)) ; bash -c '')"`` + keeps the publish at the outer body's depth, where the root tokenizer sees + only one double-quoted word. A push verb is contiguous inside one such run, + so the hidden publish is scanned, while the deep nest contributes only + short, NON-overlapping segments: both the walk and the emitted text stay + linear in the remainder, never re-walking the interior at every level (the + O(n^2) an opener run cannot afford at the 20 KB scan cap). + With *reopen* the pass starts just past a capped span's GUESSED close, whose + real depth is unknown: a ``)`` that would close the pass's base level + ends the run there and opens the next one instead, so every run up to the + end of the text is emitted however early the guess was. + The segments are read in a fresh (neutral) quoting context, the same + fail-closed direction the unproven-span over-read uses -- scanning text that + is not really in a body can only add findings, never drop one. + With *quote_blind* NO quote is read at all: every opener, backtick and ``)`` + is a boundary. A span that never closes may hold a heredoc whose body line + has an unbalanced quote (``it's``); read as live, that quote swallows the + later ``<(...)`` bash runs. The blind pass cannot be fooled that way, and + its runs are still disjoint, so it stays linear. + """ + entries: list[tuple[str, bool]] = [] + depth = 1 + in_backtick = False + seg_start = start + # One entry per span this scan currently has open (the capped span itself + # first), each the quote state ACTIVE at its opener. A ``)`` closes the top + # span only when it is unquoted-and-unescaped (``step.active``), OR when that + # span was opened inside a DOUBLE quote and this ``)`` is itself in a double + # quote -- bash runs ``"$(...)"`` and closes it at a double-quoted ``)``. A + # ``)`` that is merely double-quoted data (the ``)`` of ``echo ")"``, whose + # opener was NOT dq) or escaped (``\)``, reported ``state 0`` but inactive) + # closes nothing, so the launcher behind it stays in the run and is scanned. + # The capped span is read from a neutral context, so its own opener state is + # 0; its base entry is never popped so the index stays valid. + open_states = [0] + + def flush(upto: int) -> None: + nonlocal seg_start + if seg_start != -1 and upto > seg_start: + entries.append((text[seg_start:upto], False)) + seg_start = -1 + + stop = len(text) if stop is None else stop + if quote_blind: + steps: "Iterable[_ShellChar]" = ( + _ShellChar(k, c, c, True, 0, False, False) for k, c in enumerate(text[start:stop]) + ) + else: + steps = _iter_shell_chars(text[start:stop], 0, False) + for step in steps: + off = start + step.offset + # Inside a single quote every ``)``/``$(``/backtick is LITERAL data, so + # no boundary there -- the run accumulates through it. + if step.state == 1: + continue + if text.startswith(("$(", "<(", ">("), off): + flush(off) + depth += 1 + open_states.append(step.state) + seg_start = off + 2 + continue + # A backtick opens/closes a substitution, and a ``)`` closes one we + # opened, in any state but a single quote -- and inside a DOUBLE quote + # ``_iter_shell_chars`` reports both inactive even though bash runs the + # substitution there, so they are tested BEFORE the active guard, the way + # the paren walk tests the backtick. Only a ``)`` for a span THIS scan + # opened (``depth > 0``) is a boundary; a bare ``)`` at depth 0 is data. + if step.char == "`": + flush(off) + in_backtick = not in_backtick + seg_start = off + 1 if (in_backtick or depth > 0) else -1 + continue + if step.char == ")" and depth > 0: + # Boundary only when unquoted-and-unescaped, or a double-quoted ``)`` + # closing a span whose opener was also double-quoted. A double-quoted + # ``)`` whose opener was NOT dq (``echo ")"``), and an escaped ``\)`` + # (``state 0`` but ``not active``), are DATA: the run accumulates + # through them so a later ``; bash -c ''`` is still scanned. + opener_dq = len(open_states) > 1 and open_states[-1] == 2 + if not (step.active or (step.state == 2 and opener_dq)): + continue + flush(off) + if reopen and depth == 1: + seg_start = off + 1 + continue + depth -= 1 + if len(open_states) > 1: + open_states.pop() + seg_start = off + 1 if (depth > 0 or in_backtick) else -1 + flush(stop) + return entries + + +# A span nested deeper than this is read WITHOUT opening fresh contexts for its +# inner constructs, so the walk never recurses past it; that reading is reported +# unproven. Python's own recursion limit would otherwise end a deep ``${a:-${a:-`` +# chain with a RecursionError before the payload behind it is queued. +_MAX_SPAN_NESTING = 32 + + +def _capped_span_possible(text: str) -> bool: + """Whether a span in *text* can reach :data:`_MAX_SPAN_NESTING`. + + Each nesting level is opened by a ``$(``, ``<(``, ``>(`` or ``${``, so text + with fewer of them than the cap holds no capped span. A capped span is read + in shallow pieces, which can part a ``kill`` from the substitution naming its + target. It is a count, not a walk: a quoted ``)`` cannot hide a level from + it. It is only the cheap gate; the self-kill floor then asks the span reader + itself whether a span was capped, since 32 FLAT openers nest nothing. + """ + n = text.count("$(") + text.count("<(") + text.count(">(") + text.count("${") + return n >= _MAX_SPAN_NESTING + + +class _HeredocInSpan(Exception): + """A heredoc body would start inside the span being walked. The walk does not + read heredocs, so the caller reads that span the flat way (:func:`_flat_close`).""" + + +def _matching_close_brace(text: str, nest: int = 0) -> "tuple[int, bool, bool, bool]": + """``(index just past the ``}``, proven, heredoc_pending, capped)`` for a ``${`` + opened just before *text*. + + A parameter expansion's body is quote-aware and nests: a ``}`` inside a quote + (``${x:-"}"}``) or inside a nested ``${``/``$(`` is data, not the close. + :func:`_close_paren_walk` skips the expansion as a unit through this walk, + so its interior cannot close or re-quote the enclosing span. + + ``proven`` is False when the brace never closes (the index is then + ``len(text)``), when a nested span is unproven, or past + :data:`_MAX_SPAN_NESTING`; the enclosing walk then reports its own span + unproven too. ``heredoc_pending`` and ``capped`` are as in + :func:`_close_paren_walk`. + """ + pos = 0 + state = 0 + ansi = False + deep = nest >= _MAX_SPAN_NESTING + proven = not deep + pending = False + capped = deep + while pos < len(text): + jumped = False + # Literal ``$`` steps right before this one (a walk resumes after a + # ``)``/``}``/backtick/newline, never after a ``$``). + run = 0 + for step in _iter_shell_chars(text[pos:], state, ansi): + off = pos + step.offset + prev_run, run = run, run + 1 if step.text == "$" else 0 + if step.state != 1 and not deep: + # bash pairs a run of dollars off from the left (``$$`` is the + # PID), so after an ODD run of literal dollars this ``$`` opens + # nothing and the ``{``/``(`` after it is data. An escaped ``\$`` + # is one step whose text is not ``$``, so it pairs with nothing. + lone = text[off] != "$" or prev_run % 2 == 0 + nested = None + if lone and text.startswith("$(", off): + nested = _close_paren_walk(text[off + 2 :], nest + 1) + elif step.state == 0 and text.startswith(("<(", ">("), off): + # A process sub opens only UNQUOTED: a ``"<("`` is data. + nested = _close_paren_walk(text[off + 2 :], nest + 1) + elif lone and text.startswith("${", off): + nested = _matching_close_brace(text[off + 2 :], nest + 1) + if nested is not None: + rel, ok, inner_pending, inner_capped = nested + proven = proven and ok + pending = pending or inner_pending + capped = capped or inner_capped + pos, state, ansi = off + 2 + rel, step.state, step.ansi + jumped = True + break + if step.char == "`": + # A backtick delimiter is escaped only by a backslash -- a + # single quote does NOT protect it -- so close on the + # bash-faithful ``_backtick_closer``; ``-1`` means no closer, + # so the rest is backtick data. + closer = _backtick_closer(text, off + 1) + pos = closer + 1 if closer != -1 else len(text) + state, ansi = step.state, step.ansi + jumped = True + break + if not step.active: + continue + # A bare ``{`` is data: bash closes ``${x:-a{b}c}`` at the first + # ``}``. Only a nested ``${`` (skipped above) nests. A newline while a + # nested heredoc is pending starts its body inside this span. + if step.char == "\n" and pending: + raise _HeredocInSpan + if step.char == "}": + return (off + len(step.text), proven, pending, capped) + if not jumped: + break + return (len(text), False, pending, capped) + + +def _close_paren_walk(text: str, nest: int = 0) -> "tuple[int, bool, bool, bool]": + """:func:`_matching_close_paren` plus ``heredoc_pending`` and ``capped``. + + ``heredoc_pending`` is True when a ``<<`` (not a here-string ``<<<``) is still + waiting for its body when the span closes; bash reads that body from the + lines after the close, so a nested walk passes it up and the enclosing walk + raises :class:`_HeredocInSpan` at its next newline. A newline reached while a + heredoc is pending raises it too: the body starts INSIDE the span, and a + heredoc is not parsed here. So ``rm -rf "$(cat <= _MAX_SPAN_NESTING + proven = not deep + pending = False + capped = deep + here_string_end = 0 + while pos <= len(text): + jumped = False + run = 0 # literal ``$`` steps right before this one, as in the brace walk + for step in _iter_shell_chars(text[pos:], state, ansi): + off = pos + step.offset + prev_run, run = run, run + 1 if step.text == "$" else 0 + # A nested substitution / expansion opens a FRESH quoting context, so + # skip it whole as a unit and resume the OUTER quote state -- its + # inner quotes and parens must not leak into THIS span. A ``$`` is + # literal only inside a single quote (state 1); ``<(`` / ``>(`` are + # process subs only when unquoted; a ``$`` paired into ``$$`` (the + # PID) opens nothing. + if step.state != 1 and not deep: + lone = text[off] != "$" or prev_run % 2 == 0 + nested = None + if lone and text.startswith("$(", off): + nested = _close_paren_walk(text[off + 2 :], nest + 1) + elif step.state == 0 and text.startswith(("<(", ">("), off): + nested = _close_paren_walk(text[off + 2 :], nest + 1) + elif lone and text.startswith("${", off): + nested = _matching_close_brace(text[off + 2 :], nest + 1) + if nested is not None: + rel, ok, inner_pending, inner_capped = nested + proven = proven and ok + pending = pending or inner_pending + capped = capped or inner_capped + pos = off + 2 + rel + state, ansi = step.state, step.ansi + jumped = True + break + if step.char == "`": + # Escaped only by a backslash, never by a single quote. A + # ``-1`` (no closer) is an UNTERMINATED backtick -- broken + # text, not a live substitution -- so mark the span unproven + # and step over just this one backtick (the fail-closed + # direction), rather than swallowing the rest as backtick data + # and hiding the operand after the span's real ``)``. + closer = _backtick_closer(text, off + 1) + if closer == -1: + proven = False + pos = off + 1 + else: + pos = closer + 1 + state, ansi = step.state, step.ansi + jumped = True + break + if not step.active: + continue + ch = step.char + # A ``#`` opens a comment only at the START of a word, and the + # character before it is read as the shell reads it -- a folded + # continuation leaves ``a\`` + newline + ``#b`` one ordinary word. + if ch == "#" and _opens_comment(text, off): + newline = text.find("\n", off) + if newline == -1: + return (len(text), False, pending, capped) + if pending: + raise _HeredocInSpan + pos = newline + 1 + state, ansi = 0, False + jumped = True + break + if ch == "<" and off >= here_string_end: + if text.startswith("<<<", off): + here_string_end = off + 3 + continue + if text.startswith("<<", off): + pending = True + if ch == "\n" and pending: + raise _HeredocInSpan + if _word_at(text, off, "case") and _arms_case_context(text, off, cases > 0): + cases += 1 + continue + if _word_at(text, off, "esac") and _in_command_position(text, off): + cases = max(0, cases - 1) + continue + if cases and ch in "()": + continue + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if depth == 0: + return (off + len(step.text), proven, pending, capped) + if not jumped: + break + return (len(text), False, pending, capped) + + +def _span_walk(text: str) -> "tuple[int, bool, bool]": + """``(index just past the close, proven, capped)`` for the extractor. A span a + heredoc body starts inside is UNPROVEN, closed where :func:`_flat_close` + closes it, so the extractor scans its whole remainder.""" + try: + rel, proven, _pending, capped = _close_paren_walk(text) + except _HeredocInSpan: + return (_flat_close(text)[0], False, False) + return (rel, proven, capped) + + +def _flat_close(text: str, open_end: int = 0) -> "tuple[int, bool]": + """``main``'s span reading: ``(index just past the matching ``)``, proven)`` + for a paren opened before *open_end*, walked through :func:`_iter_shell_chars` + with comments and ``case`` arms read, but no nested construct skipped as a + unit and no heredoc read. It needs no recursion, so no nesting cap.""" + depth = 1 + cases = 0 + pos = 0 + state = 0 + ansi = False + while pos <= len(text): + jumped = False + for step in _iter_shell_chars(text[pos:], state, ansi): + off = pos + step.offset + if off < open_end or not step.active: + continue + ch = step.char + if ch == "#" and _opens_comment(text, off): + newline = text.find("\n", off) + if newline == -1: + return (len(text), False) + pos = newline + 1 + state, ansi = 0, False + jumped = True + break + if _word_at(text, off, "case") and _arms_case_context(text, off, cases > 0): + cases += 1 + continue + if _word_at(text, off, "esac") and _in_command_position(text, off): + cases = max(0, cases - 1) + continue + if cases and ch in "()": + continue + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if depth == 0: + return (off + len(step.text), True) + if not jumped: + break + return (len(text), False) + + +def _matching_close_paren(text: str) -> "tuple[int, bool]": + """``(index just past the matching ``)``, proven)`` for a paren opened just + before *text*, walked QUOTE-AWARELY through :func:`_iter_shell_chars`. + + *text* is the slice after the opener, so the walk starts at neutral state: + bash parses a substitution body in its own quoting context. + + THE one span computation for a substitution or subshell body. Both the + git-publish boundary walk and the nested-payload EXTRACTOR read it, because a + body extracted over a different span than the boundary proved is exactly how a + nested command escapes the scan: ``git push origin my-feature > >(X=')' git + push origin main)`` truncated the extracted body at the QUOTED ``)``, so the + payload came back as ``X='`` and the nested publish of a protected branch was + never scanned at all -- ``is_denied`` allowed a command bash runs. + + Quoting is not the only way a ``)`` reaches this walk as an ordinary + character. Two COMMAND-GRAMMAR constructs put one there too, and each was + measured allowing a payload this module still refuses: + + * a ``#`` COMMENT runs to the end of its line, so the ``)`` in + ``$(: # )`` is commented out and the substitution closes on a LATER line. + Counting it closed the body at ``: # ``, and the ``printf`` behind it that + computed the credential-minting verb was never scanned. + * a ``case`` PATTERN is terminated by ``)``. In + ``$(case x in x) printf token;; esac)`` the pattern's ``)`` is not the + substitution's, and reading it as one truncated the body to ``case x in x``. + + Between ``case`` and its ``esac`` both parens are therefore ignored, which + keeps a ``(x|y)`` pattern balanced-neutral as well. ARMING is generous and + DISARMING is strict on purpose: missing a real ``case`` closes the body EARLY, + which is the bypass, while missing a real ``esac`` only runs it long, which is + imprecision. So ``case`` arms on any standalone word IN COMMAND POSITION + (:func:`_arms_case_context` -- bash only reads the reserved word there, so + ``echo case x in y`` does not arm on ordinary commands, and every + undecidable position still arms) and ``esac`` disarms only in command + position -- an ``esac`` passed to a command as an ARGUMENT would otherwise + end the rule early and let the next paren close. + + A nested ``$(``, ``<(``, ``>(`` or ``${...}`` is skipped as a unit + (:func:`_close_paren_walk`). A span that walk cannot prove -- a heredoc body + starts inside it, it is read past :data:`_MAX_SPAN_NESTING`, or it does not + balance -- is read the flat way ``main`` reads every span + (:func:`_flat_close`), so the rm floor and the self-kill walk resume where + ``main`` resumes. ``proven`` is False when the parens never balance before the + text ends; the caller must fail CLOSED on that. + """ + try: + end, proven, _pending, capped = _close_paren_walk(text) + except _HeredocInSpan: + return _flat_close(text) + return (end, True) if proven and not capped else _flat_close(text) + + +def _matching_close_backtick(text: str, start: int) -> "tuple[int, bool]": + """``(index just past the closing backtick, proven)`` for a body at *start*. + + The paren walk above got quote awareness; the BACKTICK form was still taken + pairwise with a plain ``find``, so a backtick the shell reads as a literal -- + ``` `A='`'; ...` ``` -- was counted as the closer and truncated the body there. + One state machine now answers both spellings, which is what keeps them from + drifting apart again. + + A backtick inside DOUBLE quotes really is a closer (``"`cmd`"`` runs ``cmd``), + so only single-quoted and escaped ones are data. Read through the same + ``in_single`` idiom the argv walk uses, rather than a second reading of the + state. + + HARDENING, not a patched hole: no payload this module refuses was reachable + through the pairwise version -- the strings it mis-read are ones bash itself + rejects, because backticks do not nest unescaped. It is fixed so the two + closers cannot disagree, not because a bypass was measured. Sibling + scanner: :func:`_backtick_closer` is the exact bash-faithful variant + (backslash-only, returns ``-1``) -- reach for it when the caller needs the + closer bash itself would pick rather than a fail-closed span. + """ + for step in _iter_shell_chars(text[start:]): + escaped = len(step.text) == 2 + in_single = step.state == 1 and not (step.char == "'" and step.active) + if not escaped and not in_single and step.char == "`": + return (start + step.offset + len(step.text), True) + return (len(text), False) + + +def _backtick_closer(source: str, start: int) -> int: + """Index of the backtick that CLOSES a substitution opened before *start*. + + Within backticks bash strips a backslash before ``$``, ``\\``` and ``\\\\``, + so an escaped backtick is data and must not be taken as the closer -- + ``str.find`` did, and it truncated ``kill `printf '\\`' ; pgrep -f ``` + one clause short of the target's name (found in pre-push review, bash- + measured: the inner command past the escaped backtick runs). Quotes do NOT + protect a backtick from closing, so this scan honours backslashes only. + + -1 when no unescaped closer exists before the text ends. Sibling scanner: + :func:`_matching_close_backtick` is the quote-aware over-approximating + variant returning ``(index, proven)`` -- reach for that one when an + unproven span must fail closed rather than fall through. + """ + j = start + n = len(source) + while j < n: + if source[j] == "\\": + j += 2 + continue + if source[j] == "`": + return j + j += 1 + return -1 diff --git a/src/kiro_crew/security/shell_normalizer_walk.py b/src/kiro_crew/security/shell_normalizer_walk.py index 149c7484e25..ba8ab7b3c3b 100644 --- a/src/kiro_crew/security/shell_normalizer_walk.py +++ b/src/kiro_crew/security/shell_normalizer_walk.py @@ -3,16 +3,16 @@ The part of :mod:`kiro_crew.security.shell_normalizer` that reads a command line the way a shell's lexer does, before any token is attributed to a program: the quote and escape state machine (``_iter_shell_chars``, ``_shell_quote_walk``), line -continuations, the structural matches a substitution or a ``case`` arm needs -(closing parens and backticks, comments, command position), substitution bodies, -word splitting and dequoting, statement separators, the push-token reads, and the -word views that peel brackets, parameter defaults and empty substitutions off a -word. +continuations, the structural reads a substitution or a ``case`` arm needs +(comments, command position), word splitting and dequoting, statement separators, +the push-token reads, and the word views that peel brackets, parameter defaults and +empty substitutions off a word. Nothing here decides anything, and nothing here imports ``shell_normalizer``: the attribution layer there reads this one, never the reverse. ``kiro_crew.security.shell_normalizer`` re-exports every name a caller imports from -it, so that import path holds. New lexical reading belongs here. +it, so that import path holds. New lexical reading belongs here; the span +matcher and the substitution bodies live in ``shell_normalizer_span``. """ from __future__ import annotations @@ -242,53 +242,6 @@ def _ends_argv(token: str) -> bool: return "|" in token or ";" in token -def _substitution_bodies(text: str) -> "list[str]": - """The body of each command substitution in *text*. - - ``$(...)`` is scanned with paren nesting so a nested substitution closing - first does not truncate the outer body; backticks are taken pairwise. Only - the BODY is returned -- a bare ``kill`` must not be attributed a name that - merely appears in a LATER, unrelated command of the same line. - - The nesting walk is QUOTE-AWARE, through the same - :func:`_matching_close_paren` span the git-publish boundary walk uses. A - private, quote-unaware copy of it truncates the body at a QUOTED ``)``, and - that loses the nested command entirely rather than merely mis-sizing the span: - ``git push origin my-feature > >(X=')' git push origin main)`` extracts the - body ``X='``, so the nested publish of a protected branch is never scanned - and ``is_denied`` returns None for a command bash executes. An UNPROVEN span - yields the whole remainder, which is the fail-closed direction -- scanning - text that is not really in the body can only add findings. - - Quoting was not the whole of it. That span helper counted a ``)`` that shell - COMMAND GRAMMAR also puts there as an ordinary character, and two such - spellings were measured allowing a payload this module still refuses: - a ``#`` comment (``$(: # )`` closes on a later line) and a ``case`` pattern - (``$(case x in x) printf token;; esac)``). Both truncated the body before the - verb, so the value assembled from it was never recognised. The closer for the - BACKTICK form reads the same state machine now too, so the two spellings - cannot disagree. - """ - bodies: list[str] = [] - i = 0 - while i < len(text): - # PROCESS substitutions run their body as a command just as a command - # substitution does -- ``cat <(kirocrew token)`` executes the inner command and - # feeds its output through a pipe. Same paren-nesting walk. - if text.startswith(("<(", ">(", "$("), i): - end, proven = _matching_close_paren(text, i + 2) - bodies.append(text[i + 2 : end - 1] if proven else text[i + 2 :]) - i = end - continue - if text[i] == "`": - end, proven = _matching_close_backtick(text, i + 1) - bodies.append(text[i + 1 : end - 1] if proven else text[i + 1 :]) - i = end - else: - i += 1 - return bodies - - # A shell removes ``backslash + newline`` while lexing (line continuation), so # ``kirocrew \restart`` runs ``kirocrew restart``. ``shlex`` instead keeps # the escaped newline as a literal in the token, so the floor pre-joins it to @@ -811,147 +764,6 @@ def _arms_case_context(text: str, index: int, in_case_body: bool = False) -> boo return True # chain too deep to decide -- arm, the long direction -def _matching_close_paren(text: str, open_end: int) -> "tuple[int, bool]": - """``(index just past the matching ``)``, proven)`` for a paren opened before - *open_end*, walked QUOTE-AWARELY through :func:`_iter_shell_chars`. - - THE one span computation for a substitution or subshell body. Both the - git-publish boundary walk and the nested-payload EXTRACTOR read it, because a - body extracted over a different span than the boundary proved is exactly how a - nested command escapes the scan: ``git push origin my-feature > >(X=')' git - push origin main)`` truncated the extracted body at the QUOTED ``)``, so the - payload came back as ``X='`` and the nested publish of a protected branch was - never scanned at all -- ``is_denied`` allowed a command bash runs. - - Quoting is not the only way a ``)`` reaches this walk as an ordinary - character. Two COMMAND-GRAMMAR constructs put one there too, and each was - measured allowing a payload this module still refuses: - - * a ``#`` COMMENT runs to the end of its line, so the ``)`` in - ``$(: # )`` is commented out and the substitution closes on a LATER line. - Counting it closed the body at ``: # ``, and the ``printf`` behind it that - computed the credential-minting verb was never scanned. - * a ``case`` PATTERN is terminated by ``)``. In - ``$(case x in x) printf token;; esac)`` the pattern's ``)`` is not the - substitution's, and reading it as one truncated the body to ``case x in x``. - - Between ``case`` and its ``esac`` both parens are therefore ignored, which - keeps a ``(x|y)`` pattern balanced-neutral as well. ARMING is generous and - DISARMING is strict on purpose: missing a real ``case`` closes the body EARLY, - which is the bypass, while missing a real ``esac`` only runs it long, which is - imprecision. So ``case`` arms on any standalone word IN COMMAND POSITION - (:func:`_arms_case_context` -- bash only reads the reserved word there, so - ``echo case x in y`` does not arm on ordinary commands, and every - undecidable position still arms) and ``esac`` disarms only in command - position -- an ``esac`` passed to a command as an ARGUMENT would otherwise - end the rule early and let the next paren close. - - ``proven`` is False when the parens never balance before the text ends. The - caller must fail CLOSED on that: for an extractor the safe reading is the - whole remainder (scan more, never less). All three call sites are extractors, - so a span that reaches too far only feeds them text to inspect. - """ - depth = 1 - cases = 0 - pos = 0 - state = 0 - ansi = False - while pos <= len(text): - jumped = False - for step in _iter_shell_chars(text[pos:], state, ansi): - off = pos + step.offset - if off < open_end: - continue - if not step.active: - continue - ch = step.char - # A ``#`` opens a comment only at the START of a word, and the - # character before it is read as the shell reads it -- a folded - # continuation leaves ``a\`` + newline + ``#b`` one ordinary word. - if ch == "#" and _opens_comment(text, off): - newline = text.find("\n", off) - if newline == -1: - return (len(text), False) - pos = newline + 1 - state, ansi = 0, False - jumped = True - break - if _word_at(text, off, "case") and _arms_case_context(text, off, cases > 0): - cases += 1 - continue - if _word_at(text, off, "esac") and _in_command_position(text, off): - cases = max(0, cases - 1) - continue - if cases and ch in "()": - continue - if ch == "(": - depth += 1 - elif ch == ")": - depth -= 1 - if depth == 0: - return (off + len(step.text), True) - if not jumped: - break - return (len(text), False) - - -def _matching_close_backtick(text: str, start: int) -> "tuple[int, bool]": - """``(index just past the closing backtick, proven)`` for a body at *start*. - - The paren walk above got quote awareness; the BACKTICK form was still taken - pairwise with a plain ``find``, so a backtick the shell reads as a literal -- - ``` `A='`'; ...` ``` -- was counted as the closer and truncated the body there. - One state machine now answers both spellings, which is what keeps them from - drifting apart again. - - A backtick inside DOUBLE quotes really is a closer (``"`cmd`"`` runs ``cmd``), - so only single-quoted and escaped ones are data. Read through the same - ``in_single`` idiom the argv walk uses, rather than a second reading of the - state. - - HARDENING, not a patched hole: no payload this module refuses was reachable - through the pairwise version -- the strings it mis-read are ones bash itself - rejects, because backticks do not nest unescaped. It is fixed so the two - closers cannot disagree, not because a bypass was measured. Sibling - scanner: :func:`_backtick_closer` is the exact bash-faithful variant - (backslash-only, returns ``-1``) -- reach for it when the caller needs the - closer bash itself would pick rather than a fail-closed span. - """ - for step in _iter_shell_chars(text[start:]): - escaped = len(step.text) == 2 - in_single = step.state == 1 and not (step.char == "'" and step.active) - if not escaped and not in_single and step.char == "`": - return (start + step.offset + len(step.text), True) - return (len(text), False) - - -def _backtick_closer(source: str, start: int) -> int: - """Index of the backtick that CLOSES a substitution opened before *start*. - - Within backticks bash strips a backslash before ``$``, ``\\``` and ``\\\\``, - so an escaped backtick is data and must not be taken as the closer -- - ``str.find`` did, and it truncated ``kill `printf '\\`' ; pgrep -f ``` - one clause short of the target's name (found in pre-push review, bash- - measured: the inner command past the escaped backtick runs). Quotes do NOT - protect a backtick from closing, so this scan honours backslashes only. - - -1 when no unescaped closer exists before the text ends. Sibling scanner: - :func:`_matching_close_backtick` is the quote-aware over-approximating - variant returning ``(index, proven)`` -- reach for that one when an - unproven span must fail closed rather than fall through. - """ - j = start - n = len(source) - while j < n: - if source[j] == "\\": - j += 2 - continue - if source[j] == "`": - return j - j += 1 - return -1 - - def _resolved_word_view(word: str) -> str: """*word* with parameter defaults resolved, empty substitutions collapsed, then one-character bracket classes removed -- the composed per-word diff --git a/test/campaign_facade_inventories/kiro_crew.security.shell_normalizer.json b/test/campaign_facade_inventories/kiro_crew.security.shell_normalizer.json index a153bd33136..5caaebf2db6 100644 --- a/test/campaign_facade_inventories/kiro_crew.security.shell_normalizer.json +++ b/test/campaign_facade_inventories/kiro_crew.security.shell_normalizer.json @@ -3,7 +3,7 @@ "ENV_ASSIGNMENT_RE", "normalize_shell_command" ], - "note": "The shell reader keeps argv attribution and re-imports what callers name from its lexical owners (shell_normalizer_walk, _redirects, _escapes, _programs); its two public names are the path normalizer it defines and the assignment pattern it imports.", + "note": "The shell reader keeps argv attribution and re-imports what callers name from its lexical owners (shell_normalizer_walk, _span, _redirects, _escapes, _programs); its two public names are the path normalizer it defines and the assignment pattern it imports.", "rebound": { "ENV_ASSIGNMENT_RE": "kiro_crew.trust_patterns" } diff --git a/test/test_denied_commands_security_kill.py b/test/test_denied_commands_security_kill.py index 2158966e7d1..84eca5a7ce4 100644 --- a/test/test_denied_commands_security_kill.py +++ b/test/test_denied_commands_security_kill.py @@ -980,6 +980,88 @@ def test_pipe_still_separates_the_argv_window(self, cmd): assert security._is_self_kill(cmd.format(n=_NAME).lower()) is False +class TestSelfKillHeredocSpanDoesNotStopTheRawWalk: + """A HEREDOC inside the raw window's span must not end the raw walk early. + + ``_matching_close_paren`` reads a span a heredoc body starts inside the flat + way ``main`` reads every span, so ``_bare_kill_raw_bodies`` resumes where + ``main`` resumes and still reaches the LATER top-level ``$k $(printf ')' ; + pgrep -f )`` -- the decoyed-alias clause only this raw walk catches. + ``main`` refuses the self-kill rows and allows the twins, and so does this + reading; a product named by a LATER separate command + (``kill $(cat <<'EOF'⏎4242⏎EOF⏎); kirocrew status``) is that command's word, + not the kill's operand. + """ + + @pytest.mark.parametrize( + "cmd", + [ + # the issue's three spellings: a heredoc span, then the decoyed + # alias clause the raw walk must still reach + "k=kill; x=$(cat <(echo >(git push origin main))") == [ - "echo >(git push origin main)" + "echo >(git push origin main)", ] assert "git push origin main" in security._shell_payload_sources( "cat >(echo >(git push origin main))" ) - # UNPROVEN span -> the whole remainder, the fail-closed direction. + # UNPROVEN span -> the whole remainder, the fail-closed direction, plus the + # span up to the last ``)``. assert _substitution_bodies(">(git push origin main") == ["git push origin main"] assert _substitution_bodies(">(X=')' git push origin main") == [ - "X=')' git push origin main" + "X=')' git push origin main", + "X='", ] def test_the_span_helper_is_quote_aware_and_reports_proof(self): from kiro_crew.security import _matching_close_paren - # offset 2 is just past a ">(" / "$(" opener. - assert _matching_close_paren(">(a)", 2) == (4, True) - assert _matching_close_paren(">(a(b))", 2) == (7, True) + # The text is the slice just past a ">(" / "$(" opener. + assert _matching_close_paren("a)") == (2, True) + assert _matching_close_paren("a(b))") == (5, True) # A quoted closer is data and must not end the span. - assert _matching_close_paren(">(X=')' a)", 2) == (10, True) - assert _matching_close_paren('>(X=")" a)', 2) == (10, True) - assert _matching_close_paren(">(X=$')' a)", 2) == (11, True) + assert _matching_close_paren("X=')' a)") == (8, True) + assert _matching_close_paren('X=")" a)') == (8, True) + assert _matching_close_paren("X=$')' a)") == (9, True) # A quoted OPENER must not deepen it either. - assert _matching_close_paren(">(X='(' a)", 2) == (10, True) + assert _matching_close_paren("X='(' a)") == (8, True) # Never balanced -> not proven, and the span runs to the end. - assert _matching_close_paren(">(a", 2) == (3, False) - assert _matching_close_paren(">(X=')' a", 2) == (9, False) + assert _matching_close_paren("a") == (1, False) + assert _matching_close_paren("X=')' a") == (7, False) def test_the_self_protection_substitution_scan_is_quote_aware_too(self): # Same class at the self-protection rule: its private str.count walk @@ -688,3 +692,1063 @@ def test_every_earlier_round_still_holds(self): ) assert security._GIT_PUBLISH_UNGATED in floor("git push origin feature > >(echo main") assert security._git_push_args("bash -c '(cd /tmp && git push origin my-feature)'") is None + + +class TestSubstitutionInsideDoubleQuotesIsExtractedFromAFreshContext: + """A ``$(`` inside a double-quoted word starts a FRESH quoting context. + + bash parses a substitution body on its own, so ``"$(echo ")")"`` holds the + body ``echo ")"``. The extractor walked the span from the start of the whole + command instead, so the outer ``"`` made every ``)`` in the body read as + quoted. Two failures followed from the one mis-delimit: + + * the span never proved, so the body swallowed the rest of the line and an + ordinary ``git commit -m "$(date)" && git push origin feat`` was refused + as an unverifiable push; + * a ``"`` inside the body flipped the outer state, so a nested wrapper after + it was cut out of the body: ``echo "$(echo ")"; bash -c '')"`` + was allowed although bash runs the push. + """ + + def test_the_body_is_delimited_at_its_own_closer(self): + from kiro_crew.security import _substitution_bodies + + assert _substitution_bodies('git commit -m "$(date)" && x') == ["date"] + assert _substitution_bodies('echo "$(echo ")")"') == ['echo ")"'] + # A heredoc body inside the span leaves it unproven: the whole + # remainder, the span up to the last ")", and -- since the span still + # has a real close -- its proven-close body NON-suppressed, so a nested + # opener bash runs after the heredoc body is extracted. + heredoc = "git commit -m \"$(cat <<'eof'\nsubject\neof\n)\" && x" + assert _substitution_bodies(heredoc) == [ + "cat <<'eof'\nsubject\neof\n)\" && x", + "cat <<'eof'\nsubject\neof\n", + "cat <<'eof'\nsubject\neof\n", + ] + # An unbalanced body is still the whole remainder, the fail-closed side. + assert _substitution_bodies('echo "$(date" && x') == ['date" && x'] + + def test_a_feature_push_after_a_quoted_substitution_is_allowed(self): + for cmd in ( + 'git commit -m "$(date)" && git push origin feat', + 'echo "$(date)"; git push origin feat', + 'git commit -m "fix $(git rev-parse --short HEAD)" && git push -u origin feat', + ): + assert security.is_denied(cmd) is None, cmd + + def test_a_protected_push_around_a_quoted_substitution_stays_denied(self): + for cmd in ( + 'git commit -m "$(date)" && git push origin main', + 'git commit -m "$(date)" && git push', + 'git commit -m "$(date)" && git push origin $b', + 'git commit -m "$(date)" && git push origin {feat,main}', + "git commit -m \"$(date)\" && git push origin $'main'", + 'git commit -m "$(git push origin main)"', + 'echo "$(echo ")"; bash -c \'git push origin main\')"', + "echo \"$(echo ')'; eval 'git push origin main')\"", + "echo '$(' \"$(bash -c 'git push origin main')\" ')'", + ): + assert security.is_denied(cmd) is not None, cmd + + def test_a_heredoc_body_inside_the_substitution_stays_scanned(self): + msg = 'git commit -m "$(cat <<{tag}\n{body}\n{end}\n)" && git push origin feat' + for tag, body, end in ( + ("'EOF'", "git push origin main", "EOF"), + ("EOF", "git push origin main", "EOF"), + ('"EOF"', "git push origin $b", "EOF"), + ("'EOF'", "git push origin {x,main}", "EOF"), + ("'EOF'", "git push origin main", " EOF"), + ("-'EOF'", "\tgit push origin main", "\tEOF"), + ("EOF", "$(git push origin main)", "EOF"), + ("'EOF'", ")\nEOF\nbash -c 'git push origin main'", "x"), + ): + cmd = msg.format(tag=tag, body=body, end=end) + assert security.is_denied(cmd) is not None, cmd + unterminated = ( + "git commit -m \"$(cat <<'EOF'\ngit push origin main\n)\" && git push origin feat" + ) + assert security.is_denied(unterminated) is not None + + +class TestANestedConstructDoesNotHideAPushBehindItsOwnQuoting: + """A substitution body, a ``${...}`` expansion and a backtick each open their + OWN quoting context, so a ``)`` or ``"`` inside one is DATA, not the close of + the enclosing ``$(``. The matcher skips each as a unit; without that, the + inner quoting leaks out and closes the span early, so a ``bash -c ''`` + sitting after the real close goes unscanned. A heredoc body is data too, but + the matcher does not parse it: a body starting inside the span leaves the + span unproven, so the whole remainder is scanned. + + The exact command that regressed: + ``echo "$(echo "$(echo ")")" ; bash -c 'git push origin main')"`` -- bash runs + the push (the first ``)`` after the inner ``"`` closed the span and the body + came back as ``echo "$(echo "``). Every row below is a command bash really + runs a protected push for, generated across nesting depth, the construct that + carries the ``)``, the launcher and the push target. + """ + + _OPUS = 'echo "$(echo "$(echo ")")" ; bash -c \'git push origin main\')"' + + def test_the_opus_command_is_denied(self) -> None: + assert security.is_denied(self._OPUS) is not None + + def test_a_push_hidden_behind_any_nested_construct_is_denied(self) -> None: + # Each CARRIER puts a bare ``)`` (or re-quote) in a context the matcher + # must skip whole; a wrong delimit lets the launcher behind it run free. + carriers = ( + 'echo ")"', # a nested $( whose body re-quotes + "echo ')'", + 'echo "${x:-")"}"', # a ${...} default value + 'echo "`echo ")"`"', # a backtick sub + ) + launchers = ("bash -c", "sh -c", "eval") + targets = ("main", "mainline", "$b", "{feat,main}", "$'main'") + for depth in (1, 2, 3): + for carrier in carriers: + for sep in (" ; ", "; ", " && ", "\n"): + for launcher in launchers: + for target in targets: + push = f"git push origin {target}" + inner = f"\"$({carrier}{sep}{launcher} '{push}')\"" + cmd = inner + for _ in range(depth - 1): + cmd = f'"$(echo {cmd})"' + cmd = "echo " + cmd + assert security.is_denied(cmd) is not None, cmd + # a bare ``git push`` (current branch) is protected too. + assert security.is_denied('echo "$(echo ")" ; bash -c \'git push\')"') is not None + + def test_a_heredoc_body_holding_the_close_paren_is_denied(self) -> None: + for sep in (" ; ", "; ", " && ", "\n"): + cmd = ( + "echo \"$(cat <<'E'\n)\nE\n" + + sep.lstrip("\n") + + "bash -c 'git push origin main')\"" + ) + assert security.is_denied(cmd) is not None, cmd + # the substitution closing right after the heredoc, push after it. + assert ( + security.is_denied("git commit -m \"$(cat <<'E'\n)\nE\n)\" && git push origin main") + is not None + ) + + def test_a_feature_push_after_a_quoted_substitution_stays_allowed(self) -> None: + for cmd in ( + 'git commit -m "$(date)" && git push origin feat', + 'echo "$(date)"; git push -u origin feat', + 'git commit -m "fix $(git rev-parse --short HEAD)" && git push origin feat', + 'echo "$(echo ")")" && git push origin feat', + ): + assert security.is_denied(cmd) is None, cmd + + def test_a_heredoc_left_pending_by_a_nested_substitution_is_denied(self) -> None: + # ``$(cat < None: + # bash ends the body at the tag AFTER quote removal: ``$'\\x45'`` is ``E`` + # and ``"E\\"F"`` is ``E"F``. The matcher reads no tag at all, so no + # spelling can make it end the body early. + for tag, end in (("$'E'", "E"), ("$'\\x45'", "E"), ('"E\\"F"', 'E"F'), ("\\E", "E")): + for push in ("git push origin main", "git push"): + cmd = f"echo \"$(cat <<{tag}\n)\n{end}\nbash -c '{push}')\"" + assert security.is_denied(cmd) is not None, cmd + + def test_a_body_line_that_differs_only_in_case_does_not_end_the_body(self) -> None: + # The gate reads lowercased text, but bash ends ``< None: + from kiro_crew.security.shell_normalizer import _matching_close_paren + + # bash closes at the first ")" (index 10) + assert _matching_close_paren("cat << None: + from kiro_crew.security.shell_normalizer_span import _span_walk + + # the body's ")" is data, so the span is not proven at it. + assert _span_walk("cat << None: + from kiro_crew.security.shell_normalizer_span import _span_walk + + assert _span_walk(" (( 1<<2 ))\n) ; P\n2\n)")[1] is False + # ``$(( 1<<2\n))``: the body handed over starts with the second "(". + assert _span_walk("( 1<<2\n))\n2\n)")[1] is False + assert _span_walk("echo $(( 1<<2\n))\n) ; P\n2\n)")[1] is False + # A heredoc after the arithmetic is unproven the same way. + assert _span_walk(" (( 1<<2 )); cat < None: + launch = f"\"bash -c '{self._PUSH}'\"" + for cmd in ( + f'echo "$( (( 1<<2 ))\necho hi)" {launch}', + f'echo "$(( 1<<2\n))" {launch}', + f'echo "$(echo $(( 1<<2\n)))" {launch}', + ): + assert security.is_denied(cmd) is not None, cmd + + def test_a_shift_closed_on_its_own_line_still_proves_the_span(self) -> None: + from kiro_crew.security.shell_normalizer import _matching_close_paren + + # No body line is reached before the close, so the span is proven. + assert _matching_close_paren(": ; ((1<<2)) ; date)\nx") == ( + len(": ; ((1<<2)) ; date)"), + True, + ) + assert security.is_denied('echo "$(: ; ((1<<2)) ; date)" && git push origin feat') is None + assert security.is_denied("echo $(( 1<<2 )) && git push origin feat") is None + + +class TestAMisreadSpanFailsClosed: + """Spellings the span matcher misread, each letting a command through that + the gate refuses on main: a ``$[...]`` arithmetic shift read as a heredoc, a + heredoc still open at a top-level ``)`` that left the rm floor's span + unproven, a deep ``${a:-${a:-...}}`` chain that ran the walk out of recursion + before the payload behind it was queued, a single-quoted ``$(`` whose span + swallowed the real opener after it, and a bare ``{`` counted as nesting. So + did a quoted ``<(`` or a ``$$`` read as an opener, a ``'`` carried out of an + earlier span, and a capped span given no reading past its heredoc. And so + did a ``'`` in a comment or a heredoc body, read as opening a quote; a capped + span's remainder whose re-extraction was suppressed by its value; and a + single-quoted backtick opened as a substitution. And a capped span a data + ``)`` at the cap kept from ever closing, hiding a later top-level opener's + push inside its value-suppressed remainder. And a capped span nested inside + an outer body whose OWN text held the launcher, and a proven body that equals + a suffix of its source, suppressed by that value.""" + + _PUSH = "git push origin main" + + def test_an_arithmetic_bracket_shift_cannot_hide_a_push(self) -> None: + cmd = f'echo "$(echo $[1< None: + for cmd in ( + 'rm -rf "$(cat < None: + from kiro_crew.security.shell_normalizer_span import _span_walk + + deep = "${a:-" * 1000 + "x" + "}" * 1000 + assert security.is_denied(f"bash -c '{self._PUSH}'; echo $(echo {deep})") is not None + assert security.is_denied(f'rm -rf "$(echo {deep})" ~') is not None + # past the nesting cap the reading is unproven, never an exception + assert _span_walk(f"echo {deep})")[1] is False + + def test_a_single_quoted_opener_cannot_swallow_the_real_one(self) -> None: + cmd = f"echo '$(' \"$(echo ')' ; bash -c '{self._PUSH}')\"" + assert security.is_denied(cmd) is not None + + def test_a_quote_in_a_comment_or_heredoc_cannot_hide_an_opener(self) -> None: + payload = f"echo \"$(bash -c '{self._PUSH}')\" # '" + for cmd in ( + f"# don't\n{payload}", + f"echo hi # don't\n{payload}", + f"cat < None: + from kiro_crew.security.shell_normalizer_span import _span_walk + + # Arithmetic is not modelled, so the ``<<`` in ``((cat < None: + # bash prints ``"<(echo '"`` as text and runs the push after it. + push = f"bash -c '{self._PUSH}'" + for cmd in ( + f'echo "<(echo \'" "$({push})" "\')"', + f'echo ">(echo \'" "$({push})" "\')"', + ): + assert security.is_denied(cmd) is not None, cmd + assert security.is_denied('echo "<(x)" && git push origin feat') is None + + def test_a_run_command_after_a_heredoc_body_in_a_bare_span_is_denied(self) -> None: + # A ``$(...)`` left unproven by a HEREDOC still has a real close, and a + # ``<(bash -c '')`` (or ``$(...)`` / backtick) bash RUNS can sit + # after the body inside it: ``echo $(cat <'))``. The suppressed over-reads scanned that text but never + # emitted the nested opener, so its push was no payload source and the + # floor never ran. The span's proven-close body is emitted non-suppressed + # now, so the opener is reached. ``main`` refuses these; the parent + # commit allowed them. + for inner in ("<(bash -c '{p}')", "$(bash -c '{p}')", "`bash -c '{p}'`"): + launch = inner.format(p=self._PUSH) + for cmd in ( + f"echo $(cat < None: + """An arithmetic ``<<`` before a newline counts as a heredoc, so the outer + ``$(...)`` is unproven and its remainder is over-read; the ``<(...)``, + ``$(...)`` or backtick bash runs after the newline must still be a payload + source. ``main`` refuses these; the feature-push and ``nginx`` twins stay + allowed.""" + for tail in (" ; true", ""): + for inner in ("<(bash -c '{p}')", "$(bash -c '{p}')", "`bash -c '{p}'`"): + launch = inner.format(p=self._PUSH) + cmd = f"echo $(echo $((1<<2))\ncat {launch}){tail}" + assert security.is_denied(cmd) is not None, cmd + assert ( + security.is_denied("echo $(echo $((1<<2))\ncat <(kirocrew restart)) ; true") is not None + ) + # Harmless twins stay at main's verdict. + assert ( + security.is_denied( + "echo $(echo $((1<<2))\ncat <(bash -c 'git push origin feat/x')) ; true" + ) + is None + ) + assert security.is_denied("echo $(echo $((1<<2))\ncat <(nginx -s reload)) ; true") is None + + def test_a_heredoc_quote_after_a_span_that_never_closes_hides_no_opener(self) -> None: + """A span that never closes can hold a REAL heredoc whose body line has an + unbalanced quote. Read as live, that quote would swallow the later + ``<(...)``/``>(...)``, so the never-close remainder is also read quote-blind. + ``main`` refuses the self-restart rows.""" + restart = "kirocrew " + "restart" + for body in ("it's", "$'x\\'y"): + for opener in ("cat <({c})", "echo x > >({c})"): + for outer in ("echo $({x}) ; true", "x=$({x}) ; true", 'echo "$({x})" ; true'): + for c, denied in ((restart, True), ("nginx -s reload", False)): + inner = f"echo $((1<<2))\ncat < None: + """bash shifts at ``$((1<<2))`` and runs the ``rm`` after the span's ``)``. + ``main`` refuses every home/root row; the rm floor reads a span a heredoc + body starts inside as ``main`` does.""" + span = '"$(echo $((1<<2))\n)"' + for lead in ("echo ", "x=", "rm -f "): + for sep in (" ; ", " && ", " | ", "\n"): + for tail in ("rm -r -f $HOME", "rm -r -f ~", "rm -rf ${HOME}", "rm -r -f /"): + cmd = f"{lead}{span}{sep}{tail}" + assert security.is_denied(cmd) is not None, cmd + assert security.is_denied(f"{lead}{span}{sep}rm -r -f /tmp/x") is None + # A span the reader proves keeps main's verdict on the later command. + assert security.is_denied('echo "$(echo $((1<<2)))" ; rm -r -f /tmp/x') is None + + def test_a_self_kill_after_a_span_that_never_closes_is_denied(self) -> None: + """The decoyed ``$k $(...)`` sibling after a ``$((1<<2))`` + newline span is + still walked: the raw self-kill walk reads that span as ``main`` does. + ``main`` refuses these.""" + decoy = "$k $(printf ')' ; pgrep -f kirocrew)" + for span in ("x=$(echo $((1<<2))\n); ", "x=$(cat <(echo $((1<<2))\n)); "): + assert security.is_denied(f"k=kill; {span}{decoy}") is not None + assert security.is_denied(f"k=kill; {span}{decoy.replace('kirocrew', 'nginx')}") is None + + def test_a_heredoc_terminator_glued_to_the_close_ends_the_body(self) -> None: + """A commit message ending ``EOF)"`` holds a literal ``rm`` in + ``confirm``/``format`` and stays allowed, as on ``main``; a home ``rm`` after + it is refused.""" + for q in ("'EOF'", "EOF", '"EOF"'): + msg = f'git commit -m "$(cat <<{q}\nfix: confirm the terminal format\nEOF)"' + assert security.is_denied(msg) is None, msg + assert security.is_denied(msg + " ; rm -rf ~") is not None + assert security.is_denied(f'rm -rf "$(cat <<{q}\nx\nEOF)" ~') is not None + assert security.is_denied(f'rm -rf "$(cat <<{q}\nx\nEOF)" /tmp/x') is None + tab = 'git commit -m "$(cat <<-EOF\n\tfix: confirm\n\tEOF)"' + assert security.is_denied(tab) is None + + def test_a_double_quoted_delimiter_keeps_a_backslash_bash_keeps(self) -> None: + """``"E\\q"`` is the delimiter ``E\\q`` in bash, so a body line ``Eq)"`` does + not close the span. ``main`` refuses every home/root row; the rm floor reads + a span a heredoc body starts inside as ``main`` does.""" + for quote in ('"', '$"'): + for ch in ("q", "z"): + delim = f'{quote}E\\{ch}"' + body = f'"\nE{ch})"\nE\\{ch}\n)' + for home in ("~", "$HOME", "/"): + cmd = f'rm -rf "$(cat <<{delim}\n{body}" {home}' + assert security.is_denied(cmd) is not None, cmd + assert security.is_denied(f'rm -rf "$(cat <<{delim}\n{body}" /tmp/x') is None + # An escapable character still loses its backslash: ``"E\$"`` ends at ``E$``. + assert security.is_denied('rm -rf "$(cat <<"E\\$"\nx\nE$\n)" /tmp/x') is None + assert security.is_denied('rm -rf "$(cat <<"E\\$"\nx\nE$\n)" ~') is not None + + def test_an_apostrophe_in_a_quoted_assignment_prefix_keeps_the_home_rm(self) -> None: + """An apostrophe in a double-quoted assignment prefix (``MSG="it's"``) opens no + quote, so the ``rm`` after it is a command. ``main`` refuses the home rows.""" + for span in ('"$(echo $(( 1<<2\n)))"', '"$(cat < None: + """``<<$'E'`` / ``<<$"E"`` heredoc spans: the ``/tmp`` cleanup stays allowed and + a home operand after the span is refused, as on ``main``.""" + tmp = "rm -rf \"$(cat <<$'E'\n/tmp/scratch-one\nE\n)\" /tmp/scratch-two" + assert security.is_denied(tmp) is None + assert security.is_denied("rm -rf \"$(cat <<$'E'\n/tmp/scratch-one\nE\n)\" ~") is not None + # The locale ``$"E"`` spelling removes its quote the same way. + assert security.is_denied('rm -rf "$(cat <<$"E"\n/tmp/one\nE\n)" /tmp/two') is None + + def test_an_arithmetic_shift_closing_late_in_a_quote_keeps_the_home_rm(self) -> None: + """``rm -r -f "$(echo $((1< str: + return '"$(echo $((' + expr + ")) " + pad + nl + ')"' + + for expr, pad in (("1< None: + """A commit or PR message that QUOTES a shift (``$((1<<2))``) in a heredoc body + stays allowed next to a harmless ``rm``, as on ``main``.""" + nl = "\n" + for body in ( + "fix(security): the rm floor reads $((1<<2)) as a shift", + "don't read $((1<<2)) as a heredoc (it's a shift", + "- the rm floor reads ((x)) and then a << b as a heredoc", + ): + commit = "git commit -m \"$(cat <<'EOF'" + nl + body + nl + "EOF" + nl + ')"' + assert security.is_denied(commit) is None, commit + assert security.is_denied(commit + " && rm -f build.log") is None + assert security.is_denied(commit + " && rm -rf ~") is not None + + def test_an_array_subscript_shift_cannot_hide_a_home_rm(self) -> None: + """bash reads an array subscript (``a[1< None: + """A decoyed ``$k $(...)`` after a ``$((1< None: + """A ``~`` after a span nested past the cap reaches the rm floor: such a span is + read the flat way ``main`` reads it.""" + body = "x" + for _ in range(18): + body = '"$(echo ${x:-' + body + '})"' + assert security.is_denied('rm -rf "$(echo ' + body + ')" ~') is not None + assert security.is_denied("rm -rf \"$(cat <<'E'\nbuild\nE\n)\"") is None + + def test_a_self_kill_inside_a_capped_span_fails_closed(self) -> None: + """Past the nesting cap the span is read in shallow pieces, so the frame + holding ``kill`` and the one holding ``$(pgrep -f kiro${x:-crew})`` part + and neither names the target (Opus). A kill is refused once the span reader + really caps a span; 32+ FLAT openers nest nothing, so those commands keep + main's verdict (Opus, Security Scope on 241829bec2). An ANSI-C spelled + ``kill`` (hex, octal, unicode) reads as ``kill`` too (bolichen97).""" + kill, pg = "ki" + "ll", "pgr" + "ep -f " + ansi = ("$'\\x6b'ill", "$'\\153\\151\\154\\154'", "$'\\u006b'ill", "ki$'\\x6c'l") + for n in (32, 33, 64): + deep = "$(echo " * n + "x" + ")" * n + cmd = f'echo "$(echo {deep} ; {kill} $({pg}kiro${{x:-crew}}))"' + assert security.is_denied(cmd) is not None, n + for k in ansi: + cmd = f'echo "$(echo {deep} ; {k} $({pg}kiro${{x:-crew}}))"' + assert security.is_denied(cmd) is not None, cmd + mixed = "$(echo ${y:-" * 17 + "x" + "})" * 17 + for k in ansi: + cmd = f'echo "$(echo {mixed} ; {k} $({pg}kiro${{x:-crew}}))"' + assert security.is_denied(cmd) is not None, cmd + # A verb written as a parameter default resolves as the frame loop reads a + # program, so it counts as ``kill`` too (Opus on 3e4a4173d1). + defaults = ("${k:-" + kill + "}", "ki${x:-ll}", "k${x:-i}ll", "${k-" + kill + "}") + defaults += ('"${k:-' + kill + '}"', "${k:-ki''ll}", "${k:-$'\\x6b'ill}") + for span in (mixed, "$(echo " * 33 + "x" + ")" * 33): + for k in defaults: + cmd = f'echo "$(echo {span} ; {k} $({pg}kiro${{x:-crew}}))"' + assert security.is_denied(cmd) is not None, cmd + shallow = "$(echo " * 2 + "x" + ")" * 2 + assert security.is_denied(f'echo "$(echo {shallow} ; {kill} $({pg}nginx))"') is None + flat = "".join(f'echo "${{A{i}}}"\n' for i in range(34)) + for cmd in ( + f"""cat > svc.sh <<'EOF'\n{flat}{kill} "${{PID}}"\nEOF""", + "echo $(date); " * 34 + f"timeout --{kill}-after=5 30 ./run.sh", + "echo $(date); " * 34 + f"{kill} $(pgr" + "ep nginx)", + "echo $(date); " * 34 + f'{ansi[0]} "$PID"', + "echo $(date); " * 34 + "${k:-" + kill + '} "$PID"', + "echo $(date); " * 34 + "echo ${k:-" + kill + "ed}", + ): + assert security.is_denied(cmd) is None, cmd + + def test_a_capped_span_body_is_also_read_whole(self) -> None: + """Shallow segments of a capped span part ``t=$(printf token)`` from + ``kirocrew $t`` and stop at the ``)`` in ``${x:-)}``, so the body is also + one flat frame, as ``main`` reads it (Opus, bolichen97 on 6d57c284f2).""" + kc, push = "kiro" + "crew", "bash -c 'git p\"\"ush origin main'" + for n in (32, 33, 64): + deep = "$(echo " * n + "x" + ")" * n + mixed = "$(echo ${y:-" * (n // 2 + 1) + "x" + "})" * (n // 2 + 1) + for span in (deep, mixed): + for cmd in ( + f'echo "$(echo {span} ; t=$(printf token); {kc} $t)"', + f'echo "$({span} ; echo ${{x:-)}} ; {push})"', + f"echo $({span} ; echo ${{x:-)}} ; {push})", + ): + assert security.is_denied(cmd) is not None, cmd + # The same shapes minting nothing, or pushing a feature, stay allowed. + assert ( + security.is_denied(f'echo "$(echo {span} ; t=$(printf status); {kc} $t)"') + is None + ) + feat = push.replace("main", "feat/x") + assert security.is_denied(f"echo $({span} ; echo ${{x:-)}} ; {feat})") is None + + def test_a_never_closing_capped_span_is_read_whole_and_folded(self) -> None: + """A ``${x:-(}`` data paren at the cap leaves the span open, so its body is + also one flat frame there; and ``$\\(`` counts as ``$(`` for the + capped kill check (Opus on 09fc4fb63e). ``main`` refuses every row.""" + kc = "kiro" + "crew" + for n in (32, 33, 48): + for opener in ("$(", "$\\\n("): + span = (opener + "echo ") * n + "${x:-(}" + ")" * n + assert ( + security.is_denied(f'echo "$(echo {span} ; t=$(printf token); {kc} $t)"') + is not None + ) + kill = f'echo "{opener}echo {span} ; kill $(pgrep -f kiro${{x:-crew}}))"' + assert security.is_denied(kill) is not None, kill + # Minting nothing stays allowed. + assert ( + security.is_denied(f'echo "$(echo {span} ; t=$(printf status); {kc} $t)"') + is None + ) + + def test_a_newline_capped_run_is_read_in_linear_time(self) -> None: + """Each capped span's post-newline over-read lies inside the first one's, + so it is emitted once per walk (GPT on 09fc4fb63e: 27.8 s at 20 KB).""" + import time + + def cost(reps: int) -> float: + cmd = "echo " + " ; ".join(["$(echo " * 33 + "x\n" + ")" * 33] * reps) + start = time.perf_counter() + security.is_denied(cmd) + return time.perf_counter() - start + + small, large = cost(9), cost(36) + # Linear is about 4x for 4x the text; the per-span over-read was about 16x. + assert large < 8 * small + 0.5, (small, large) + + def test_a_capped_span_remainder_is_re_extracted_for_a_nested_push(self) -> None: + # A span nested past the cap is read unproven and its whole remainder is + # the only body emitted for it, so the payload walk MUST re-extract that + # remainder: the real ``"$(...)"`` openers after the capped span live + # only inside it. Suppressing re-extraction by the body VALUE (every + # full-remainder suffix) let this push through. + capped = "echo " + "$(echo " * 33 + "x" + ")" * 33 + hidden = 'echo "$(echo "$(bash -c \'git p""ush origin main\')")"' + for cmd in ( + f"{capped} ; {hidden}", + f'{capped} ; echo "$(bash -c \'git p""ush origin main\')"', + ): + assert security.is_denied(cmd) is not None, cmd + # The same shape pushing a feature branch is still allowed. + assert security.is_denied(f"{capped} ; git push origin feat") is None + + def test_a_capped_span_as_an_assignment_value_cannot_mask_a_home_rm(self) -> None: + """A span nested past the cap as a ``VAR="..."`` assignment value feeds a + variable, not the ``rm`` after it. The rm floor reads that span as ``main`` + does, so the real ``rm`` is the command. ``main`` refuses every home row.""" + deep = "$(echo " * 33 + "x" + ")" * 33 + braces = '"$(echo ' * 18 + "${x:-x}" + ')"' * 18 + for span in (deep, braces): + for assign in ('x="$(echo {s})"', 'a=1 b=2 x="$(echo {s})"'): + prefix = assign.format(s=span) + for home in ("~", "$HOME", '"$HOME"', "${HOME}", "~/", "~/*"): + for rm in ("rm -rf", "rm -r -f", "rm --recursive --force"): + cmd = f"{prefix} {rm} {home}" + assert security.is_denied(cmd) is not None, cmd + # The capped span as an rm OPERAND keeps its ``~``. + assert security.is_denied(f'rm -rf "{deep}" ~') is not None + # A capped assignment value with no rm after it is allowed, as on main. + assert security.is_denied(f'x="$(echo {deep})"') is None + assert security.is_denied(f'x="$(echo {deep})" echo hi') is None + assert security.is_denied("rm -rf \"$(cat <<'E'\nbuild\nE\n)\"") is None + # Text before the ``NAME="`` (a separator, an earlier assignment, a + # redirection, a partial value) keeps the real ``rm`` the command; each row + # bash-wipes ``$HOME``. + for prefix in ( + 'true; x="$(echo {s})"', # ``;``-cut leaves a leading space + 'true && x="$(echo {s})"', + 'true | x="$(echo {s})"', + ' x="$(echo {s})"', # a bare leading space + 'a="p q" x="$(echo {s})"', # an earlier assignment with a quoted space + '2>/dev/null x="$(echo {s})"', # a leading redirection runs first + '2> /dev/null x="$(echo {s})"', # ... with a separate target word + ' None: + """A span nested past the cap as a LEADING REDIRECTION TARGET (``>"$()" + rm -fr ~``) cannot hide the ``rm`` after it. ``main`` refuses every home row.""" + deep = "$(echo " * 33 + "x" + ")" * 33 + for redir in ( + '>"{s}"', + '> "{s}"', + '2>"{s}"', + '<"{s}"', + 'x=1 >"{s}"', + '>"{s}" x=1', + 'true ; >"{s}"', + ): + prefix = redir.format(s=deep) + for home in ("~", "$HOME", "${HOME}", "~/", "~/*", "/"): + for rm in ("rm -fr", "rm -rf", "rm --recursive --force"): + cmd = f"{prefix} {rm} {home}" + assert security.is_denied(cmd) is not None, cmd + # The capped span as an rm OPERAND keeps its ``~``, and a redirection + # with no rm stays allowed. + assert security.is_denied(f'rm -fr "{deep}" ~') is not None + assert security.is_denied(f'>"{deep}" echo hi') is None + + def test_a_heredoc_in_a_quoted_span_cannot_mask_a_home_rm(self) -> None: + """A heredoc body holding an UNTERMINATED backtick inside ``"$(...)"``: the home + operand after the span is refused and the ``/tmp`` twin allowed, as on + ``main``.""" + # `rm -rf "$(cat < None: + """A ``${`` / ``${x`` / ``${x:-`` on a heredoc BODY line is stdin data. ``main`` + refuses every home/root row; the rm floor reads a span a heredoc body starts + inside as ``main`` does. The ``/tmp`` twin stays allowed.""" + for opener in ('"$(cat < None: + """A backtick (with a LATER backtick), a ``$(`` / ``<(`` / ``>(`` opener and a + bare ``(`` on a heredoc BODY line, direct or nested one level inside a quoted + substitution. ``main`` refuses every home/root row; the rm floor reads a span + a heredoc body starts inside as ``main`` does. Every ``/tmp`` twin stays + allowed.""" + homes = ("~", "$HOME", '"$HOME"', "${HOME}", "~/", "/") + for tag in ("E", "'E'"): + # backtick with a LATER backtick clause (direct), and every opener / + # bare ``(`` NESTED one level inside a quoted substitution. + direct_bt = [(f'"$(cat <<{tag}\n`\nE\n)"', "; echo '`'")] + nested = [ + (f'"$(echo "$(cat <<{tag}\n{body}\nE\n)")"', tail) + for body in ("`", "(", "$(", "<(", ">(") + for tail in ("", "; echo '`'") + ] + for span, tail in direct_bt + nested: + for home in homes: + cmd = f"rm -rf {span} {home}{tail}" + assert security.is_denied(cmd) is not None, cmd + # /tmp twin stays at main's verdict (allowed). + assert security.is_denied(f"rm -rf {span} /tmp/x{tail}") is None, (span, tail) + + def test_a_single_quoted_backtick_is_literal_not_an_opener(self) -> None: + # bash reads a backtick inside a single quote as DATA (``echo '`'``), so + # the span matcher must not open a bogus backtick span whose body then + # carries that ``'`` over the REAL ``$(`` after it. + push = "bash -c 'git p\"\"ush origin main'" + for cmd in ( + f"echo '`' \"$({push})\" '`'", + f"echo '`' \"$(echo \"$({push})\")\" '`'", + f"echo 'a`b' \"$({push})\" 'c`d'", + f"echo $'`' \"$({push})\" $'`'", + f"echo '`' <({push}) '`'", + ): + assert security.is_denied(cmd) is not None, cmd + # A real (unquoted or double-quoted) backtick substitution still opens. + assert security.is_denied(f"echo `{push}`") is not None + assert security.is_denied(f'echo "`{push}`"') is not None + # A backtick beside a quote leaves the flat quote reading untrusted, so a + # single-quoted backtick with a ``$(`` after it fails closed even before a + # feature push; a real backtick with a quote beside it stays allowed. + assert security.is_denied("echo '`' \"$(date)\" '`' && git push origin feat") is not None + assert security.is_denied("echo `date` 'x' && git push origin feat") is None + assert security.is_denied('git commit -m "`date` don\'t"; git push origin feat') is None + + def test_a_quote_inside_a_backtick_body_cannot_hide_a_push(self) -> None: + """bash closes a backtick at the next backtick whatever the quotes, so in + ``echo `: '` $() `: '` `` the ``'`` quotes nothing and the ``$(`` + runs. With a backtick present the flat quote reading is not trusted, so + the opener's remainder is scanned (Opus, bolichen97 on f14b41d53c).""" + for push in ('bash -c "git p""ush origin main"', "bash -c 'git p\"\"ush origin main'"): + for cmd in ( + f"echo `: '` $({push}) `: '`", + f"echo `echo '` $({push}) `'`", + f"x=`: '` $({push}) `'`", + f"echo `: $'` $({push}) `'`", + ): + assert security.is_denied(cmd) is not None, cmd + force = 'bash -c "git p""ush origin --force main"' + assert security.is_denied(f"echo `echo '` $({force}) `'`") is not None + # The bogus single-quoted opener is still stepped past with a backtick around. + hidden = "echo '$(' \"$(echo ')'; bash -c 'git p\"\"ush origin main')\"" + for tail in (" `true`", "; echo `date`"): + assert security.is_denied(hidden + tail) is not None, tail + + def test_a_bare_brace_in_a_default_is_data(self) -> None: + from kiro_crew.security.shell_normalizer_span import _matching_close_brace + + # bash closes ``${x:-a{b}c}`` at the first ``}`` + assert _matching_close_brace("x:-a{b}c}")[:2] == (len("x:-a{b}"), True) + assert security.is_denied('rm -rf "$(echo ${x:-{})" ~') is not None + + def test_a_heredoc_span_still_reaches_the_mint_and_kill_checks(self) -> None: + for cmd in ( + "T=$(cat < None: + """``$((cat <'`` glued after the heredoc. Arithmetic is not + modelled, so ``<<`` is a heredoc and the span is unproven, which scans + the whole remainder and finds the push.""" + cmd = "echo \"$((cat < None: + """The new backtick unit-skip used the QUOTE-AWARE closer, which reads a + single-quoted backtick as data and swallows the rest of the span -- so + ``rm_floor`` lost the ``~`` after an unproven ``$(`` and allowed a home + wipe. The skip now uses the escape-only ``_backtick_closer`` (a single + quote does not protect a backtick), so the span resumes at the real + closer and the ``~`` reaches the ``rm`` segment.""" + cmd = 'rm -rf "$(echo "`echo \'`")" ~' + assert security.is_denied(cmd) is not None, cmd + + def test_a_quoted_process_sub_in_a_default_is_data(self) -> None: + """Inside ``${x:-"<("}`` the ``<(`` is double-quoted, so bash reads it as + text. The brace walk opened a process sub there that never closed, so + the span ran to the end and the rm floor lost the ``~``.""" + cmd = 'rm -rf "$(echo ${x:-"<("})" ~' + assert security.is_denied(cmd) is not None, cmd + + def test_a_pid_dollar_opens_no_expansion(self) -> None: + """``$$`` is the PID, so the ``{`` after it is text and the span closes + at the ``)``. Reading the second ``$`` as a ``${`` opener ran the span to + the end, and the rm floor lost the ``~``. A ``\\$`` is data and pairs + with nothing, so ``\\$${`` still opens one.""" + cmd = 'rm -rf "$(echo $${)" ~' + assert security.is_denied(cmd) is not None, cmd + + from kiro_crew.security.shell_normalizer_span import _span_walk + + # After a paired ``$`` the ``{`` is data and the ``)`` proves the span; + # after a lone one ``${)`` never closes, so the span stays unproven. + assert [_span_walk(f"echo {t}{{)")[1] for t in ("$", "$$", "$$$", "\\$$", "\\$$$")] == [ + False, + True, + False, + False, + True, + ] + + def test_an_arithmetic_seed_that_closes_a_subshell_first_is_dropped(self) -> None: + """In ``$((a);(cat < None: + """bash reads each span in a fresh context, so the ``'`` in + ``"$(echo "'")"`` quotes nothing after its ``)``. A flat quote walk + carried it out and marked the next, real ``$(`` as single-quoted.""" + cmd = 'echo "$(echo "\'")" "$(git p""ush origin main)"' + assert security.is_denied(cmd) is not None, cmd + + def test_a_capped_span_still_scans_past_its_heredoc(self) -> None: + """A heredoc body starting inside a span nested past the cap leaves the + outer span unproven, so its whole remainder -- the command after the + heredoc's body glued to its ``)`` -- is scanned.""" + deep = "$(echo " * 32 + "x" + ")" * 32 + for push in ("git push", self._PUSH): + cmd = f"echo \"$(echo {deep}; cat < None: + """A span past the cap reads its deep interior WITHOUT fresh contexts, so + a ``)`` that is only DATA there -- a ``${x:-(}`` default value's paren -- + is counted as a close and consumes one of the span's own ``)``. The span + then never closes in the gate's reading, though bash balances it and runs + the opener AFTER it as a sibling. The whole-remainder body is a SUFFIX the + payload walk value-suppresses, so a ``$(bash -c '')`` hidden behind + the span was never a payload source and the git-publish floor never ran: + ``is_denied`` returned None for a command bash pushes ``main`` with (Opus, + head a22b3355b6). The never-close branch now scans the remainder once, + emitting each later substitution segment as its own shallow entry, so the + hidden launcher is read. Every row is a protected push bash actually runs + (verified with a stub git); the nested-bash ``$(...)``, the backtick form + and the plain-eval form each fail (allowed) on a22b3355b6 and pass now.""" + prefix = "echo " + "$(echo " * 33 + "${x:-(}" + ")" * 33 + " ; " + launchers = ("bash -c", "sh -c", "eval") + targets = ("main", "mainline") + for sep in (" ; ", " && ", "\n"): + pfx = "echo " + "$(echo " * 33 + "${x:-(}" + ")" * 33 + sep + for launcher in launchers: + for target in targets: + push = f"git push origin {target}" + for sub in ( + f"echo \"$({launcher} '{push}')\"", # launcher in a $() sub + f"echo \"`{launcher} '{push}'`\"", # launcher in a backtick sub + f"$({launcher} '{push}')", # bare $() sub + f"`{launcher} '{push}'`", # bare backtick sub + ): + cmd = pfx + sub + assert security.is_denied(cmd) is not None, cmd + # A direct (launcher-less) push behind the span is caught by the raw + # floor already; it must stay denied. + assert security.is_denied(prefix + 'echo "$(git push origin main)"') is not None + # A never-close span hiding a HARMLESS later command must NOT be refused + # -- the tail scan adds findings, it does not deny on its own. + for benign in ("echo \"$(bash -c 'echo hi')\"", 'echo "`date`"', 'echo "$(ls -l)"'): + assert security.is_denied(prefix + benign) is None, benign + # The span's body is also one flat frame, as ``main`` reads it, so a feature + # launcher inside a quoted substitution after it is refused, as on ``main``. + # A bare feature push, or a bare launcher sub, stays allowed. + assert security.is_denied(prefix + "echo \"$(bash -c 'git push origin feat')\"") is not None + for feat in ("git push origin feat", "$(bash -c 'git push origin feat')"): + assert security.is_denied(prefix + feat) is None, feat + + def test_a_capped_span_closing_late_cannot_hide_a_process_substitution(self) -> None: + """The same data ``)`` can make the capped span close LATE instead of + never: at the ``)`` of an unquoted PROCESS substitution bash runs as a + sibling after its own close. The span then reads as closed, so the walk + skipped past it and the ``<(bash -c '')`` body was never a payload + source; the ``$(bash`` spelling was caught only because a ``$(bash`` + token reads as a launcher, which ``<(bash`` does not (bolichen97, head + d42c8f4354). The late-closing span is now scanned once for its + substitution segments, as the never-close span is. Every row is a + protected push bash runs (stub git); each was allowed on d42c8f4354.""" + for sep in (" ; ", " && "): + prefix = "echo " + "$(echo " * 33 + "${x:-(}" + ")" * 33 + sep + for target in ("main", "mainline"): + push = f"git push origin {target}" + for sub in ( + f"cat <(bash -c '{push}')", + f"cat <(sh -c '{push}')", + f"cat <(eval '{push}')", + f"echo x > >(bash -c '{push}')", + f"echo x > >(sh -c '{push}')", + ): + assert security.is_denied(prefix + sub) is not None, prefix + sub + # A harmless or FEATURE process substitution behind it stays allowed. + for allowed in ("cat <(echo done)", "cat <(bash -c 'git push origin my-feature')"): + assert security.is_denied(prefix + allowed) is None, prefix + allowed + + def test_a_capped_span_inside_an_outer_body_cannot_hide_its_launcher(self) -> None: + """A capped span nested inside an outer ``"$(...)"`` makes the OUTER span + capped too. Its segment scan emitted only the text inside the NESTED + openers, so the outer body's own ``; bash -c ''`` was never a + payload source, and at the root the double-quoted word stays one token. + The scan starts inside the outer span, so its own text is a segment. + Every row is a protected push bash runs (stub git); unquoted, the same + shape was already refused.""" + nest = "$(" * 33 + ":" + ")" * 33 + for opener in ('echo "$(', 'x="$('): + for sep in (" ; ", " && "): + for launcher in ("bash -c", "sh -c", "eval"): + for target in ("main", "mainline"): + cmd = f"{opener}{nest}{sep}{launcher} 'git push origin {target}')\"" + assert security.is_denied(cmd) is not None, cmd + # The same shape with a FEATURE push or a harmless tail stays allowed. + for tail in ("bash -c 'git push origin my-feature'", "echo done"): + cmd = f'{opener}{nest}{sep}{tail})"' + assert security.is_denied(cmd) is None, cmd + + def test_a_capped_span_closing_early_cannot_hide_its_launcher(self) -> None: + """The data ``)`` in a ``${x:-)}`` default, read past the nesting cap, + closes the capped span EARLY instead of late: one ``)`` too soon, so the + span's own ``; bash -c ''`` before bash's real close sat after the + guessed close, where no opener reaches it, and at the root the + double-quoted word stays one token. The text past a capped span's close + is scanned once for its segments, every ``)`` there a boundary that + closes nothing, so the launcher is read however early the guess was. + Most of these rows push a protected branch when bash runs them (stub + git); the rest are over-refused, the fail-closed direction.""" + for data in ("${x:-)}", "${x:-))}", "`echo )`"): + for nest_open, nest_close in (("$(echo ", ")"), ('"$(echo ', ')"'), ("<(echo ", ")")): + early = nest_open * 33 + data + nest_close * 33 + for opener, close in (('echo "$(', ')"'), ("echo $(", ")"), ("cat <(", ")")): + for sep in (" ; ", " && ", "\n"): + for target in ("main", "mainline"): + push = f"git push origin {target}" + for tail in (f"bash -c '{push}'", f"eval '{push}'", push): + cmd = f"{opener}{early}{sep}{tail}{close}" + assert security.is_denied(cmd) is not None, cmd + # A harmless tail behind the same early close stays allowed. + cmd = f"{opener}{early} ; echo done{close}" + assert security.is_denied(cmd) is None, cmd + + def test_a_proven_body_equal_to_a_source_suffix_is_re_extracted(self) -> None: + """``echo $(x <(...) \\') 'x <(...) \\'`` has a PROVEN ``$(`` body whose + text equals the end of its source. Suppression by VALUE (any suffix body) + queued that body without extracting its substitutions, so its + ``<(bash -c '')`` was never read, and the later copy was a + duplicate of a payload already seen. Only an over-read entry (an + unproven span's remainder or tails) is suppressible; a proven body is + always re-extracted. Every row is a protected push bash runs.""" + for spell in ("git push", 'git p""ush'): + for sub in ("<", ">"): + for launcher in ("bash -c", "sh -c", "eval"): + for target in ("main", "mainline"): + inner = f"{sub}({launcher} '{spell} origin {target}')" + cmd = f"echo $(x {inner} \\') 'x {inner} \\'" + assert security.is_denied(cmd) is not None, cmd + # The FEATURE push twin stays allowed. + inner = f"{sub}({launcher} '{spell} origin my-feature')" + assert security.is_denied(f"echo $(x {inner} \\') 'x {inner} \\'") is None + assert security.is_denied("echo $(x <(echo done) \\') 'x <(echo done) \\'") is None + + def test_a_quoted_or_escaped_paren_cannot_end_the_capped_tail_early(self) -> None: + """The capped tail scan ended its span at the FIRST ``)`` it read other + than a single-quoted one, so a double-quoted ``)`` that is data + (``echo ")"``) or an escaped ``\\)`` closed the span early and the + ``; bash -c ''`` behind it was never a segment. bash keeps the + span open there -- a double-quoted ``)`` whose opener was not itself + double-quoted, and an escaped ``)``, are both literal -- and closes it + at the real ``)``, so the launcher runs as a sibling. The scan now + treats a ``)`` as a boundary only when it is unquoted-and-unescaped, or + a double-quoted ``)`` closing a span whose opener was double-quoted. + Every protected row is a push bash runs (stub git).""" + nest = "$(" * 33 + ":" + ")" * 33 + decoys = (' ; echo ")" ; ', " ; echo \\) ; ") + for opener in ('echo "$(', 'x="$('): + for decoy in decoys: + for launcher in ("bash -c", "sh -c", "eval"): + for target in ("main", "mainline"): + cmd = f"{opener}{nest}{decoy}{launcher} 'git push origin {target}')\"" + assert security.is_denied(cmd) is not None, cmd + # A FEATURE push or a harmless tail behind the same decoy stays allowed. + for tail in ("bash -c 'git push origin my-feature'", "echo done"): + cmd = f'{opener}{nest}{decoy}{tail})"' + assert security.is_denied(cmd) is None, cmd + # A genuine double-quoted ``)`` that DOES close an inner double-quoted + # ``$(`` is still a boundary, so a push after that real close is seen. + inner = "\"$(bash -c 'git push origin main')\"" + assert security.is_denied(f'echo "$({nest} ; echo {inner})"') is not None diff --git a/test/test_push_branch_gate_parser_heredoc.py b/test/test_push_branch_gate_parser_heredoc.py new file mode 100644 index 00000000000..f039db80a3a --- /dev/null +++ b/test/test_push_branch_gate_parser_heredoc.py @@ -0,0 +1,142 @@ +"""Heredoc bodies inside a substitution span, pinned against the rm floor. + +Split out of ``test_push_branch_gate_parser.py`` (2,000-line cap). Each test here +is a way bash ends -- or does not end -- a heredoc body inside ``$(...)``: an +unquoted body's ``\\``-newline fold, and a body line that starts with a +delimiter and carries the span's ``)``. The span reader does not parse heredocs; +a span a heredoc body starts inside is read as ``main`` reads it, so every +home/root row is refused as on ``main`` and the ``/tmp`` twins stay allowed. +""" + +from kiro_crew import security + + +class TestAHeredocBodyCannotMaskAHomeRm: + """A heredoc body inside a substitution cannot hide the ``rm`` operand after + the span.""" + + def test_an_unquoted_heredoc_line_continuation_cannot_mask_a_home_rm(self) -> None: + """An unquoted heredoc body line ending in a backslash joins the next line + before bash matches the terminator. ``main`` refuses every home/root row; the + rm floor reads a span a heredoc body starts inside as ``main`` does. The + quoted-delimiter ``/tmp`` twins stay allowed.""" + for home in ("~", "$HOME", "${HOME}", "/"): + cmd = "rm -r -f \"$(cat < None: + """A body line ``E)`` glued to the span's ``)`` that ends in a continuation + backslash: bash closes at that physical ``)``. ``main`` refuses every + home/root row; the rm floor reads a span a heredoc body starts inside as + ``main`` does.""" + nl = "\n" + for home in ("~", "$HOME", "${HOME}", "/"): + for mid in ("", "a", "abc", "a\\" + nl + "bc"): + cmd = ( + 'rm -r -f "$(cat < None: + """A body line ``E\\`` then ``)`` folds to ``E)`` and closes the substitution + (bash 5.2 runs ``rm -r -f "" ~``). ``main`` refuses every home/root row; the + rm floor reads a span a heredoc body starts inside as ``main`` does. The + ``/tmp`` twin stays allowed.""" + nl = "\n" + tail = ' ; : "' + nl + "E" + nl + ')"' + for home in ("~", "$HOME", "${HOME}", "/"): + for body in ("E\\" + nl + ")", "E\\" + nl + "\\" + nl + ")"): + cmd = 'rm -r -f "$(cat < None: + """Inside ``$(...)`` bash ends a heredoc body at a line that STARTS with the + delimiter and carries the span's ``)`` (``E )``, ``Ex)``, ``EOFX)``). + ``main`` refuses every home/root row; the rm floor reads a span a heredoc + body starts inside as ``main`` does. The ``/tmp`` twins stay allowed.""" + nl = "\n" + tail = ' ; : "' + nl + "E" + nl + ')"' + etail = ' ; : "' + nl + "EOF" + nl + ')"' + for home in ("~", "$HOME", "${HOME}", "/"): + # delimiter word, then separated text, then the span's ``)`` + for last in ("E )", "E\t)", "Ex)", "E foo)"): + cmd = 'rm -r -f "$(cat < None: + """With several heredocs pending (``cat < str: #: bare-secret pass: the closed ``_ELB_TARGET_GROUP_ARN_RE`` shape, its two span #: helpers, the lazy lookup in pass 3 that honours ``judged_sri_values``, and their #: comments; the exemption only narrows what pass 3 redacts. -_PACKAGE_LINE_BUDGET = 32_970 +#: +#: Raised again, from 32,970, so the paren matcher skips a NESTED substitution +#: and a ``${...}`` expansion (``_matching_close_brace``) as units, keeping their +#: inner quotes and parens sealed inside them instead of closing an enclosing +#: span early at a quoted ``")"`` and leaving a ``bash -c ''`` behind +#: it unscanned. A ``$`` paired into ``$$`` opens nothing, a quoted ``<(`` is +#: data, a backtick closes only on an unescaped backtick, and nesting past +#: ``_MAX_SPAN_NESTING`` is read without unit skipping. Heredocs and arithmetic +#: are not modelled: a span a heredoc body starts inside (any ``<<`` but a +#: here-string, a shift included) is read the flat way ``main`` reads it +#: (``_flat_close``) and is unproven, so the extractor scans its whole remainder. +#: The rm floor and the self-kill raw walk take the unit-skipping close only +#: when it proves the span, else ``main``'s. The extractor emits an unproven +#: span's over-reads once and continues the structural walk, and the payload +#: walk does not re-extract an over-read, so a run of such spans stays linear; a +#: capped span is scanned in shallow segments (``_capped_tail_entries``), and +#: a kill behind a capped span fails closed (its verb read as a frame token, so +#: a parameter-default spelling counts). A backtick in the text leaves the flat +#: quote reading untrusted. A capped span's body is also read whole, once per +#: walk, whether it closes or not, and its post-newline over-read runs once per +#: walk too. The capped kill check counts openers with line continuations folded. +#: The span matcher and the +#: substitution bodies moved to ``shell_normalizer_span``, keeping +#: ``shell_normalizer_walk`` under the split line. The ``argv_floor`` facade +#: manifest names the frame reader it forwards. +_PACKAGE_LINE_BUDGET = 33_685 #: Ceiling on any ONE file in the package. This is what the bound is really for -- #: a package total says nothing about a single file growing back into a second @@ -726,6 +754,43 @@ def _gate_seconds(command: str) -> float: return time.perf_counter() - started +def _span_walk_steps(monkeypatch, command: str) -> int: + """Characters the shell quote machine steps through while ``is_denied`` + reads an allowed *command*. The substitution span walk runs there, and + ``is_sensitive_bash_command`` never reaches it. A step count is the walk's + WORK, so a loaded CI worker cannot move it the way it moves wall time.""" + real = shell_normalizer_walk._iter_shell_chars + steps = 0 + + def counting(*args, **kwargs): + nonlocal steps + for step in real(*args, **kwargs): + steps += 1 + yield step + + monkeypatch.setattr(shell_normalizer_walk, "_iter_shell_chars", counting) + monkeypatch.setattr(shell_normalizer_span, "_iter_shell_chars", counting) + assert security.is_denied(command) is None + monkeypatch.setattr(shell_normalizer_walk, "_iter_shell_chars", real) + monkeypatch.setattr(shell_normalizer_span, "_iter_shell_chars", real) + return steps + + +#: On every span-walk shape below, this build and main both do at most +#: quadratic work in the opener count: doubling the run multiplies the steps by +#: about 3 to 4. Each super-quadratic regression the review found (a second walk +#: per failed ``$((`` seed, a rescan from every opener) measured 6x or more per +#: doubling, so the work may grow at most this much when the run doubles. +_MAX_STEP_GROWTH = 4.5 + + +def _assert_walk_scales(monkeypatch, shape, n: int = 40) -> None: + """Doubling the run of *shape* grows the walk's work at most quadratically.""" + small = _span_walk_steps(monkeypatch, shape(n)) + large = _span_walk_steps(monkeypatch, shape(2 * n)) + assert large <= _MAX_STEP_GROWTH * small, (small, large) + + def test_double_separator_10kb_is_fast() -> None: """The crash shape, at the crash size: 15 s on the shipped build.""" cmd = _double_separator_command(160) @@ -739,3 +804,171 @@ def test_url_payload_12kb_is_fast() -> None: assert 10_000 < len(cmd) <= MAX_SCANNABLE_COMMAND_CHARS assert is_sensitive_bash_command(cmd) is None assert _gate_seconds(cmd) < 2.0 + + +# A run of substitution OPENERS is the cost shape the nested-payload extractor +# faces: each ``$(`` is a span the walk must delimit, and before the extractor +# read one body per top-level opener (skipping proven and depth-capped spans +# instead of re-walking from every inner opener) a 400-byte command of these +# took tens of seconds. Each row pins NON-explosion as a bound on the walk's +# growth through ``is_denied``, where the span walk runs; main meets it too. + + +def test_a_run_of_substitution_openers_is_fast(monkeypatch) -> None: + """``echo $($($(...`` -- balanced openers, every span proven and skipped past.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$(" * n) + + +def test_a_run_of_unbalanced_quoted_openers_is_fast(monkeypatch) -> None: + """``echo "" $( "" $( ...`` -- openers inside a double quote that never close.""" + _assert_walk_scales(monkeypatch, lambda n: 'echo "' + '"$(' * n) + + +def test_alternating_single_and_double_quoted_openers_is_fast(monkeypatch) -> None: + """``echo '$(' "$( '$(' "$( ...`` -- a single-quoted opener (data to bash, + stepped past) alternating with a live one, the shape the single-quote check + guards.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "'$(' \"$(" * n) + + +def test_deeply_nested_substitutions_are_fast(monkeypatch) -> None: + """``echo $(echo $(echo ... )))`` -- past ``_MAX_SPAN_NESTING`` the outer span + is unproven only because of the cap, so the extractor takes the + whole-remainder body and lets the payload walk descend, rather than adding the + re-exploding last-``)`` and outer-context readings.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$(echo " * n + ")" * n) + + +# A run of ``$((`` openers is the ARITHMETIC-seed cost shape. Each ``$((`` is a +# subshell-or-arithmetic the span walk must settle, and before the adjacency was +# decided inside that one walk a separate forward scan ran per opener and per +# recursed body, so a few hundred bytes of these took tens of seconds. These +# three are bolichen97's shapes; each pins NON-explosion, and main meets it too. + + +def test_a_run_of_arithmetic_openers_is_fast(monkeypatch) -> None: + """``echo $(($(($((...`` -- bare ``$((`` openers that never close, the seed + cost shape where the old per-opener forward scan was super-linear.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$((" * n) + + +def test_a_run_of_closing_arithmetic_openers_is_fast(monkeypatch) -> None: + """``echo $((x+$((x+...1))))...`` -- ``$((`` openers that DO close, each with + an adjacent ``))`` that confirms the optimistic seed.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$((x+" * n + "1" + "))" * n) + + +def test_a_run_of_arithmetic_openers_with_bodies_is_fast(monkeypatch) -> None: + """``echo $((echo $((echo ...`` -- ``$((`` openers each carrying a word, the + heaviest of the three because every opener recurses a populated body.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$((echo " * n) + + +# A ``$((`` seed that fails is settled in the SAME walk: the span is read with +# and without the seed at once, so a nested span is walked once. A second walk +# per failed seed re-walked every level under it again, so these two shapes, +# each ``$((`` holding a ``<<`` and never closing on ``))``, grew about 6x per +# doubling. Both are bolichen97's shapes, and main meets the bound too. + + +def test_a_run_of_arithmetic_openers_holding_a_shift_is_fast(monkeypatch) -> None: + """``echo $((x<<$((x<<...`` -- every seed holds a ``<<`` and fails.""" + _assert_walk_scales(monkeypatch, lambda n: "echo " + "$((x<<" * n) + + +def test_a_run_of_subshell_openers_holding_a_heredoc_is_fast(monkeypatch) -> None: + """``echo $((cat < None: + """``echo "$(cat < None: + """``echo `` + N x (``$(echo `` x33 + ``x`` + ``)`` x33) joined by `` ; `` -- + the capped-span run that re-sliced the whole remainder at every opener.""" + nest = "$(echo " * 33 + "x" + ")" * 33 + shape = lambda n: "echo " + " ; ".join(nest for _ in range(n)) # noqa: E731 + small = _span_walk_steps(monkeypatch, shape(40)) + large = _span_walk_steps(monkeypatch, shape(80)) + assert large <= _MAX_LINEAR_STEP_GROWTH * small, (small, large) + + +# A run of DOUBLY-nested unproven heredoc spans -- ``"$(echo "$(cat <\nE +# \n)")" `` x N, each inner ``$(cat < None: + """``"$(echo "$(cat <\\nE\\n)")" `` x N -- the c104v double-nest shape.""" + for c in ("`", "(", "$(", "<("): + unit = f'"$(echo "$(cat < None: + """``kill `` + N x ``$(echo $((1<<2))\\n) `` (and the ``<(`` twin).""" + real = shell_normalizer._self_tokens + chars = 0 + + def counting(text, *args, **kwargs): + nonlocal chars + chars += len(text) + return real(text, *args, **kwargs) + + monkeypatch.setattr(shell_normalizer, "_self_tokens", counting) + for unit in ("$(echo $((1<<2))\n) ", "<(echo $((1<<2))\n) "): + sizes = [] + for n in (40, 80): + chars = 0 + assert security.is_denied("ki" + "ll " + unit * n) is None + sizes.append(chars) + assert sizes[1] <= _MAX_LINEAR_STEP_GROWTH * sizes[0], (unit, sizes) diff --git a/test/test_security_shell_grammar.py b/test/test_security_shell_grammar.py index 5666ba1feff..e9416f56d2c 100644 --- a/test/test_security_shell_grammar.py +++ b/test/test_security_shell_grammar.py @@ -148,7 +148,7 @@ def test_degenerate_input_does_not_raise(self, text: str) -> None: def test_an_unproven_span_still_yields_the_whole_remainder(self) -> None: """Fail-CLOSED direction: a body that reaches too far is only over-scanned.""" - (body,) = security._substitution_bodies(f"$(case x in x) printf {self.VERB}") + body, *_ = security._substitution_bodies(f"$(case x in x) printf {self.VERB}") assert f"printf {self.VERB}" in body, body @@ -204,7 +204,7 @@ def test_case_as_data_no_longer_over_arms(self, command: str, body: str) -> None ], ) def test_every_bash_spanning_form_still_arms(self, command: str) -> None: - (body,) = security._substitution_bodies(command) + body, *_ = security._substitution_bodies(command) assert "pgrep -f kirocrew" in body, body assert security._is_self_kill(command) @@ -222,7 +222,7 @@ def test_esac_after_in_stays_safe_but_unproven(self) -> None: after the word in') disarms on ``echo in esac`` inside a clause body, which is the BYPASS direction. Pinned to the fail-closed fallback: unproven span, whole remainder, extractors scan more.""" - (body,) = security._substitution_bodies("$(case z in esac; echo after)") + body, *_ = security._substitution_bodies("$(case z in esac; echo after)") assert "echo after" in body, body def test_option_words_are_transparent_in_the_chain(self) -> None: @@ -241,7 +241,7 @@ def test_option_words_are_transparent_in_the_chain(self) -> None: "$(time -p case x in x) : ;; esac; pgrep -f kirocrew)", "$(time -- case x in x) : ;; esac; pgrep -f kirocrew)", ): - assert _matching_close_paren(text, 2) == (len(text), True), text + assert _matching_close_paren(text[2:]) == (len(text) - 2, True), text assert security._substitution_bodies("echo $(echo -n case x in y) tail") == [ "echo -n case x in y" ] @@ -258,7 +258,7 @@ def test_heredoc_bodies_extract_whole_and_the_consumer_convicts(self) -> None: "kill $(echo < "list[str]": break if step.state == 2 and text.startswith("$(", step.offset): body = text[step.offset + 2 :] - close, proven = _shell._matching_close_paren(body, 0) + close, proven = _shell._matching_close_paren(body) out.extend(_oracle_segments(body[: close - 1 if proven else close], track)) seg.append(SUBST) in_redirect = False