Skip to content

Source spans for directive arguments, and the regions lex() blanks #32

Description

@dvejsada

Summary

lex() knows where every argument of a directive starts and ends, and which regions it treated as literal text. Its results keep neither, so tools that edit or render directive source re-lex it themselves.

  1. Argument spans. Directive has start/end for the whole directive, and decoded positional/params, but no span for each argument as written. ForLegalAI/PactTrack needs them in two places:
    • Renaming an identifier (a section or definition id) in every {{ref:}}/{{term:}} means splicing a new positional value. _positional_end in app/routes/pages.py re-scans the source to find it.
    • Assembly §15.7.2 step 5 keeps a placeholder's note "exactly as written". _argument_spans in app/services/assembly.py is a second argument lexer, copying _lex_arguments/_lex_value.
  2. Literal regions. lex() blanks fenced code, code spans and HTML comments in Lexed.view but doesn't return where they are. PactTrack's export_inline._literal_regions/_fenced_code rebuild them by diffing text against view and re-matching backtick runs, to render those regions verbatim.

Proposal

  • Directive.positional_span: tuple[int, int] | None and Directive.param_spans: dict[str, tuple[int, int]]. The spans cover the value as written, quotes included, as offsets into the lexed text.
  • Lexed.literals: list[tuple[str, int, int]], holding (kind, start, end) with kind being fence, code, comment or escape.
  • Export closes_fence publicly next to FENCE_OPEN_RE/fence_end. PactTrack's closing-fence checks duplicate it.

Related: #27 (source positions for sections, blocks and diagnostics) and #28 (comment pairing in lex()).

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions