[Velli20]

Fixes #590.

Summary

• Lexer: Type 4 programs are now lexed from the raw bytes. Previously the braces were padded with spaces on a lossy UTF-8 copy, which was then split on Unicode whitespace.
  • The lexer skips PDF whitespace (NUL, HT, LF, FF, CR, SP) and % comments.
  • { and } are their own tokens, so no spaces are needed around braces.
  • The other PostScript delimiters (( ) < > [ ] /) are rejected with UnexpectedDelimiter.
• Parser: parser::parse(&[u8]) replaces parse_tokens(&[&str]).
  • It checks that the program is one balanced { … } procedure and that nested procedures only appear as operands of if/ifelse.
  • Nesting is tracked on an explicit stack, so deeply nested braces can't overflow the native stack.
  • New errors: MissingProgramProcedure, UnbalancedBraces, UnexpectedProcedure, UnexpectedDelimiter. The unused EmptyBlockStack and TokenIndexOverflow are removed.
• Literals: numbers are parsed straight from bytes (TryFrom<&[u8]> on Value and Operator).
  • Radix forms (16#FF) are read as unsigned 32-bit values, so 16#FFFFFFFF is -1.
  • Exponent forms (1e2, .5) parse as reals.
  • An integer outside the 32-bit range reads as a real.
• Callers: the Type 4 decoder, evaluate_postscript and the bench call parse directly. There is no String, replace or token Vec anywhere.
• Tests: the disabled #[cfg(all(test, not(test)))] test module, which still used the removed API, is deleted. The existing tests now wrap their programs in { }.

The issue's side notes needed no changes:

• neg, exp, ln, ceiling and round already landed in Implement missing Type 4 operators and speed up execution #606.
• Filtered streams are decoded when the object collection loads them, so raw_data() is already decoded.

Test plan

• cargo test -p pdf-postscript -p pdf-function passes.
• cargo clippy -p pdf-postscript -p pdf-function --all-targets -- -D warnings is clean.
• cargo fmt --check passes.
• cargo bench -p pdf-postscript --no-run builds.
• A throwaway check confirmed:
  • These parse: no spaces around braces, a % comment, NUL/FF separators, radix and exponent literals, and comments before and after the program.
  • These are rejected with the expected errors: a missing outer braces, unbalanced braces, a stray procedure, a ( delimiter, and trailing tokens.
• The full workspace clippy with --all-features was not run locally because the Skia download was blocked; CI covers it.

🤖 Generated with Claude Code

[github-actions]

Conformance reads

Documents that Safe-PDF reads differently from the base commit, compared with PDFium. A read
regresses when Safe-PDF fails to open a document it opened before, crashes or times out, or
counts a different number of pages than PDFium.

pdf.js corpus

1009 documents compared with ec5d7a5: 2 read worse, 0 read better.

Regressions

Document	Base	This pull request
issue17190_1	read, 1 page	timeout: time limit exceeded (stage safe-read)
issue19517	read, 1 page	timeout: time limit exceeded (stage safe-read)

840 documents of the PDFium corpus read as on ec5d7a5.

Details are in the job summaries of the workflow run. Rerun a case locally with
cargo conformance run --corpus <corpus> --read-only --case <id>.