From e25c1cbc871c6931c9d568e4bcfbe434cf970670 Mon Sep 17 00:00:00 2001 From: Hashem Aldhaheri Date: Mon, 7 Sep 2026 12:54:25 +0400 Subject: [PATCH] =?UTF-8?q?feat(pdf):=20PDF=20object=20layer=20=E2=80=94?= =?UTF-8?q?=20M6=206.1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The package layer of a format that is not a package. `mirsam-ooxml` opens a ZIP and asks for a part by name; `mirsam-pdf` opens a file and asks for an object by number — from the body, from inside a compressed object stream, or from a scan when the cross-reference table has gone stale. Both spellings of the cross-reference, `/Prev` chains, hybrid-reference files, object streams, the general-purpose stream filters, the page tree with the four attributes it inherits, and content streams decoded and joined. Nothing is wired into the CLI: `mirsam audit` still turns a `.pdf` away, because text extraction is §6.2 and a binary that accepted one before it could read it would be making a claim this item has not earned. The parser is written here and the decompressor is taken, which §6.1 asked for as an ADR rather than a `Cargo.toml` line. What settled it was not the size of ISO 32000-1 but which half of it mirsam asks for: the object graph and the content streams are §7.2 to §7.8, and everything from §8 on is the rendering this project has said it will never do. The deciding cost was the second one — a library's object model becomes the adapter's vocabulary, and with it every answer to *is a dangling reference null, is a wrong `/Length` readable, is a stale `startxref` fatal*. Those are exactly the honesty decisions §6.2 turns on, and inheriting them from a dependency would have been inheriting them unexamined. `flate2` is taken for `FlateDecode`, held to the `zlib-rs` backend `zip` already resolves, so the lockfile gains one workspace member and no third-party crate. See ADR 0010. What the file says about itself is checked rather than believed. A `/Length` may be an indirect reference — the value is in an object the table has not been parsed yet to find — and plenty of writers get the direct form wrong by a byte or two, so it is a hint, verified against `endstream`. One rule covers indirect, absent and simply wrong, and no caller had to be handed a resolver that cannot exist yet. Newest wins, and it is the walk that enforces it. Every insertion is first-wins down the `/Prev` chain, so an object an update redefined is found in the section that redefined it, and a free entry in a newer section *hides* one an older section still names. The trailers merge the same way, which is why a `/Root` a signature moved is the one used. A hybrid file's `/XRefStm` is read before the classic table beside it, because that table deliberately understates what the file holds — a reader that took it as the whole truth would draw a blank page and report nothing wrong with it. A stale `startxref` is recovered from rather than refused. Offsets go stale, and every viewer opens such a file anyway by scanning for `obj`: a document reported unreadable while a reader opens it is the wrong half of standing rule 4. `Reconstruction` says so out loud instead of hiding it. The refusals are the design. An encrypted document is refused at `Pdf::open` with the algorithm read out of the `/Encrypt` dictionary §7.6.1 keeps in the clear — `AES-256 (/Standard security handler, V5 R6)`, and a handler this tool does not know is named rather than guessed at. Nothing decrypts, including the permissions-only wrapper with an empty user password: a tool that opened those is one nobody could run on a file they were given. The four image codecs are named, never decoded, because a stream behind `DCTDecode` is not text this crate failed to read — it is text that is not in the file. That is ADR 0009 reaching the format that needs it most, and `note_unread` is open so §6.2's own unread sources reach one list rather than two. Seventy tests over eight fixtures, one per generator family, written byte by byte by `scripts/make-pdf-fixture.py` — a fixture built by the library under test proves only that the library agrees with itself. They sit beside the crate rather than in `tests/fixtures/`, because the golden corpus is what the *binary* does to a document and the binary does not read a `.pdf` yet. Every prefix and every single-byte corruption of all eight comes back as a document or as an error and never as a panic, which is what the bounds on nesting depth, reference-chain length, `/Prev` chain length and page-tree depth are for: this tool is run on files people were sent. --- AGENTS.md | 13 + CHANGELOG.md | 48 +- Cargo.lock | 8 + Cargo.toml | 6 + Makefile | 6 +- crates/mirsam-pdf/Cargo.toml | 16 + crates/mirsam-pdf/src/document.rs | 606 ++++++++++++++++++ crates/mirsam-pdf/src/encrypt.rs | 88 +++ crates/mirsam-pdf/src/filter.rs | 413 ++++++++++++ crates/mirsam-pdf/src/lexer.rs | 485 ++++++++++++++ crates/mirsam-pdf/src/lib.rs | 54 ++ crates/mirsam-pdf/src/object.rs | 353 ++++++++++ crates/mirsam-pdf/src/page.rs | 184 ++++++ crates/mirsam-pdf/src/xref.rs | 486 ++++++++++++++ crates/mirsam-pdf/tests/corpus.rs | 197 ++++++ crates/mirsam-pdf/tests/document.rs | 357 +++++++++++ crates/mirsam-pdf/tests/encrypt.rs | 97 +++ crates/mirsam-pdf/tests/filter.rs | 189 ++++++ crates/mirsam-pdf/tests/lexer.rs | 202 ++++++ .../mirsam-pdf/tests/pdfs/ascii-filters.pdf | Bin 0 -> 1947 bytes crates/mirsam-pdf/tests/pdfs/classic.pdf | Bin 0 -> 1292 bytes crates/mirsam-pdf/tests/pdfs/compressed.pdf | Bin 0 -> 1025 bytes crates/mirsam-pdf/tests/pdfs/damaged.pdf | Bin 0 -> 1292 bytes crates/mirsam-pdf/tests/pdfs/encrypted.pdf | Bin 0 -> 1726 bytes crates/mirsam-pdf/tests/pdfs/hybrid.pdf | Bin 0 -> 1285 bytes crates/mirsam-pdf/tests/pdfs/image-only.pdf | Bin 0 -> 820 bytes crates/mirsam-pdf/tests/pdfs/incremental.pdf | Bin 0 -> 1576 bytes crates/mirsam-pdf/tests/xref.rs | 131 ++++ docs/ARCHITECTURE.md | 37 ++ docs/PLAN.md | 87 ++- ...-pdf-object-layer-take-the-decompressor.md | 130 ++++ scripts/make-pdf-fixture.py | 546 ++++++++++++++++ 32 files changed, 4731 insertions(+), 8 deletions(-) create mode 100644 crates/mirsam-pdf/Cargo.toml create mode 100644 crates/mirsam-pdf/src/document.rs create mode 100644 crates/mirsam-pdf/src/encrypt.rs create mode 100644 crates/mirsam-pdf/src/filter.rs create mode 100644 crates/mirsam-pdf/src/lexer.rs create mode 100644 crates/mirsam-pdf/src/lib.rs create mode 100644 crates/mirsam-pdf/src/object.rs create mode 100644 crates/mirsam-pdf/src/page.rs create mode 100644 crates/mirsam-pdf/src/xref.rs create mode 100644 crates/mirsam-pdf/tests/corpus.rs create mode 100644 crates/mirsam-pdf/tests/document.rs create mode 100644 crates/mirsam-pdf/tests/encrypt.rs create mode 100644 crates/mirsam-pdf/tests/filter.rs create mode 100644 crates/mirsam-pdf/tests/lexer.rs create mode 100644 crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/classic.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/compressed.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/damaged.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/encrypted.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/hybrid.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/image-only.pdf create mode 100644 crates/mirsam-pdf/tests/pdfs/incremental.pdf create mode 100644 crates/mirsam-pdf/tests/xref.rs create mode 100644 docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md create mode 100644 scripts/make-pdf-fixture.py diff --git a/AGENTS.md b/AGENTS.md index cc18992..5626349 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -460,6 +460,19 @@ all of them against the published ECMA-376 schemas. A document the application offers to repair cannot answer "does it open the repaired file without a prompt", which is the M1 application check. +**The PDF fixtures are not corpus documents, and are generated too.** The eight +under `crates/mirsam-pdf/tests/pdfs/` come from `scripts/make-pdf-fixture.py` +and regenerate with `make pdfs` — one file per *generator family*, because what +PDF work hunts is a generator signature rather than an author mistake: a classic +cross-reference table, a cross-reference stream with object streams, a hybrid +file, an incremental update, a stale `startxref`, an AES-256 document, a +distiller's ASCII filter chain, and a page that is a scan. They live beside the +crate rather than in `tests/fixtures/` on purpose: the golden corpus is what the +*binary* does to a document, and `mirsam audit` does not read a `.pdf` yet. When +a `DocumentReader` lands, two of them join the corpus and the exit code `repair` +gives a readable format with no writer is recorded on a real PDF, the way +`quarterly-page.html` records it now. + **The two HTML corpus documents are the exception, and edited by hand.** A page *is* its own source: there is no package to build, so `make fixtures` has nothing to generate and `scripts/validate-ooxml.py` has nothing to validate. diff --git a/CHANGELOG.md b/CHANGELOG.md index f73537e..760e08e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,56 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Milestones M1 through M5 have landed since `v0.1.0`: byte-preserving `repair`, inheritance resolution, the DOCX and XLSX adapters with their writers, the HTML -reader, and shaping and font-coverage checks behind `--fonts`. Next is M6, the -read-only PDF adapter. See [`docs/PLAN.md`](docs/PLAN.md) and +reader, and shaping and font-coverage checks behind `--fonts`. M6, the +read-only PDF adapter, is under way: its object layer has landed and text +extraction is next. See [`docs/PLAN.md`](docs/PLAN.md) and [`docs/ROADMAP.md`](docs/ROADMAP.md). ### Added +- **`mirsam-pdf`, the PDF object layer.** The package layer of a format that is + not a package: both spellings of the cross-reference table, `/Prev` chains, + hybrid-reference files, object streams, the general-purpose stream filters, + the page tree with the four attributes it inherits, and content streams + decoded and joined. Nothing is wired into the CLI yet — `mirsam audit` + still turns a `.pdf` away — because text extraction is PLAN §6.2, and a + binary that accepted a `.pdf` before it could read one would be making a + claim this milestone has not earned. + + **The parser is written here and the decompressor is taken**, which PLAN + §6.1 asked for as an ADR rather than a `Cargo.toml` line. What settled it was + which half of ISO 32000-1 mirsam asks for: the object graph and the content + streams are §7.2 to §7.8, and everything from §8 on is the rendering this + project has said it will never do. The deciding cost was the second one — a + library's object model becomes the adapter's vocabulary, and with it every + answer to *is a dangling reference null, is a wrong `/Length` readable, is a + stale `startxref` fatal*, which are exactly the honesty decisions §6.2 turns + on. `flate2` is taken for `FlateDecode`, pinned to the `zlib-rs` backend + `zip` already resolves, so no dependency entered the tree. See + [ADR 0010](docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md). + + **An encrypted document is refused by name.** `AES-256 (/Standard security + handler, V5 R6)`, read out of the `/Encrypt` dictionary that §7.6.1 keeps in + the clear — including the permissions-only wrapper with an empty user + password, because a tool that opened those is one nobody could run on a file + they were given. The alternative was extracting nothing and reporting a clean + file, which is the failure the whole milestone is arranged against. + + **What could not be read is named, and so is what had to be rebuilt.** The + four image codecs are named rather than decoded — a stream behind `DCTDecode` + is not text this crate failed to read, it is text that is not in the file — + and a scanned page therefore reaches `unread` as `5 0 R — /DCTDecode, a + filter this tool does not decode`. That is ADR 0009 reaching the format that + needs it most. Separately, a file whose `startxref` has gone stale is + recovered from by scanning for `obj`, the way every viewer opens it, and says + so: a document reported unreadable while a reader opens it is the wrong half + of standing rule 4. + + Eight fixtures, one per generator family, are written byte by byte by + `scripts/make-pdf-fixture.py` and regenerate with `make pdfs` — a fixture + built by the library under test proves only that the library agrees with + itself. + - **The DOCX writer.** `mirsam repair report.docx fixed.docx` works, and `repair` stops turning a Word document away as a readable format without a writer. A repair writes `w:pPr/w:bidi`, `w:pPr/w:jc`, `w:rPr/w:lang@w:bidi`, diff --git a/Cargo.lock b/Cargo.lock index 7b2a07b..287ab84 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -315,6 +315,14 @@ dependencies = [ "zip", ] +[[package]] +name = "mirsam-pdf" +version = "0.1.0" +dependencies = [ + "flate2", + "mirsam-core", +] + [[package]] name = "new_debug_unreachable" version = "1.0.6" diff --git a/Cargo.toml b/Cargo.toml index f24ccf0..479cad0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,6 +4,7 @@ members = [ "crates/mirsam-core", "crates/mirsam-ooxml", "crates/mirsam-html", + "crates/mirsam-pdf", "crates/mirsam-conformance", "crates/mirsam-fonts", "crates/mirsam-cli", @@ -24,6 +25,7 @@ categories = ["command-line-utilities", "text-processing", "internationalization mirsam-core = { path = "crates/mirsam-core", version = "0.1.0" } mirsam-ooxml = { path = "crates/mirsam-ooxml", version = "0.1.0" } mirsam-html = { path = "crates/mirsam-html", version = "0.1.0" } +mirsam-pdf = { path = "crates/mirsam-pdf", version = "0.1.0" } mirsam-fonts = { path = "crates/mirsam-fonts", version = "0.1.0" } unicode-bidi = "0.3.18" unicode-normalization = "0.1.25" @@ -36,6 +38,10 @@ quick-xml = "0.42" # than taking `markup5ever_rcdom`, which names itself unsupported. html5ever = "0.39" zip = { version = "8.6", default-features = false, features = ["deflate"] } +# FlateDecode. Held to the backend `zip` already pulls in: two inflate +# implementations in one binary would be two answers to "what do these bytes +# decompress to". +flate2 = { version = "1.1", default-features = false, features = ["zlib-rs"] } serde = { version = "1", features = ["derive"] } serde_json = "1" thiserror = "2" diff --git a/Makefile b/Makefile index ba0d880..b584de5 100644 --- a/Makefile +++ b/Makefile @@ -76,8 +76,12 @@ fixtures: ## Regenerate the hand-built corpus documents, then their reports @python3 scripts/make-torture-fixture.py @python3 scripts/make-word-fixture.py @python3 scripts/make-excel-fixture.py + @$(MAKE) --no-print-directory pdfs @$(MAKE) --no-print-directory golden +pdfs: ## Regenerate the PDF fixture corpus (one file per generator family) + @python3 scripts/make-pdf-fixture.py + fonts: ## Regenerate the hand-built fonts the shaping tests run against @python3 scripts/make-shaping-fixture.py @@ -113,5 +117,5 @@ install: ## Install mirsam into ~/.cargo/bin @cargo install --path crates/mirsam-cli .PHONY: help build test fmt fmt-check lint doc clean version check-version \ - codename release-name msrv audit-deps golden fixtures fonts names \ + codename release-name msrv audit-deps golden fixtures pdfs fonts names \ validate-fixtures corpus verify pre-push hooks-install install diff --git a/crates/mirsam-pdf/Cargo.toml b/crates/mirsam-pdf/Cargo.toml new file mode 100644 index 0000000..977d1d3 --- /dev/null +++ b/crates/mirsam-pdf/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "mirsam-pdf" +description = "PDF object layer for mirsam: cross-reference tables and streams, object streams, stream filters and the page tree. Reads; never writes." +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true + +[dependencies] +mirsam-core.workspace = true +# The one thing not written here. See docs/adr/0010. Pinned to the backend the +# workspace already resolves through `zip`, so the binary carries one inflate +# implementation rather than two. +flate2 = { workspace = true } diff --git a/crates/mirsam-pdf/src/document.rs b/crates/mirsam-pdf/src/document.rs new file mode 100644 index 0000000..89f2c24 --- /dev/null +++ b/crates/mirsam-pdf/src/document.rs @@ -0,0 +1,606 @@ +//! The document: the object graph, resolved, and the page tree over it. +//! +//! This is the package layer of a format that is not a package. `Package` in +//! `mirsam-ooxml` answers "give me the part named `ppt/slides/slide1.xml`"; +//! [`Pdf`] answers "give me object `12 0 R`, wherever the file put it" — in the +//! body, inside a compressed object stream, or in a position only a scan +//! could find. +//! +//! ## Everything is loaded at open, and the file bytes are then dropped +//! +//! An object stream has to be decompressed before the objects inside it exist +//! at all, and a page's `/Resources` may be inherited from a node three levels +//! up the tree. Lazy loading would mean either interior mutability threaded +//! through every accessor or a resolver argument on every call, and it would +//! buy nothing: this tool reads the whole document. +//! +//! Streams are kept *filtered*, so what is held is about the size of the file, +//! and the file's own bytes are released once the graph is built. +//! +//! ## What could not be read is answered for +//! +//! A page whose content is a JPEG, an object at an offset that is not there, an +//! object stream behind a filter this crate does not implement — each is +//! recorded and named by [`Pdf::unread`]. ADR 0009 is the reason: a source the +//! adapter could not read is part of the report, and a document that produced +//! no text must never be indistinguishable from one with no defects. + +use mirsam_core::error::{Error, Result}; +use std::cell::RefCell; +use std::collections::{BTreeMap, BTreeSet}; +use std::fmt; +use std::path::Path; + +use crate::filter::{self, Decoded}; +use crate::lexer::{Lexer, find}; +use crate::object::{Dictionary, Name, Object, ObjectId, Stream}; +use crate::xref::{self, Slot, Xref}; +use crate::{encrypt, page}; + +/// How far into the file `%PDF-` is looked for. §7.5.2 puts it at byte zero; +/// a kilobyte of tolerance covers the shell scripts and mail gateways that put +/// something in front of it, and stops well short of finding the string inside +/// an embedded document. +const HEADER_WINDOW: usize = 1024; + +/// How many references are followed before a chain is called circular. +/// `12 0 R` pointing at `13 0 R` pointing back is a file, not a document. +const MAX_INDIRECTION: usize = 32; + +/// Something the file did not state usably and the loader had to work out. +/// +/// Not a defect and not a finding: a fact about how the document was read, +/// which a caller may want to say out loud and which the tests assert on. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Reconstruction { + /// The cross-reference chain was unusable, so the body was scanned for + /// ` obj`. + CrossReferenceTable, + /// `/Root /Pages` did not yield a walkable tree, so the pages were taken + /// from the objects that call themselves `/Type /Page`. + PageTree, +} + +impl fmt::Display for Reconstruction { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::CrossReferenceTable => f.write_str("the cross-reference table"), + Self::PageTree => f.write_str("the page tree"), + } + } +} + +/// Something in the document this crate did not read, and why. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct Unread { + /// The object it was, where there is one to name. + pub object: Option, + pub reason: String, +} + +/// Named as the document names it, so a reader can find the thing being talked +/// about: `12 0 R — /DCTDecode, an image codec this tool does not decode`. +impl fmt::Display for Unread { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.object { + Some(id) => write!(f, "{id} — {}", self.reason), + None => f.write_str(&self.reason), + } + } +} + +/// One entry of the object graph. +struct Entry { + generation: u16, + object: Object, +} + +/// A PDF, read. +pub struct Pdf { + name: String, + version: String, + objects: BTreeMap, + trailer: Dictionary, + catalog: ObjectId, + pages: Vec, + reconstructed: Vec, + /// Grown while reading, because a stream's filter is only discovered when + /// somebody asks for its bytes. A `RefCell` rather than `&mut self` on + /// every accessor: recording what was not read is bookkeeping, and making + /// it change the shape of the read API would put it in every caller. + unread: RefCell>, +} + +/// A summary rather than the graph: printing every object of a document is +/// not a thing a test failure or a `dbg!` wants, and the pieces are all +/// reachable through the accessors. +impl fmt::Debug for Pdf { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Pdf") + .field("name", &self.name) + .field("version", &self.version) + .field("objects", &self.objects.len()) + .field("pages", &self.pages.len()) + .field("reconstructed", &self.reconstructed) + .field("unread", &self.unread.borrow().len()) + .finish() + } +} + +impl Pdf { + /// Open and read a PDF. + pub fn open(path: &Path) -> Result { + if !path.exists() { + return Err(Error::NotFound); + } + let data = std::fs::read(path)?; + Self::from_bytes(&data, &path.display().to_string()) + } + + /// Read a PDF already in memory. `name` is used only in messages. + pub fn from_bytes(data: &[u8], name: &str) -> Result { + let (shift, version) = header(data)?; + + let mut reconstructed = Vec::new(); + let xref = match xref::read(data, shift) { + Ok(xref) if !xref.slots.is_empty() => xref, + _ => { + reconstructed.push(Reconstruction::CrossReferenceTable); + xref::recover(data) + } + }; + + refuse_encryption(data, &xref, shift)?; + + let mut unread = Vec::new(); + let mut objects = load(data, &xref, shift, &mut unread); + + // An offset that pointed at nothing is worth one scan of the body + // before it is reported: a file with a stale table is a file every + // viewer opens, and the objects are all still there. + let missing = xref + .slots + .iter() + .filter(|(number, slot)| { + matches!(slot, Slot::InFile { .. }) && !objects.contains_key(number) + }) + .count(); + if missing > 0 { + let rebuilt = xref::recover(data); + let mut recovered = load(data, &rebuilt, shift, &mut Vec::new()); + let mut gained = false; + for (number, entry) in std::mem::take(&mut recovered) { + if let std::collections::btree_map::Entry::Vacant(slot) = objects.entry(number) { + slot.insert(entry); + gained = true; + } + } + if gained { + reconstructed.push(Reconstruction::CrossReferenceTable); + unread.retain(|entry| { + entry + .object + .is_none_or(|id| !objects.contains_key(&id.number)) + }); + } + } + + if objects.is_empty() { + return Err(Error::Format(format!( + "{name}: no readable objects; the file states a cross-reference table and has \ + nothing where it points" + ))); + } + + let mut pdf = Self { + name: name.to_string(), + version, + objects, + trailer: xref.trailer.clone(), + catalog: ObjectId::default(), + pages: Vec::new(), + reconstructed, + unread: RefCell::new(unread), + }; + pdf.expand_object_streams(&xref); + pdf.catalog = pdf.find_catalog()?; + let (pages, rebuilt) = page::tree(&pdf); + pdf.pages = pages; + if rebuilt { + pdf.reconstructed.push(Reconstruction::PageTree); + } + pdf.reconstructed.sort_unstable(); + pdf.reconstructed.dedup(); + Ok(pdf) + } + + /// The path or label this document was opened under. + pub fn name(&self) -> &str { + &self.name + } + + /// The version in the `%PDF-` header: `1.7`, `2.0`. + /// + /// The header alone. A catalog may override it with `/Version`, and + /// [`Pdf::catalog`] is where a caller asks for that — reporting one number + /// for two different claims would hide a disagreement between them. + pub fn version(&self) -> &str { + &self.version + } + + pub fn trailer(&self) -> &Dictionary { + &self.trailer + } + + /// The document catalog, `/Root`. + pub fn catalog(&self) -> &Dictionary { + self.object(self.catalog) + .and_then(Object::as_dict) + .unwrap_or(&EMPTY) + } + + /// The pages, in the order the page tree gives them. + pub fn pages(&self) -> &[page::Page] { + &self.pages + } + + /// How many objects were loaded. + pub fn object_count(&self) -> usize { + self.objects.len() + } + + /// What the loader had to work out because the file did not state it + /// usably. Empty for a well-formed document. + pub fn reconstructed(&self) -> &[Reconstruction] { + &self.reconstructed + } + + /// What this crate did not read, in object order, each named as the + /// document names it. + /// + /// This is what a `DocumentReader` over PDF answers `unread_sources` with. + pub fn unread(&self) -> Vec { + let mut entries = self.unread.borrow().clone(); + entries.sort(); + entries.dedup(); + entries.iter().map(ToString::to_string).collect() + } + + /// Record something that was not read. Public because the text layer above + /// this one has its own reasons — a font with no `ToUnicode` map is a + /// source it could not read, and the list has to be one list. + pub fn note_unread(&self, object: Option, reason: impl Into) { + self.unread.borrow_mut().push(Unread { + object, + reason: reason.into(), + }); + } + + /// The object at `id`, or `Null` if the document has none. + /// + /// **Matched on the object number alone.** A reference carries a + /// generation, and matching on it would be more faithful to §7.3.10 — but + /// a generation that disagrees with the table is something writers produce + /// and every reader ignores, and refusing to follow such a reference would + /// lose an object the file plainly holds. + pub fn object(&self, id: ObjectId) -> Option<&Object> { + self.objects.get(&id.number).map(|entry| &entry.object) + } + + /// The generation the file stored `number` under. + pub fn generation(&self, number: u32) -> Option { + self.objects.get(&number).map(|entry| entry.generation) + } + + /// Every object, in object-number order — which is the order the file + /// wrote them, and the only order available to a document whose page tree + /// has to be reconstructed. + pub fn objects(&self) -> impl Iterator { + self.objects + .iter() + .map(|(number, entry)| (ObjectId::new(*number, entry.generation), &entry.object)) + } + + /// Follow references until something else is reached. + pub fn resolve<'a>(&'a self, object: &'a Object) -> &'a Object { + let mut current = object; + for _ in 0..MAX_INDIRECTION { + let Object::Reference(id) = current else { + return current; + }; + match self.object(*id) { + Some(next) => current = next, + None => return &Object::Null, + } + } + // A cycle. `Null` is the honest answer: §7.3.9 already says an + // undefined reference is null, and a document whose objects point at + // each other has no value there to give. + &Object::Null + } + + /// A dictionary entry, resolved. Absent and explicitly null are one + /// answer, as §7.3.7 says they are. + pub fn get<'a>(&'a self, dict: &'a Dictionary, key: &str) -> Option<&'a Object> { + match dict.get(key).map(|value| self.resolve(value)) { + Some(Object::Null) | None => None, + Some(value) => Some(value), + } + } + + /// The bytes of a stream, decoded. + /// + /// `Ok(None)` means the stream is behind a filter this crate does not + /// decode — an image codec, all but always — and the fact has been + /// recorded in [`Pdf::unread`]. It is not an error: a scanned page is a + /// page whose text is not in the file, which is a true thing to report and + /// not a failure to read one. + pub fn decoded(&self, id: ObjectId) -> Result>> { + let Some(stream) = self.object(id).and_then(Object::as_stream) else { + return Ok(None); + }; + self.decode_stream(Some(id), stream) + } + + fn decode_stream(&self, id: Option, stream: &Stream) -> Result>> { + let filters = self.filters(&stream.dict); + let parms = self.decode_parms(&stream.dict, filters.len()); + match filter::decode(&stream.data, &filters, &parms)? { + Decoded::Bytes(bytes) => Ok(Some(bytes)), + Decoded::Stopped { filter } => { + self.note_unread(id, format!("{filter}, a filter this tool does not decode")); + Ok(None) + } + } + } + + /// `/Filter`, which is a name or an array of them, either of which may be + /// written indirectly. + fn filters(&self, dict: &Dictionary) -> Vec { + let Some(filter) = self.get(dict, "Filter").or_else(|| self.get(dict, "F")) else { + return Vec::new(); + }; + match filter { + Object::Name(name) => vec![name.clone()], + Object::Array(items) => items + .iter() + .filter_map(|item| self.resolve(item).as_name()) + .cloned() + .collect(), + _ => Vec::new(), + } + } + + /// `/DecodeParms`, padded to the length of the filter chain so index `i` + /// belongs to filter `i` whatever shape the file wrote it in. + fn decode_parms(&self, dict: &Dictionary, filters: usize) -> Vec { + let mut parms = match self + .get(dict, "DecodeParms") + .or_else(|| self.get(dict, "DP")) + { + Some(Object::Dictionary(one)) => vec![one.clone()], + Some(Object::Array(items)) => items + .iter() + .map(|item| self.resolve(item).as_dict().cloned().unwrap_or_default()) + .collect(), + _ => Vec::new(), + }; + parms.resize(filters, Dictionary::new()); + parms + } + + /// Decompress every object stream the table pointed into, and add the + /// objects it holds. + /// + /// Objects already loaded from the body win: a compressed object is only + /// reachable through the table that named it, so a body definition of the + /// same number came from a section that superseded it. + fn expand_object_streams(&mut self, xref: &Xref) { + for (container, members) in xref.in_streams() { + let id = ObjectId::new(container, self.generation(container).unwrap_or(0)); + let Some(stream) = self.object(id).and_then(Object::as_stream) else { + self.note_unread(Some(id), "an object stream the file does not contain"); + continue; + }; + if !stream + .dict + .get("Type") + .is_none_or(|t| t.as_name().is_some_and(|name| name.is("ObjStm"))) + { + self.note_unread(Some(id), "referenced as an object stream, and is not one"); + continue; + } + let count = self + .get(&stream.dict, "N") + .and_then(Object::as_i64) + .unwrap_or(0); + let first = self + .get(&stream.dict, "First") + .and_then(Object::as_i64) + .unwrap_or(0); + let bytes = match self.decode_stream(Some(id), stream) { + Ok(Some(bytes)) => bytes, + Ok(None) => continue, + Err(e) => { + self.note_unread(Some(id), format!("object stream: {e}")); + continue; + } + }; + + let Ok(first) = usize::try_from(first) else { + self.note_unread(Some(id), "object stream: /First is not an offset"); + continue; + }; + // The header is `N` pairs of ` `, each + // offset relative to /First. + let mut header = Lexer::new(&bytes); + let mut offsets = Vec::new(); + for _ in 0..count.max(0) { + let (Ok(Some(number)), Ok(Some(offset))) = ( + header.object().map(|o| o.as_i64()), + header.object().map(|o| o.as_i64()), + ) else { + break; + }; + match (u32::try_from(number), usize::try_from(offset)) { + (Ok(number), Ok(offset)) => offsets.push((number, offset)), + _ => break, + } + } + + let wanted: BTreeSet = members.iter().map(|(number, _)| *number).collect(); + for (number, offset) in offsets { + if !wanted.contains(&number) || self.objects.contains_key(&number) { + continue; + } + let Some(at) = first.checked_add(offset).filter(|at| *at < bytes.len()) else { + self.note_unread( + Some(ObjectId::new(number, 0)), + format!("object stream {container}: its offset is past the end"), + ); + continue; + }; + match Lexer::at(&bytes, at).object() { + // §7.5.7: every object in an object stream has generation 0. + Ok(object) => { + self.objects.insert( + number, + Entry { + generation: 0, + object, + }, + ); + } + Err(e) => self.note_unread( + Some(ObjectId::new(number, 0)), + format!("in object stream {container}: {e}"), + ), + } + } + } + } + + /// `/Root`, or the object that calls itself the catalog when the trailer + /// does not say. + fn find_catalog(&self) -> Result { + if let Some(id) = self.trailer.get("Root").and_then(Object::as_reference) + && self.object(id).is_some_and(|root| root.is_type("Catalog")) + { + return Ok(id); + } + let found = self + .objects + .iter() + .find(|(_, entry)| entry.object.is_type("Catalog")) + .map(|(number, entry)| ObjectId::new(*number, entry.generation)); + found.ok_or_else(|| { + Error::Format(format!( + "{}: no document catalog; the trailer names none and no object calls itself one", + self.name + )) + }) + } +} + +static EMPTY: Dictionary = Dictionary::empty(); + +/// `%PDF-`, and where in the file it starts. +fn header(data: &[u8]) -> Result<(usize, String)> { + let window = &data[..data.len().min(HEADER_WINDOW)]; + let Some(at) = find(window, b"%PDF-") else { + return Err(Error::Format( + "not a PDF: no `%PDF-` header in the first kilobyte".into(), + )); + }; + let version: String = data[at + 5..] + .iter() + .take(8) + .take_while(|b| b.is_ascii_digit() || **b == b'.') + .map(|b| *b as char) + .collect(); + if version.is_empty() { + return Err(Error::Format( + "not a PDF: the `%PDF-` header states no version".into(), + )); + } + Ok((at, version)) +} + +/// Refuse an encrypted document, naming the encryption. +/// +/// The `/Encrypt` dictionary is read on its own, before the body: §7.6.1 keeps +/// it in the clear precisely so a reader can find out what it is up against +/// without being able to decrypt anything. +fn refuse_encryption(data: &[u8], xref: &Xref, shift: usize) -> Result<()> { + let Some(entry) = xref.trailer.get("Encrypt") else { + return Ok(()); + }; + let dict = match entry { + Object::Dictionary(dict) => Some(dict.clone()), + Object::Reference(id) => match xref.slots.get(&id.number) { + Some(Slot::InFile { offset, .. }) => usize::try_from(*offset) + .ok() + .and_then(|at| at.checked_add(shift)) + .filter(|at| *at < data.len()) + .and_then(|at| Lexer::at(data, at).indirect().ok()) + .and_then(|(_, object)| object.as_dict().cloned()), + _ => None, + }, + _ => None, + }; + + let what = dict + .as_ref() + .map(encrypt::describe) + .unwrap_or_else(|| "an encryption the file does not describe".into()); + Err(Error::Format(format!( + "encrypted with {what}; mirsam does not decrypt, so this document is unread rather than \ + clean. Rebuild it without protection, or remove the protection in the application that \ + wrote it" + ))) +} + +/// Parse every object the table places in the body. +fn load(data: &[u8], xref: &Xref, shift: usize, unread: &mut Vec) -> BTreeMap { + let mut objects = BTreeMap::new(); + for (&number, slot) in &xref.slots { + let Slot::InFile { offset, generation } = *slot else { + continue; + }; + let Ok(offset) = usize::try_from(offset) else { + continue; + }; + let id = ObjectId::new(number, generation); + + // The stored offset, then the same offset past a prologue. A file with + // bytes before its header may have been written either way round. + let mut parsed = None; + for at in [Some(offset), offset.checked_add(shift)] + .into_iter() + .flatten() + { + if at >= data.len() { + continue; + } + match Lexer::at(data, at).indirect() { + Ok((found, object)) if found.number == number => { + parsed = Some((found.generation, object)); + break; + } + _ => continue, + } + } + match parsed { + Some((generation, object)) => { + objects.insert(number, Entry { generation, object }); + } + None => unread.push(Unread { + object: Some(id), + reason: format!( + "the cross-reference table puts it at byte {offset}, and it is not there" + ), + }), + } + } + objects +} diff --git a/crates/mirsam-pdf/src/encrypt.rs b/crates/mirsam-pdf/src/encrypt.rs new file mode 100644 index 0000000..f38559e --- /dev/null +++ b/crates/mirsam-pdf/src/encrypt.rs @@ -0,0 +1,88 @@ +//! Encryption: what it is, said precisely, so the document can be refused. +//! +//! mirsam does not decrypt. That is not a gap to be filled later — a tool that +//! opened protected documents by trying the empty password would be a tool +//! nobody could run on a file they were given, and one that asked for a +//! password would have to hold it. +//! +//! What matters is what the refusal *says*. An encrypted PDF whose strings and +//! streams cannot be read produces no text, and a reader that stopped there +//! would report a document with no defects — the exact failure ADR 0009 exists +//! to prevent, arriving as a whole file rather than one source inside it. So +//! the refusal names the algorithm, from the `/Encrypt` dictionary alone, which +//! is never itself encrypted (§7.6.1). A person reading it knows whether they +//! are looking at a permissions-only wrapper they can strip or a document they +//! need a password for, without opening anything. + +use crate::object::{Dictionary, Object}; + +/// The `/Encrypt` dictionary in one line: the algorithm, the handler and the +/// version the file states. +/// +/// Nothing here is inferred. Where the dictionary does not say, the answer says +/// so — an undeclared algorithm is reported as undeclared, never guessed at, +/// because the point of the message is that a reader can trust it. +pub fn describe(encrypt: &Dictionary) -> String { + let handler = encrypt + .get("Filter") + .and_then(Object::as_name) + .map(ToString::to_string) + .unwrap_or_else(|| "/Standard".into()); + let v = encrypt.get("V").and_then(Object::as_i64); + let r = encrypt.get("R").and_then(Object::as_i64); + let bits = encrypt.get("Length").and_then(Object::as_i64).unwrap_or(40); + + let algorithm = match v { + Some(1) => "RC4 40-bit".to_string(), + Some(2) | Some(3) => format!("RC4 {bits}-bit"), + Some(4) | Some(5) => crypt_filter(encrypt).unwrap_or_else(|| match v { + Some(5) => "AES-256".into(), + _ => format!("RC4 {bits}-bit"), + }), + Some(0) => "an algorithm the document does not state".to_string(), + _ => "an undeclared algorithm".to_string(), + }; + + let version = match (v, r) { + (Some(v), Some(r)) => format!(", V{v} R{r}"), + (Some(v), None) => format!(", V{v}"), + (None, Some(r)) => format!(", R{r}"), + (None, None) => String::new(), + }; + format!("{algorithm} ({handler} security handler{version})") +} + +/// §7.6.5's crypt filters, where PDF 1.5 and later actually state the method. +/// +/// `/StdCF` is the name every writer uses, but the *default* filter for streams +/// is whatever `/StmF` names, so that is asked first: a file whose stream +/// filter is not the one called `StdCF` would otherwise be described by a +/// dictionary entry nothing in it uses. +fn crypt_filter(encrypt: &Dictionary) -> Option { + let filters = encrypt.get("CF")?.as_dict()?; + let named = encrypt + .get("StmF") + .and_then(Object::as_name) + .and_then(|name| name.as_str().map(str::to_string)) + .unwrap_or_else(|| "StdCF".into()); + let filter = filters + .get(&named) + .or_else(|| filters.get("StdCF")) + .and_then(Object::as_dict)?; + + let method = filter.get("CFM").and_then(Object::as_name)?; + let bits = filter + .get("Length") + .and_then(Object::as_i64) + // §7.6.5 lets a crypt filter state its length in bytes; the trailer + // dictionary states it in bits. Both spellings appear. + .map(|n| if n <= 64 { n * 8 } else { n }); + + Some(match method { + m if m.is("AESV2") => "AES-128".into(), + m if m.is("AESV3") => "AES-256".into(), + m if m.is("V2") => format!("RC4 {}-bit", bits.unwrap_or(128)), + m if m.is("None") => "no stream encryption".into(), + other => format!("{other}"), + }) +} diff --git a/crates/mirsam-pdf/src/filter.rs b/crates/mirsam-pdf/src/filter.rs new file mode 100644 index 0000000..2e4e117 --- /dev/null +++ b/crates/mirsam-pdf/src/filter.rs @@ -0,0 +1,413 @@ +//! Stream filters: §7.4, the half of it that carries text. +//! +//! ## Which filters are implemented, and why the rest are named rather than +//! failed +//! +//! `FlateDecode`, `LZWDecode`, `ASCIIHexDecode`, `ASCII85Decode` and +//! `RunLengthDecode` are general-purpose: a content stream, a `ToUnicode` CMap, +//! an object stream or a cross-reference stream may arrive through any of them, +//! and a document whose text is behind one is a document this tool must read. +//! +//! `DCTDecode`, `JPXDecode`, `CCITTFaxDecode` and `JBIG2Decode` are image +//! codecs. Decoding one produces a raster, and mirsam has said it will never +//! rasterise — so a stream behind one is not text this tool failed to read, it +//! is text that is not in the file at all. [`Decoded::Stopped`] carries the +//! filter's name so the document can say which, which is ADR 0009's shape: a +//! source the adapter could not read is part of the report, and a scanned page +//! must never come back as a page with no defects. +//! +//! `Crypt` is the same shape for a different reason — a document reaching this +//! module through it was not refused at [`crate::Pdf::open`], which would be a +//! bug rather than a file. +//! +//! ## A truncated stream keeps what it had +//! +//! Real files are truncated, and inflate reports that as an error after +//! producing perfectly good bytes. Discarding them would turn a document with +//! a damaged tail into a document with no text — the failure this project +//! treats as worse than a miss. So a decode that produced output keeps it, and +//! only one that produced nothing is an error. + +use flate2::read::{DeflateDecoder, ZlibDecoder}; +use mirsam_core::error::{Error, Result}; +use std::io::Read; + +use crate::object::{Dictionary, Name, Object}; + +/// What came out of a filter chain. +pub enum Decoded { + /// Every filter in the chain was applied. + Bytes(Vec), + /// The chain reached a filter this crate does not decode. Named, never + /// silently dropped. + Stopped { filter: Name }, +} + +/// The filters that end a chain rather than continue it: an image codec +/// produces pixels, and mirsam does not rasterise. +fn is_image_codec(name: &Name) -> bool { + ["DCTDecode", "JPXDecode", "CCITTFaxDecode", "JBIG2Decode"] + .iter() + .any(|codec| name.is(codec)) +} + +/// Run `data` through `filters` in order, with `parms[i]` the parameters of +/// `filters[i]`. +/// +/// A missing entry in `parms` is an empty dictionary, which is what an absent +/// `/DecodeParms` means. +pub fn decode(data: &[u8], filters: &[Name], parms: &[Dictionary]) -> Result { + let mut bytes = data.to_vec(); + let empty = Dictionary::new(); + for (index, filter) in filters.iter().enumerate() { + let parms = parms.get(index).unwrap_or(&empty); + bytes = match filter { + f if f.is("FlateDecode") || f.is("Fl") => predict(inflate(&bytes)?, parms)?, + f if f.is("LZWDecode") || f.is("LZW") => { + let early = parms + .get("EarlyChange") + .and_then(Object::as_i64) + .unwrap_or(1); + predict(lzw(&bytes, early != 0)?, parms)? + } + f if f.is("ASCIIHexDecode") || f.is("AHx") => ascii_hex(&bytes), + f if f.is("ASCII85Decode") || f.is("A85") => ascii85(&bytes)?, + f if f.is("RunLengthDecode") || f.is("RL") => run_length(&bytes), + // §7.4.1: an identity filter, which some writers spell explicitly. + f if f.is("Identity") => bytes, + f if is_image_codec(f) || f.is("Crypt") => { + return Ok(Decoded::Stopped { + filter: filter.clone(), + }); + } + _ => { + return Ok(Decoded::Stopped { + filter: filter.clone(), + }); + } + }; + } + Ok(Decoded::Bytes(bytes)) +} + +/// zlib, then raw deflate. +/// +/// Both spellings are in the wild: §7.4.4 requires the zlib wrapper, and a +/// long tail of writers emits the bare deflate stream. Trying the wrapper +/// first keeps the conformant case exact. +fn inflate(data: &[u8]) -> Result> { + let trimmed = { + let start = data + .iter() + .position(|b| !crate::object::is_whitespace(*b)) + .unwrap_or(data.len()); + &data[start..] + }; + + let mut out = Vec::new(); + let zlib = ZlibDecoder::new(trimmed).read_to_end(&mut out); + if zlib.is_ok() || !out.is_empty() { + return Ok(out); + } + + out.clear(); + let raw = DeflateDecoder::new(trimmed).read_to_end(&mut out); + if raw.is_ok() || !out.is_empty() { + return Ok(out); + } + Err(Error::Format(format!( + "FlateDecode: {}", + zlib.err() + .map(|e| e.to_string()) + .unwrap_or_else(|| "not a deflate stream".into()) + ))) +} + +/// §7.4.4.4's predictors, which a stream carries when it holds rows of equal +/// width — a cross-reference stream always does, and an image often does. +fn predict(data: Vec, parms: &Dictionary) -> Result> { + let int = |key: &str, default: i64| parms.get(key).and_then(Object::as_i64).unwrap_or(default); + let predictor = int("Predictor", 1); + if predictor < 2 { + return Ok(data); + } + let colors = int("Colors", 1).clamp(1, 32) as usize; + let bpc = int("BitsPerComponent", 8).clamp(1, 32) as usize; + let columns = int("Columns", 1).max(1) as usize; + + let bpp = (colors * bpc).div_ceil(8).max(1); + let row = (colors * bpc * columns).div_ceil(8); + + if predictor == 2 { + return Ok(tiff_predictor(data, colors, bpc, row)); + } + + // PNG: each row is preceded by its filter type. + let mut out = Vec::with_capacity(data.len()); + let mut previous = vec![0u8; row]; + for chunk in data.chunks(row + 1) { + // A truncated final row is dropped rather than un-filtered against + // bytes that are not there. + if chunk.len() < 2 { + break; + } + let (kind, encoded) = (chunk[0], &chunk[1..]); + let mut current = encoded.to_vec(); + for i in 0..current.len() { + let left = if i >= bpp { current[i - bpp] } else { 0 }; + let up = *previous.get(i).unwrap_or(&0); + let up_left = if i >= bpp { + *previous.get(i - bpp).unwrap_or(&0) + } else { + 0 + }; + current[i] = match kind { + 0 => current[i], + 1 => current[i].wrapping_add(left), + 2 => current[i].wrapping_add(up), + 3 => current[i].wrapping_add(((u16::from(left) + u16::from(up)) / 2) as u8), + 4 => current[i].wrapping_add(paeth(left, up, up_left)), + other => { + return Err(Error::Format(format!( + "unknown PNG predictor filter type {other}" + ))); + } + }; + } + out.extend_from_slice(¤t); + previous = current; + previous.resize(row, 0); + } + Ok(out) +} + +fn tiff_predictor(mut data: Vec, colors: usize, bpc: usize, row: usize) -> Vec { + // Only the byte-aligned case is undone. Sub-byte components appear in + // images and never in a stream this crate reads for text, and guessing at + // one would corrupt bytes rather than leave them alone. + if bpc != 8 || row == 0 { + return data; + } + for start in (0..data.len()).step_by(row) { + let end = (start + row).min(data.len()); + for i in (start + colors)..end { + data[i] = data[i].wrapping_add(data[i - colors]); + } + } + data +} + +fn paeth(left: u8, up: u8, up_left: u8) -> u8 { + let p = i16::from(left) + i16::from(up) - i16::from(up_left); + let (dl, du, dul) = ( + (p - i16::from(left)).abs(), + (p - i16::from(up)).abs(), + (p - i16::from(up_left)).abs(), + ); + if dl <= du && dl <= dul { + left + } else if du <= dul { + up + } else { + up_left + } +} + +/// §7.4.2. White space is ignored, `>` ends the data, and an odd final digit +/// is padded with a zero. +fn ascii_hex(data: &[u8]) -> Vec { + let mut out = Vec::with_capacity(data.len() / 2); + let mut half: Option = None; + for &b in data { + match b { + b'>' => break, + b if b.is_ascii_hexdigit() => { + let value = match b { + b'0'..=b'9' => b - b'0', + b'a'..=b'f' => b - b'a' + 10, + _ => b - b'A' + 10, + }; + match half.take() { + Some(hi) => out.push(hi * 16 + value), + None => half = Some(value), + } + } + _ => {} + } + } + if let Some(hi) = half { + out.push(hi * 16); + } + out +} + +/// §7.4.3, base-85. `z` is four zero bytes, `~>` ends the data, and a partial +/// final group encodes fewer than four bytes. +fn ascii85(data: &[u8]) -> Result> { + let data = data.strip_prefix(b"<~").unwrap_or(data); + let mut out = Vec::with_capacity(data.len() * 4 / 5); + let mut group = [0u8; 5]; + let mut filled = 0usize; + + let flush = |group: &[u8; 5], filled: usize, out: &mut Vec| { + if filled == 0 { + return; + } + let mut value: u32 = 0; + for (i, digit) in group.iter().enumerate() { + // A short final group is padded with the highest digit, `u`. + let digit = if i < filled { *digit } else { 84 }; + value = value.wrapping_mul(85).wrapping_add(u32::from(digit)); + } + out.extend_from_slice(&value.to_be_bytes()[..filled - 1]); + }; + + for b in data.iter().copied() { + match b { + b'~' => break, + b'z' if filled == 0 => out.extend_from_slice(&[0, 0, 0, 0]), + b'z' => return Err(Error::Format("ASCII85Decode: `z` inside a group".into())), + b'!'..=b'u' => { + group[filled] = b - b'!'; + filled += 1; + if filled == 5 { + flush(&group, 5, &mut out); + filled = 0; + } + } + b if crate::object::is_whitespace(b) => {} + other => { + return Err(Error::Format(format!( + "ASCII85Decode: `{}` is not a base-85 digit", + other as char + ))); + } + } + } + if filled == 1 { + return Err(Error::Format( + "ASCII85Decode: a final group of one digit encodes no bytes".into(), + )); + } + flush(&group, filled, &mut out); + Ok(out) +} + +/// §7.4.5. A length byte under 128 introduces that many literal bytes plus +/// one; over 128 it repeats the next byte `257 - length` times; 128 ends the +/// data. +fn run_length(data: &[u8]) -> Vec { + let mut out = Vec::new(); + let mut i = 0usize; + while i < data.len() { + let length = data[i]; + i += 1; + match length { + 128 => break, + 0..=127 => { + let count = usize::from(length) + 1; + let end = (i + count).min(data.len()); + out.extend_from_slice(&data[i..end]); + i = end; + } + _ => { + let Some(&byte) = data.get(i) else { break }; + out.extend(std::iter::repeat_n(byte, 257 - usize::from(length))); + i += 1; + } + } + } + out +} + +/// §7.4.4.2's variable-code LZW, which is TIFF's rather than GIF's: codes grow +/// from nine bits, 256 clears the table and 257 ends the data. +/// +/// `early` is `/EarlyChange`, and it is not cosmetic — it decides whether the +/// code width grows one entry before the table is full. A decoder on the wrong +/// side of it produces plausible-looking rubbish rather than an error. +fn lzw(data: &[u8], early: bool) -> Result> { + const CLEAR: u16 = 256; + const END: u16 = 257; + + let mut table: Vec> = Vec::new(); + let reset = |table: &mut Vec>| { + table.clear(); + table.extend((0u16..=255).map(|b| vec![b as u8])); + table.push(Vec::new()); // 256, clear + table.push(Vec::new()); // 257, end + }; + reset(&mut table); + + let mut out = Vec::new(); + let mut width = 9u32; + let mut previous: Option = None; + let mut bits: u32 = 0; + let mut held: u32 = 0; + + for &byte in data { + bits = (bits << 8) | u32::from(byte); + held += 8; + while held >= width { + let code = ((bits >> (held - width)) & ((1 << width) - 1)) as u16; + held -= width; + + match code { + CLEAR => { + reset(&mut table); + width = 9; + previous = None; + } + END => return Ok(out), + _ => { + let entry = match table.get(usize::from(code)) { + Some(entry) => entry.clone(), + None => { + // The one legal forward reference: the code being + // defined by this very step. + let Some(previous) = previous.and_then(|p| table.get(usize::from(p))) + else { + return Err(Error::Format(format!( + "LZWDecode: code {code} is not in the table" + ))); + }; + let mut entry = previous.clone(); + let Some(&first) = previous.first() else { + return Err(Error::Format( + "LZWDecode: a code expanding to nothing".into(), + )); + }; + entry.push(first); + entry + } + }; + out.extend_from_slice(&entry); + if let Some(previous) = previous + && let Some(prefix) = table.get(usize::from(previous)) + { + let mut new = prefix.clone(); + new.push(entry[0]); + table.push(new); + } + previous = Some(code); + + let limit = table.len() as u32 + u32::from(early); + width = match limit { + ..512 => 9, + 512..1024 => 10, + 1024..2048 => 11, + _ => 12, + }; + if table.len() >= 4096 { + reset(&mut table); + width = 9; + previous = None; + } + } + } + } + } + // Data that ran out before its end-of-data code: keep what decoded, for the + // reason the module note gives. + Ok(out) +} diff --git a/crates/mirsam-pdf/src/lexer.rs b/crates/mirsam-pdf/src/lexer.rs new file mode 100644 index 0000000..6cf7b9c --- /dev/null +++ b/crates/mirsam-pdf/src/lexer.rs @@ -0,0 +1,485 @@ +//! Bytes to objects: the syntax half of ISO 32000-1 §7.2 and §7.3. +//! +//! A borrowing scanner over the whole file, so an object can be read at an +//! offset the cross-reference table names without copying anything before it. +//! +//! ## A stream's length is checked, not believed +//! +//! `/Length` may be an indirect reference — `<< /Length 12 0 R >>` — which is +//! a value the lexer cannot resolve, because resolving it means consulting the +//! cross-reference table that is itself being parsed out of streams. Worse, +//! plenty of real writers get the direct form *wrong* by a byte or two. +//! +//! So the length is treated as a hint and verified: if `/Length` is a direct +//! integer and the bytes it names are followed by `endstream`, it is taken. In +//! every other case — indirect, absent, or simply wrong — the data runs to the +//! next `endstream`. One rule covers all three, and no caller has to hand the +//! lexer a resolver it cannot have yet. + +use mirsam_core::error::{Error, Result}; + +use crate::object::{Dictionary, Name, Object, ObjectId, Stream, is_regular, is_whitespace}; + +/// How deep a nesting of arrays and dictionaries is read before the file is +/// called malformed. Well past anything a document contains, and short of the +/// depth at which a hand-written adversarial file would exhaust the stack. +const MAX_DEPTH: usize = 128; + +pub struct Lexer<'a> { + data: &'a [u8], + pos: usize, +} + +/// A parse failure, always carrying the byte offset it happened at: an object +/// layer that reported "malformed dictionary" without saying where would be +/// unusable against a document nobody can open in an editor. +fn malformed(at: usize, what: impl std::fmt::Display) -> Error { + Error::Format(format!("at byte {at}: {what}")) +} + +impl<'a> Lexer<'a> { + pub fn new(data: &'a [u8]) -> Self { + Self { data, pos: 0 } + } + + /// A lexer positioned at `pos`, which is what a cross-reference entry + /// hands over. + pub fn at(data: &'a [u8], pos: usize) -> Self { + Self { data, pos } + } + + pub fn pos(&self) -> usize { + self.pos + } + + pub fn seek(&mut self, pos: usize) { + self.pos = pos.min(self.data.len()); + } + + fn peek(&self) -> Option { + self.data.get(self.pos).copied() + } + + fn peek_at(&self, ahead: usize) -> Option { + self.data.get(self.pos + ahead).copied() + } + + /// White space and comments, which §7.2.4 says are equivalent to a single + /// space anywhere a token boundary is allowed. + pub fn skip_space(&mut self) { + loop { + match self.peek() { + Some(b) if is_whitespace(b) => self.pos += 1, + Some(b'%') => { + while let Some(b) = self.peek() { + if b == b'\r' || b == b'\n' { + break; + } + self.pos += 1; + } + } + _ => return, + } + } + } + + /// Consume `word` if it is the next token, and answer whether it was. + /// + /// The token has to *end* where the keyword does: without that check, + /// `endstream` would match a lexer looking for `end`, and `obj` would + /// match inside `objstm`. + pub fn keyword(&mut self, word: &str) -> bool { + self.skip_space(); + let end = self.pos + word.len(); + if self.data.get(self.pos..end) != Some(word.as_bytes()) { + return false; + } + if self.data.get(end).copied().is_some_and(is_regular) { + return false; + } + self.pos = end; + true + } + + /// The run of regular characters at the cursor: a keyword, a number, or + /// whatever a malformed file has there instead. + fn token(&mut self) -> &'a [u8] { + self.skip_space(); + let start = self.pos; + while self.peek().is_some_and(is_regular) { + self.pos += 1; + } + &self.data[start..self.pos] + } + + /// One object, following a reference no further than recognising it. + pub fn object(&mut self) -> Result { + self.object_at_depth(0) + } + + fn object_at_depth(&mut self, depth: usize) -> Result { + if depth > MAX_DEPTH { + return Err(malformed(self.pos, "objects nested past any real document")); + } + self.skip_space(); + let at = self.pos; + match self.peek() { + None => Err(malformed(at, "expected an object, found end of file")), + Some(b'/') => Ok(Object::Name(self.name()?)), + Some(b'(') => Ok(Object::String(self.literal_string()?)), + Some(b'[') => { + self.pos += 1; + let mut items = Vec::new(); + loop { + self.skip_space(); + match self.peek() { + Some(b']') => { + self.pos += 1; + return Ok(Object::Array(items)); + } + None => return Err(malformed(at, "array with no closing `]`")), + _ => items.push(self.object_at_depth(depth + 1)?), + } + } + } + Some(b'<') if self.peek_at(1) == Some(b'<') => self.dictionary_or_stream(depth), + Some(b'<') => Ok(Object::String(self.hex_string()?)), + Some(b) if b.is_ascii_digit() || matches!(b, b'+' | b'-' | b'.') => self.numeric(), + Some(stray @ (b']' | b'>' | b')' | b'}')) => { + Err(malformed(at, format!("stray `{}`", stray as char))) + } + Some(_) => match self.token() { + b"true" => Ok(Object::Boolean(true)), + b"false" => Ok(Object::Boolean(false)), + b"null" => Ok(Object::Null), + [] => Err(malformed(at, "expected an object, found a delimiter")), + other => Err(malformed( + at, + format!("unknown keyword {:?}", String::from_utf8_lossy(other)), + )), + }, + } + } + + /// A number, or the `n g R` reference two of them may begin. + /// + /// Reference recognition is lookahead with a rewind rather than a + /// post-pass over a token list: `1 0 R` and `1 0` differ only in what + /// follows, and every context that admits one admits the other. + fn numeric(&mut self) -> Result { + let at = self.pos; + let first = number(self.token(), at)?; + + if let Object::Integer(number) = first + && let Ok(number) = u32::try_from(number) + { + let rewind = self.pos; + let generation = self.token(); + if !generation.is_empty() + && generation.iter().all(u8::is_ascii_digit) + && let Ok(generation) = String::from_utf8_lossy(generation).parse::() + && self.keyword("R") + { + return Ok(Object::Reference(ObjectId::new(number, generation))); + } + self.pos = rewind; + } + Ok(first) + } + + fn name(&mut self) -> Result { + let at = self.pos; + self.pos += 1; // the `/` + let mut bytes = Vec::new(); + while let Some(b) = self.peek() { + if !is_regular(b) { + break; + } + self.pos += 1; + if b != b'#' { + bytes.push(b); + continue; + } + // `#xx`. A `#` not followed by two hex digits is kept literally: + // the alternative is failing a whole document over one byte a + // writer forgot to escape, and every reader in the world opens it. + match (self.peek(), self.peek_at(1)) { + (Some(hi), Some(lo)) if hi.is_ascii_hexdigit() && lo.is_ascii_hexdigit() => { + bytes.push(hex(hi) * 16 + hex(lo)); + self.pos += 2; + } + _ => bytes.push(b'#'), + } + } + if bytes.is_empty() && self.pos == at + 1 { + // `/` alone is the empty name, which is legal. + } + Ok(Name::new(bytes)) + } + + /// `( ... )`, with §7.3.4.2's escapes and balanced inner parentheses. + fn literal_string(&mut self) -> Result> { + let at = self.pos; + self.pos += 1; // the `(` + let mut bytes = Vec::new(); + let mut depth = 1usize; + while let Some(b) = self.peek() { + self.pos += 1; + match b { + b'(' => { + depth += 1; + bytes.push(b); + } + b')' => { + depth -= 1; + if depth == 0 { + return Ok(bytes); + } + bytes.push(b); + } + b'\\' => { + let Some(escape) = self.peek() else { break }; + self.pos += 1; + match escape { + b'n' => bytes.push(b'\n'), + b'r' => bytes.push(b'\r'), + b't' => bytes.push(b'\t'), + b'b' => bytes.push(0x08), + b'f' => bytes.push(0x0c), + b'(' | b')' | b'\\' => bytes.push(escape), + // A backslash before an end-of-line joins the lines and + // contributes nothing. + b'\n' => {} + b'\r' => { + if self.peek() == Some(b'\n') { + self.pos += 1; + } + } + b'0'..=b'7' => { + let mut value = u32::from(escape - b'0'); + for _ in 0..2 { + match self.peek() { + Some(d @ b'0'..=b'7') => { + value = value * 8 + u32::from(d - b'0'); + self.pos += 1; + } + _ => break, + } + } + // §7.3.4.2: high-order overflow is ignored. + bytes.push((value & 0xff) as u8); + } + // "\q" is q. The backslash goes, the byte stays. + other => bytes.push(other), + } + } + _ => bytes.push(b), + } + } + Err(malformed(at, "string with no closing `)`")) + } + + /// `<...>`, hexadecimal. White space between digits is ignored and an odd + /// final digit is padded with a zero, both per §7.3.4.3. + fn hex_string(&mut self) -> Result> { + let at = self.pos; + self.pos += 1; // the `<` + let mut bytes = Vec::new(); + let mut half: Option = None; + while let Some(b) = self.peek() { + self.pos += 1; + match b { + b'>' => { + if let Some(hi) = half { + bytes.push(hi * 16); + } + return Ok(bytes); + } + b if b.is_ascii_hexdigit() => match half.take() { + Some(hi) => bytes.push(hi * 16 + hex(b)), + None => half = Some(hex(b)), + }, + b if is_whitespace(b) => {} + other => { + return Err(malformed( + self.pos - 1, + format!("`{}` in a hexadecimal string", other as char), + )); + } + } + } + Err(malformed(at, "hexadecimal string with no closing `>`")) + } + + fn dictionary_or_stream(&mut self, depth: usize) -> Result { + let at = self.pos; + self.pos += 2; // the `<<` + let mut dict = Dictionary::new(); + loop { + self.skip_space(); + match self.peek() { + Some(b'>') if self.peek_at(1) == Some(b'>') => { + self.pos += 2; + break; + } + Some(b'/') => { + let key = self.name()?; + let value = self.object_at_depth(depth + 1)?; + dict.insert(key, value); + } + None => return Err(malformed(at, "dictionary with no closing `>>`")), + Some(other) => { + return Err(malformed( + self.pos, + format!("expected a key, found `{}`", other as char), + )); + } + } + } + + let rewind = self.pos; + if !self.keyword("stream") { + self.pos = rewind; + return Ok(Object::Dictionary(dict)); + } + // §7.3.8.1: CRLF or LF after the keyword, never CR alone. A file that + // uses CR alone is still read: dropping it costs nothing, and treating + // it as data would shift every byte of the stream. + match (self.peek(), self.peek_at(1)) { + (Some(b'\r'), Some(b'\n')) => self.pos += 2, + (Some(b'\n'), _) | (Some(b'\r'), _) => self.pos += 1, + _ => {} + } + let data = self.stream_data(&dict)?; + Ok(Object::Stream(Stream { dict, data })) + } + + /// The bytes between `stream` and `endstream`, leaving the cursor past the + /// closing keyword. See the module note on why `/Length` is verified. + fn stream_data(&mut self, dict: &Dictionary) -> Result> { + let start = self.pos; + + if let Some(length) = dict.get("Length").and_then(Object::as_i64) + && let Ok(length) = usize::try_from(length) + && let Some(end) = start.checked_add(length) + && end <= self.data.len() + { + let mut probe = Lexer::at(self.data, end); + if probe.keyword("endstream") { + self.pos = probe.pos; + return Ok(self.data[start..end].to_vec()); + } + } + + let Some(found) = find(&self.data[start..], b"endstream") else { + return Err(malformed(start, "stream with no `endstream`")); + }; + let mut end = start + found; + self.pos = end + b"endstream".len(); + // The end-of-line before `endstream` belongs to the syntax, not to the + // data, and only that one: a stream may legitimately end in a newline. + if end > start && self.data[end - 1] == b'\n' { + end -= 1; + } + if end > start && self.data[end - 1] == b'\r' { + end -= 1; + } + Ok(self.data[start..end].to_vec()) + } + + /// `n g obj endobj`, at the cursor. + /// + /// `endobj` is not required: a writer that omits it produces a file every + /// reader opens, and the object is complete without it. + pub fn indirect(&mut self) -> Result<(ObjectId, Object)> { + self.skip_space(); + let at = self.pos; + let number = self.token(); + let generation = self.token(); + let header = ( + String::from_utf8_lossy(number).parse::(), + String::from_utf8_lossy(generation).parse::(), + ); + let (Ok(number), Ok(generation)) = header else { + return Err(malformed(at, "expected ` obj`")); + }; + if !self.keyword("obj") { + return Err(malformed(at, "expected `obj` after the object number")); + } + let object = self.object()?; + let rewind = self.pos; + if !self.keyword("endobj") { + self.pos = rewind; + } + Ok((ObjectId::new(number, generation), object)) + } +} + +fn hex(b: u8) -> u8 { + match b { + b'0'..=b'9' => b - b'0', + b'a'..=b'f' => b - b'a' + 10, + _ => b - b'A' + 10, + } +} + +/// A number token, which §7.3.3 spells more loosely than it looks: `4.`, `-.2` +/// and `34.5` are all legal, and writers emit `--5` and `6.-2` often enough +/// that every reader tolerates them. +fn number(token: &[u8], at: usize) -> Result { + if token.is_empty() { + return Err(malformed(at, "expected a number")); + } + let text = String::from_utf8_lossy(token); + if !token.contains(&b'.') + && let Ok(value) = text.parse::() + { + return Ok(Object::Integer(value)); + } + if let Ok(value) = text.parse::() { + return Ok(Object::Real(value)); + } + // Salvage the leading well-formed number, which is what the malformed + // spellings above amount to. `--5` is 0 with a stray sign, not a failure + // to read the file. + let mut cleaned = String::new(); + for (i, c) in text.chars().enumerate() { + match c { + '+' | '-' if i == 0 => cleaned.push(c), + '.' if !cleaned.contains('.') => cleaned.push(c), + '0'..='9' => cleaned.push(c), + _ => break, + } + } + match cleaned.trim_start_matches(['+', '-']) { + "" | "." => Ok(Object::Integer(0)), + _ => cleaned + .parse::() + .map(Object::Real) + .map_err(|_| malformed(at, format!("unreadable number {text:?}"))), + } +} + +/// The first offset of `needle` in `haystack`. +/// +/// A plain scan: the needles here are short keywords and the streams they are +/// looked for in are the exception rather than the rule, so a search structure +/// would cost more to build than it saves. +pub(crate) fn find(haystack: &[u8], needle: &[u8]) -> Option { + if needle.is_empty() || haystack.len() < needle.len() { + return None; + } + haystack + .windows(needle.len()) + .position(|window| window == needle) +} + +/// The last offset of `needle` in `haystack`. +pub(crate) fn rfind(haystack: &[u8], needle: &[u8]) -> Option { + if needle.is_empty() || haystack.len() < needle.len() { + return None; + } + haystack + .windows(needle.len()) + .rposition(|window| window == needle) +} diff --git a/crates/mirsam-pdf/src/lib.rs b/crates/mirsam-pdf/src/lib.rs new file mode 100644 index 0000000..6bdf23c --- /dev/null +++ b/crates/mirsam-pdf/src/lib.rs @@ -0,0 +1,54 @@ +//! PDF object layer for mirsam. +//! +//! The package layer of a format that is not a package. `mirsam-ooxml` opens a +//! ZIP and asks for a part by name; this crate opens a PDF and asks for an +//! object by number — from the body, from inside a compressed object stream, +//! or from a scan of the file when the cross-reference table has gone stale. +//! +//! What it does: classic cross-reference tables and cross-reference streams, +//! `/Prev` chains, hybrid-reference files, object streams, the general-purpose +//! stream filters, the page tree with its inheritable attributes, and content +//! streams decoded and joined. +//! +//! What it deliberately does not do: **decrypt**, **rasterise**, and **write**. +//! An encrypted document is refused by name at [`Pdf::open`], an image codec +//! is named rather than decoded, and there is no writer here or planned — a +//! broken Arabic PDF is rebuilt from its source document, not edited in place. +//! `DocumentWriter` is not implemented, and the absence is the design. +//! +//! ```no_run +//! use mirsam_pdf::Pdf; +//! +//! let pdf = Pdf::open(std::path::Path::new("report.pdf"))?; +//! for page in pdf.pages() { +//! println!("page {} — {} bytes of content", page.index + 1, page.content(&pdf).len()); +//! } +//! for source in pdf.unread() { +//! println!("not read: {source}"); +//! } +//! # Ok::<(), mirsam_core::error::Error>(()) +//! ``` +//! +//! Text extraction sits on top of this and is PLAN §6.2: what this crate hands +//! over is the object graph and the bytes of the content streams, never a +//! string. A PDF stores glyphs with positions, and turning those back into +//! text is a reconstruction with its own rules about what may honestly be +//! claimed. +//! +//! See [`docs/adr/0010`] for why the parser is written here rather than taken. +//! +//! [`docs/adr/0010`]: https://github.com/aenawi/mirsam/blob/main/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md + +#![forbid(unsafe_code)] + +pub mod document; +pub mod encrypt; +pub mod filter; +pub mod lexer; +pub mod object; +pub mod page; +pub mod xref; + +pub use document::{Pdf, Reconstruction, Unread}; +pub use object::{Dictionary, Name, Object, ObjectId, Stream}; +pub use page::Page; diff --git a/crates/mirsam-pdf/src/object.rs b/crates/mirsam-pdf/src/object.rs new file mode 100644 index 0000000..cdabfe4 --- /dev/null +++ b/crates/mirsam-pdf/src/object.rs @@ -0,0 +1,353 @@ +//! PDF's object types: what everything in the file is made of. +//! +//! ISO 32000-1 §7.3 gives eight basic types and two composites. This module is +//! the model alone — [`crate::lexer`] builds it out of bytes, [`crate::document`] +//! resolves the references between the pieces. +//! +//! ## A name is bytes, not a string +//! +//! §7.3.5 defines a name as a sequence of bytes with `#xx` escapes, and says +//! nothing about their encoding. A well-behaved writer emits ASCII, but a name +//! is also the one place a PDF stores a *key*, so decoding one lossily would +//! silently merge two dictionary entries that the file kept apart. [`Name`] +//! therefore holds the decoded bytes and compares on them; [`Name::as_str`] +//! offers the text form to whoever wants it and answers `None` rather than +//! guessing. + +use std::fmt; + +/// A PDF name: `/Type`, `/FlateDecode`, `/Root`. +/// +/// Holds the *decoded* bytes, so `/A#42` and `/AB` are one name, which is what +/// §7.3.5 requires and what a dictionary lookup depends on. +#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Default)] +pub struct Name(Vec); + +impl Name { + pub fn new(bytes: impl Into>) -> Self { + Self(bytes.into()) + } + + pub fn as_bytes(&self) -> &[u8] { + &self.0 + } + + /// The name as text, or `None` when it is not UTF-8. + /// + /// Never lossy: a caller comparing names compares bytes, and one printing + /// a name gets [`fmt::Display`], which escapes rather than replaces. + pub fn as_str(&self) -> Option<&str> { + std::str::from_utf8(&self.0).ok() + } + + /// Whether this is the name `other` spells. + pub fn is(&self, other: &str) -> bool { + self.0 == other.as_bytes() + } +} + +impl From<&str> for Name { + fn from(s: &str) -> Self { + Self(s.as_bytes().to_vec()) + } +} + +impl PartialEq for Name { + fn eq(&self, other: &str) -> bool { + self.is(other) + } +} + +/// Written the way the file writes it, with the escapes a PDF requires — so a +/// name in a diagnostic can be searched for in the document it came from. +impl fmt::Display for Name { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("/")?; + for &b in &self.0 { + match b { + b'!'..=b'~' if !is_delimiter(b) && b != b'#' => { + f.write_str(std::str::from_utf8(&[b]).unwrap_or("?"))? + } + _ => write!(f, "#{b:02X}")?, + } + } + Ok(()) + } +} + +impl fmt::Debug for Name { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{self}") + } +} + +/// The characters §7.2.2 reserves as delimiters. Everything else outside +/// whitespace is a regular character, and may appear in a name unescaped. +pub(crate) fn is_delimiter(b: u8) -> bool { + matches!( + b, + b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%' + ) +} + +/// The six bytes §7.2.2 calls white space. The null is one of them, which is +/// why a byte-level scanner cannot use `u8::is_ascii_whitespace` here. +pub(crate) fn is_whitespace(b: u8) -> bool { + matches!(b, 0 | b'\t' | b'\n' | 0x0c | b'\r' | b' ') +} + +/// Neither white space nor a delimiter: the bytes a name, a number or a +/// keyword is made of. +pub(crate) fn is_regular(b: u8) -> bool { + !is_whitespace(b) && !is_delimiter(b) +} + +/// The address of an indirect object: `12 0 R`. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Debug, Default)] +pub struct ObjectId { + pub number: u32, + pub generation: u16, +} + +impl ObjectId { + pub fn new(number: u32, generation: u16) -> Self { + Self { number, generation } + } +} + +/// Printed as the file writes a reference to it, so an `unread` entry names an +/// object in the words a person would use to find it in the file. +impl fmt::Display for ObjectId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} {} R", self.number, self.generation) + } +} + +/// A dictionary: an ordered list of name/value pairs. +/// +/// A `Vec` rather than a map, for two reasons. A PDF dictionary is small — +/// tens of entries at the outside — so a linear scan beats hashing, and file +/// order is preserved, which keeps a dictionary printed back in a diagnostic +/// looking like the one in the document. +/// +/// **A repeated key keeps the last value.** §7.3.7 leaves the case undefined. +/// Last wins because the only way a duplicate arises in practice is a writer +/// appending an entry it meant to supersede an earlier one. +#[derive(Clone, PartialEq, Default)] +pub struct Dictionary(Vec<(Name, Object)>); + +impl Dictionary { + pub fn new() -> Self { + Self::default() + } + + /// A dictionary with no entries, in a `const` context — what a document + /// hands back for a dictionary it does not have, so no caller has to + /// distinguish "no `/Resources`" from "an empty one". + pub const fn empty() -> Self { + Self(Vec::new()) + } + + /// The value stored under `key`, unresolved: a `/Length` that the file + /// wrote as `12 0 R` comes back as [`Object::Reference`]. Use + /// [`crate::Pdf::get`] to follow it. + pub fn get(&self, key: &str) -> Option<&Object> { + self.0 + .iter() + .find(|(name, _)| name.is(key)) + .map(|(_, value)| value) + } + + pub fn contains_key(&self, key: &str) -> bool { + self.get(key).is_some() + } + + pub fn insert(&mut self, name: Name, value: Object) { + match self.0.iter_mut().find(|(existing, _)| *existing == name) { + Some(slot) => slot.1 = value, + None => self.0.push((name, value)), + } + } + + pub fn iter(&self) -> impl Iterator { + self.0.iter().map(|(name, value)| (name, value)) + } + + pub fn len(&self) -> usize { + self.0.len() + } + + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl FromIterator<(Name, Object)> for Dictionary { + fn from_iter>(iter: I) -> Self { + let mut dict = Self::new(); + for (name, value) in iter { + dict.insert(name, value); + } + dict + } +} + +impl fmt::Debug for Dictionary { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("<<")?; + for (name, value) in &self.0 { + write!(f, " {name} {value:?}")?; + } + f.write_str(" >>") + } +} + +/// A stream: a dictionary and the bytes after its `stream` keyword. +/// +/// **`data` is what the file holds, still filtered.** Decoding is +/// [`crate::Pdf::decoded`]'s, and it is deliberately not done here: a stream +/// whose filter this crate does not implement has to be *named* rather than +/// failed, and only the document knows how to record that. +#[derive(Clone, PartialEq)] +pub struct Stream { + pub dict: Dictionary, + pub data: Vec, +} + +impl fmt::Debug for Stream { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{:?} stream({} bytes)", self.dict, self.data.len()) + } +} + +/// Any PDF object. +#[derive(Clone, PartialEq)] +pub enum Object { + Null, + Boolean(bool), + Integer(i64), + Real(f64), + /// A literal or hexadecimal string, decoded to the bytes it denotes. + /// Still bytes: which of PDFDocEncoding, UTF-16BE or a font's own codes + /// they are depends on where the string was found. + String(Vec), + Name(Name), + Array(Vec), + Dictionary(Dictionary), + Stream(Stream), + Reference(ObjectId), +} + +impl Object { + pub fn is_null(&self) -> bool { + matches!(self, Self::Null) + } + + pub fn as_bool(&self) -> Option { + match self { + Self::Boolean(b) => Some(*b), + _ => None, + } + } + + /// An integer, or a real that names one exactly. A PDF writer is free to + /// spell a count `3.0`, and a reader that refused it would reject the file + /// over its punctuation. + pub fn as_i64(&self) -> Option { + match self { + Self::Integer(n) => Some(*n), + Self::Real(r) if r.fract() == 0.0 && r.is_finite() => Some(*r as i64), + _ => None, + } + } + + pub fn as_f64(&self) -> Option { + match self { + Self::Integer(n) => Some(*n as f64), + Self::Real(r) => Some(*r), + _ => None, + } + } + + pub fn as_string(&self) -> Option<&[u8]> { + match self { + Self::String(bytes) => Some(bytes), + _ => None, + } + } + + pub fn as_name(&self) -> Option<&Name> { + match self { + Self::Name(name) => Some(name), + _ => None, + } + } + + pub fn as_array(&self) -> Option<&[Object]> { + match self { + Self::Array(items) => Some(items), + _ => None, + } + } + + /// The dictionary of a dictionary *or* of a stream. + /// + /// A stream is a dictionary with bytes attached, and every caller that + /// wants `/Type` or `/Filter` wants the same answer from both. + pub fn as_dict(&self) -> Option<&Dictionary> { + match self { + Self::Dictionary(dict) => Some(dict), + Self::Stream(stream) => Some(&stream.dict), + _ => None, + } + } + + pub fn as_stream(&self) -> Option<&Stream> { + match self { + Self::Stream(stream) => Some(stream), + _ => None, + } + } + + pub fn as_reference(&self) -> Option { + match self { + Self::Reference(id) => Some(*id), + _ => None, + } + } + + /// Whether this object's `/Type` is `name`. + /// + /// Answers `false` for a dictionary with no `/Type` at all: several are + /// optional in PDF, so absence is not disagreement, and a caller that + /// wants to infer a type from an object's shape has to say so. + pub fn is_type(&self, name: &str) -> bool { + self.as_dict() + .and_then(|dict| dict.get("Type")) + .and_then(Object::as_name) + .is_some_and(|found| found.is(name)) + } +} + +impl fmt::Debug for Object { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Null => f.write_str("null"), + Self::Boolean(b) => write!(f, "{b}"), + Self::Integer(n) => write!(f, "{n}"), + Self::Real(r) => write!(f, "{r}"), + Self::String(bytes) => write!(f, "({} bytes)", bytes.len()), + Self::Name(name) => write!(f, "{name}"), + Self::Array(items) => { + f.write_str("[")?; + for item in items { + write!(f, " {item:?}")?; + } + f.write_str(" ]") + } + Self::Dictionary(dict) => write!(f, "{dict:?}"), + Self::Stream(stream) => write!(f, "{stream:?}"), + Self::Reference(id) => write!(f, "{id}"), + } + } +} diff --git a/crates/mirsam-pdf/src/page.rs b/crates/mirsam-pdf/src/page.rs new file mode 100644 index 0000000..ea1aa79 --- /dev/null +++ b/crates/mirsam-pdf/src/page.rs @@ -0,0 +1,184 @@ +//! The page tree: §7.7.3, and the four attributes a page inherits down it. +//! +//! A PDF's pages are the leaves of a tree of `/Pages` nodes, and four +//! attributes — `/Resources`, `/MediaBox`, `/CropBox` and `/Rotate` — may be +//! stated on any node and are inherited by everything below it. `/Resources` is +//! the one that matters here: it is where a page's fonts are, and a text layer +//! that read it from the leaf alone would find no fonts at all on the very +//! common document that states them once at the root. +//! +//! So the walk carries them down and a [`Page`] is handed over with them +//! already filled in. A page's own value always wins; an inherited one is only +//! ever an answer to a question the page did not answer itself. +//! +//! ## The tree is walked, and scanned for only if that fails +//! +//! `/Kids` can point at anything, including back up at itself. The walk keeps +//! a set of the nodes it has been through and stops at a repeat, which is the +//! difference between reading a malformed document and hanging on one. If it +//! yields nothing at all — no catalog `/Pages`, or a root whose kids are all +//! missing — the objects are scanned for `/Type /Page` instead, and the +//! document says so through [`crate::Reconstruction::PageTree`]. + +use std::collections::BTreeSet; + +use crate::document::Pdf; +use crate::object::{Dictionary, Name, Object, ObjectId}; + +/// §7.7.3.4's inheritable page attributes, in the order the specification +/// tabulates them. +const INHERITED: [&str; 4] = ["Resources", "MediaBox", "CropBox", "Rotate"]; + +/// How deep the tree is walked. A page tree is balanced by every writer that +/// exists; this is a bound on malice, not on documents. +const MAX_DEPTH: usize = 64; + +/// One page, with what it inherited already on it. +#[derive(Clone, Debug)] +pub struct Page { + /// Position in the document, from zero — which is the number a person + /// counts pages by, and not the `/PageLabels` the document may print. + pub index: usize, + pub id: ObjectId, + /// The page dictionary, plus any of the four inheritable attributes an + /// ancestor supplied and the page did not state. + pub dict: Dictionary, +} + +impl Page { + /// The page's resource dictionary, where its fonts are named. + pub fn resources<'a>(&'a self, pdf: &'a Pdf) -> Option<&'a Dictionary> { + pdf.get(&self.dict, "Resources").and_then(Object::as_dict) + } + + /// The page's content streams, decoded and joined. + /// + /// `/Contents` is one stream or an array of them, and §7.8.2 says an array + /// is to be treated as a single stream *concatenated at the boundaries* — + /// so the join is a newline, without which the last operator of one stream + /// and the first of the next would run together into a token neither of + /// them wrote. + /// + /// A part behind a filter this crate does not decode contributes nothing + /// and is recorded in [`Pdf::unread`]: a page drawn as a scanned image + /// comes back empty *and named*, never merely empty. + /// + /// [`Pdf::unread`]: crate::Pdf::unread + pub fn content(&self, pdf: &Pdf) -> Vec { + let Some(contents) = self.dict.get("Contents") else { + return Vec::new(); + }; + let parts: Vec<&Object> = match pdf.resolve(contents) { + Object::Array(items) => items.iter().collect(), + _ => vec![contents], + }; + + let mut out = Vec::new(); + for part in parts { + let Some(id) = part.as_reference() else { + // §7.8.2 requires a content stream to be indirect. A direct + // one is not something to fail over, but there is nothing to + // decode: a stream object cannot appear inline. + continue; + }; + match pdf.decoded(id) { + Ok(Some(bytes)) => { + out.extend_from_slice(&bytes); + out.push(b'\n'); + } + Ok(None) => {} + Err(e) => pdf.note_unread(Some(id), format!("content stream: {e}")), + } + } + out + } +} + +/// Every page, in reading order, and whether the tree had to be reconstructed. +pub(crate) fn tree(pdf: &Pdf) -> (Vec, bool) { + let mut pages = Vec::new(); + let mut seen = BTreeSet::new(); + + if let Some(root) = pdf.catalog().get("Pages") { + let inherited = Dictionary::new(); + descend(pdf, root, &inherited, 0, &mut seen, &mut pages); + } + if !pages.is_empty() { + return (pages, false); + } + + // Nothing walkable. Take the objects that call themselves pages, in object + // order — which is the order they were written, and the closest thing to + // reading order a file with no tree has. + let mut scanned = Vec::new(); + for (id, object) in pdf.objects() { + if !object.is_type("Page") { + continue; + } + let Some(dict) = object.as_dict() else { + continue; + }; + scanned.push(Page { + index: scanned.len(), + id, + dict: dict.clone(), + }); + } + let rebuilt = !scanned.is_empty(); + (scanned, rebuilt) +} + +fn descend( + pdf: &Pdf, + node: &Object, + inherited: &Dictionary, + depth: usize, + seen: &mut BTreeSet, + pages: &mut Vec, +) { + if depth > MAX_DEPTH { + return; + } + if let Some(id) = node.as_reference() + && !seen.insert(id) + { + return; + } + let id = node.as_reference().unwrap_or_default(); + let Some(dict) = pdf.resolve(node).as_dict() else { + return; + }; + + // What this node passes down: what it was given, overridden by what it + // states itself. + let mut passing = inherited.clone(); + for key in INHERITED { + if let Some(value) = dict.get(key) { + passing.insert(Name::from(key), value.clone()); + } + } + + // A node with `/Kids` is an internal node whatever its `/Type` says, and a + // great many writers omit `/Type` entirely. Reading the shape rather than + // the label is the only way to walk those files. + if let Some(kids) = pdf.get(dict, "Kids").and_then(Object::as_array) { + for kid in kids { + descend(pdf, kid, &passing, depth + 1, seen, pages); + } + return; + } + + let mut page = dict.clone(); + for key in INHERITED { + if !page.contains_key(key) + && let Some(value) = passing.get(key) + { + page.insert(Name::from(key), value.clone()); + } + } + pages.push(Page { + index: pages.len(), + id, + dict: page, + }); +} diff --git a/crates/mirsam-pdf/src/xref.rs b/crates/mirsam-pdf/src/xref.rs new file mode 100644 index 0000000..6b63b09 --- /dev/null +++ b/crates/mirsam-pdf/src/xref.rs @@ -0,0 +1,486 @@ +//! The cross-reference: where each object is, and how to find that out when +//! the file lies about it. +//! +//! §7.5.4 gives two spellings and every real corpus contains both. A classic +//! *table* is plain text — `xref`, subsection headers, twenty-byte entries — +//! and is what anything writing PDF 1.4 or earlier emits. A cross-reference +//! *stream* is a binary table inside a `FlateDecode`d stream, introduced with +//! PDF 1.5 so that the table itself could be compressed alongside the object +//! streams it indexes. Both may be chained by `/Prev` into an arbitrarily long +//! history of incremental updates, and a *hybrid* file (§7.5.8.4) carries one +//! of each so that a 1.4 reader sees a subset of the objects a 1.5 reader sees. +//! +//! ## Newest wins, and it is the walk that enforces it +//! +//! An incremental update appends a new section and points its `/Prev` at the +//! old one, so the chain runs newest to oldest. Every insertion here is +//! therefore *first wins*: an object redefined by an update is found in the +//! section that redefined it, and the older entry is never consulted. The same +//! rule merges the trailers, which is why a `/Root` moved by an update is the +//! one that is used. +//! +//! ## And when the table is wrong +//! +//! Offsets go stale. A file concatenated by a script, truncated by a transfer +//! or edited by hand has a `startxref` pointing at nothing in particular, and +//! every viewer in the world opens it anyway by scanning for `obj`. So does +//! [`recover`] — because a document this tool refused while a reader opened it +//! would be a document reported as unreadable that is not, which is the worse +//! half of standing rule 4. + +use mirsam_core::error::{Error, Result}; +use std::collections::{BTreeMap, BTreeSet}; + +use crate::filter::{self, Decoded}; +use crate::lexer::{Lexer, find, rfind}; +use crate::object::{Dictionary, Name, Object, is_regular, is_whitespace}; + +/// How long a `/Prev` chain is followed. An incremental update per save is +/// normal; ten thousand of them is a file built by a loop. +const MAX_SECTIONS: usize = 4096; + +/// How far back from the end of the file `startxref` is looked for. §7.5.5 +/// puts it in the last line; a kilobyte covers every writer that appends +/// something after it. +const TAIL: usize = 2048; + +/// Where an object is. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Slot { + /// At a byte offset in the file. + InFile { offset: u64, generation: u16 }, + /// Inside an object stream, at `index` among the objects it holds. + InStream { stream: u32, index: u32 }, + /// Deleted. Kept rather than dropped, because a free entry in a newer + /// section is what *hides* an object an older section still names. + Free, +} + +/// A resolved cross-reference: every object's location, and the merged trailer. +#[derive(Debug, Default)] +pub struct Xref { + pub slots: BTreeMap, + pub trailer: Dictionary, +} + +impl Xref { + /// Record `slot` for `number` unless a newer section already did. + fn note(&mut self, number: u32, slot: Slot) { + self.slots.entry(number).or_insert(slot); + } + + /// Merge a section's trailer under the same first-wins rule. + fn merge_trailer(&mut self, trailer: &Dictionary) { + for (key, value) in trailer.iter() { + // The chain's own links are a property of the section that carried + // them, never of the document. + if key.is("Prev") || key.is("XRefStm") { + continue; + } + if !self.trailer.contains_key(key.as_str().unwrap_or_default()) { + self.trailer.insert(key.clone(), value.clone()); + } + } + } + + /// The objects stored in object streams, grouped by the stream holding + /// them: loading one stream answers for every object in it. + pub fn in_streams(&self) -> BTreeMap> { + let mut grouped: BTreeMap> = BTreeMap::new(); + for (&number, slot) in &self.slots { + if let Slot::InStream { stream, index } = *slot { + grouped.entry(stream).or_default().push((number, index)); + } + } + grouped + } +} + +/// Follow the chain from `startxref` and answer with every object's location. +/// +/// `shift` is the offset of `%PDF-` within the file, which §7.5.5 makes the +/// origin every stored offset is relative to. It is zero for a well-formed +/// document and non-zero for one with a prologue bolted on the front. +pub fn read(data: &[u8], shift: usize) -> Result { + let Some(start) = startxref(data) else { + return Err(Error::Format( + "no `startxref`: the file does not say where its cross-reference table is".into(), + )); + }; + + let mut xref = Xref::default(); + let mut next = Some(start); + let mut seen = BTreeSet::new(); + + for _ in 0..MAX_SECTIONS { + let Some(offset) = next.take() else { break }; + let Some(at) = locate(data, offset, shift) else { + return Err(Error::Format(format!( + "cross-reference section at byte {offset} is not there" + ))); + }; + if !seen.insert(at) { + // A `/Prev` pointing into a section already walked. Two writers + // producing the same offset is a loop, not a longer history. + break; + } + + let section = section(data, at)?; + + // §7.5.8.4: in a hybrid file the stream is the authority for the + // section, and the classic table beside it deliberately understates + // what the file holds. Reading it first makes it win. + if let Some(hybrid) = section.hybrid + && let Some(hybrid_at) = locate(data, hybrid, shift) + && seen.insert(hybrid_at) + && let Ok(stream_section) = self::section(data, hybrid_at) + { + for (number, slot) in stream_section.slots { + xref.note(number, slot); + } + xref.merge_trailer(&stream_section.trailer); + } + + for (number, slot) in section.slots { + xref.note(number, slot); + } + xref.merge_trailer(§ion.trailer); + next = section.previous; + } + + if xref.slots.is_empty() { + return Err(Error::Format( + "the cross-reference table names no objects".into(), + )); + } + Ok(xref) +} + +/// Rebuild the table by scanning for ` obj`. +/// +/// Later definitions win, which is the same rule the chain walk enforces from +/// the other end: an incremental update appends, so the last definition of an +/// object number in the file is the current one. +pub fn recover(data: &[u8]) -> Xref { + let mut xref = Xref::default(); + let mut at = 0usize; + + while let Some(found) = find(&data[at..], b"obj") { + let keyword = at + found; + at = keyword + 3; + // `obj` has to be a token of its own: `objstm` is not one. + if data.get(at).copied().is_some_and(is_regular) { + continue; + } + let Some((number, generation, start)) = header_before(data, keyword) else { + continue; + }; + xref.slots.insert( + number, + Slot::InFile { + offset: start as u64, + generation, + }, + ); + } + + // Every `trailer` in the file, oldest first, so the newest wins. + let mut at = 0usize; + while let Some(found) = find(&data[at..], b"trailer") { + let keyword = at + found; + at = keyword + b"trailer".len(); + let mut lexer = Lexer::at(data, at); + if let Ok(Object::Dictionary(dict)) = lexer.object() { + for (key, value) in dict.iter() { + if key.is("Prev") || key.is("XRefStm") { + continue; + } + xref.trailer.insert(key.clone(), value.clone()); + } + } + } + xref +} + +/// Walk back from an `obj` keyword over ` `, answering the two numbers +/// and where the indirect object begins. +fn header_before(data: &[u8], keyword: usize) -> Option<(u32, u16, usize)> { + let mut end = keyword; + let back = |end: &mut usize| -> Option<(usize, usize)> { + while *end > 0 && is_whitespace(data[*end - 1]) { + *end -= 1; + } + let stop = *end; + while *end > 0 && data[*end - 1].is_ascii_digit() { + *end -= 1; + } + (*end < stop).then_some((*end, stop)) + }; + + let (generation_start, generation_end) = back(&mut end)?; + let (number_start, number_end) = back(&mut end)?; + // The number must start a token: `x12 0 obj` is not an object header. + if number_start > 0 && is_regular(data[number_start - 1]) { + return None; + } + let number = std::str::from_utf8(&data[number_start..number_end]) + .ok()? + .parse::() + .ok()?; + let generation = std::str::from_utf8(&data[generation_start..generation_end]) + .ok()? + .parse::() + .ok()?; + Some((number, generation, number_start)) +} + +/// The offset of the last `startxref` value in the file. +fn startxref(data: &[u8]) -> Option { + let from = data.len().saturating_sub(TAIL); + let found = rfind(&data[from..], b"startxref")? + from; + let mut lexer = Lexer::at(data, found + b"startxref".len()); + lexer.object().ok()?.as_i64()?.try_into().ok() +} + +/// Turn a stored offset into a position in this file. +/// +/// A file with bytes before its `%PDF-` header stores offsets relative to the +/// header, and one whose prologue was added *after* it was written stores them +/// relative to the file. Both are common; both are tried, and the one that +/// lands on something that begins a cross-reference section wins. +fn locate(data: &[u8], offset: u64, shift: usize) -> Option { + let candidates = [ + usize::try_from(offset).ok()?.checked_add(shift)?, + usize::try_from(offset).ok()?, + ]; + for at in candidates { + if at >= data.len() { + continue; + } + let mut lexer = Lexer::at(data, at); + let rewind = lexer.pos(); + if lexer.keyword("xref") { + return Some(at); + } + lexer.seek(rewind); + if lexer.indirect().is_ok() { + return Some(at); + } + } + None +} + +/// One section of the chain, before it is merged into the whole. +struct Section { + slots: Vec<(u32, Slot)>, + trailer: Dictionary, + previous: Option, + /// `/XRefStm`, the cross-reference stream a hybrid file hides beside its + /// classic table. + hybrid: Option, +} + +fn section(data: &[u8], at: usize) -> Result
{ + let mut lexer = Lexer::at(data, at); + if lexer.keyword("xref") { + return classic(data, lexer); + } + lexer.seek(at); + let (_, object) = lexer.indirect()?; + let Some(stream) = object.as_stream() else { + return Err(Error::Format(format!( + "byte {at} begins neither an `xref` table nor a cross-reference stream" + ))); + }; + stream_section(stream) +} + +/// The plain-text table: subsection headers, then twenty-byte entries. +fn classic(data: &[u8], mut lexer: Lexer<'_>) -> Result
{ + let mut slots = Vec::new(); + loop { + let rewind = lexer.pos(); + if lexer.keyword("trailer") { + break; + } + lexer.seek(rewind); + let Ok(Some(first)) = lexer.object().map(|o| o.as_i64()) else { + lexer.seek(rewind); + break; + }; + let Ok(Some(count)) = lexer.object().map(|o| o.as_i64()) else { + return Err(Error::Format(format!( + "cross-reference subsection at byte {rewind} has no entry count" + ))); + }; + let Ok(first) = u32::try_from(first) else { + return Err(Error::Format(format!( + "cross-reference subsection at byte {rewind} starts at object {first}" + ))); + }; + + for index in 0..count.max(0) { + let offset = lexer.object()?.as_i64().unwrap_or(-1); + let generation = lexer.object()?.as_i64().unwrap_or(0); + lexer.skip_space(); + let kind = data.get(lexer.pos()).copied().unwrap_or(b'n'); + lexer.seek(lexer.pos() + 1); + + let Some(number) = first.checked_add(index as u32) else { + break; + }; + let slot = match kind { + b'f' => Slot::Free, + _ => match (u64::try_from(offset), u16::try_from(generation)) { + (Ok(offset), Ok(generation)) => Slot::InFile { offset, generation }, + // An entry a reader cannot use is a free one: it names no + // place in the file, and treating it as one would send the + // object loader to byte zero. + _ => Slot::Free, + }, + }; + slots.push((number, slot)); + } + } + + let trailer = match lexer.object()? { + Object::Dictionary(dict) => dict, + other => { + return Err(Error::Format(format!( + "expected a trailer dictionary, found {other:?}" + ))); + } + }; + Ok(Section { + previous: trailer + .get("Prev") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + hybrid: trailer + .get("XRefStm") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + slots, + trailer, + }) +} + +/// The binary table: `/W` gives each field's width in bytes, `/Index` which +/// object numbers the rows are for. +fn stream_section(stream: &crate::object::Stream) -> Result
{ + let dict = &stream.dict; + let bytes = match decode_directly(stream)? { + Decoded::Bytes(bytes) => bytes, + Decoded::Stopped { filter } => { + return Err(Error::Format(format!( + "cross-reference stream is behind {filter}, which is not a filter this tool decodes" + ))); + } + }; + + let widths: Vec = dict + .get("W") + .and_then(Object::as_array) + .ok_or_else(|| Error::Format("cross-reference stream has no /W".into()))? + .iter() + .map(|w| w.as_i64().unwrap_or(0).clamp(0, 8) as usize) + .collect(); + if widths.len() < 3 { + return Err(Error::Format(format!( + "cross-reference stream /W has {} fields, and the format has three", + widths.len() + ))); + } + let row = widths.iter().sum::(); + if row == 0 { + return Err(Error::Format( + "cross-reference stream /W gives every field a width of zero".into(), + )); + } + + // §7.5.8.2: /Index defaults to the whole of /Size. + let size = dict.get("Size").and_then(Object::as_i64).unwrap_or(0); + let index: Vec = match dict.get("Index").and_then(Object::as_array) { + Some(pairs) => pairs.iter().map(|p| p.as_i64().unwrap_or(0)).collect(), + None => vec![0, size], + }; + + let mut slots = Vec::new(); + let mut cursor = 0usize; + for pair in index.chunks(2) { + let (&first, &count) = match pair { + [first, count] => (first, count), + _ => break, + }; + for i in 0..count.max(0) { + if cursor + row > bytes.len() { + break; + } + let mut fields = [0u64; 3]; + let mut at = cursor; + for (field, &width) in fields.iter_mut().zip(&widths) { + for _ in 0..width { + *field = (*field << 8) | u64::from(bytes[at]); + at += 1; + } + } + // A zero-width first field means type 1, per §7.5.8.2. + if widths[0] == 0 { + fields[0] = 1; + } + cursor += row; + + let Ok(number) = u32::try_from(first + i) else { + continue; + }; + let slot = match fields[0] { + 0 => Slot::Free, + 1 => Slot::InFile { + offset: fields[1], + generation: u16::try_from(fields[2]).unwrap_or(0), + }, + 2 => match (u32::try_from(fields[1]), u32::try_from(fields[2])) { + (Ok(stream), Ok(index)) => Slot::InStream { stream, index }, + _ => continue, + }, + // §7.5.8.3: a type this reader does not know is to be treated + // as free rather than guessed at. + _ => Slot::Free, + }; + slots.push((number, slot)); + } + } + + Ok(Section { + previous: dict + .get("Prev") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + hybrid: None, + slots, + trailer: dict.clone(), + }) +} + +/// Decode a stream whose dictionary cannot contain indirect references. +/// +/// A cross-reference stream is the one stream that has to be read before the +/// table exists, so §7.5.8.2 requires its `/Filter`, `/W` and `/Index` to be +/// direct. That is what makes this function possible, and why the general path +/// through [`crate::Pdf::decoded`] is not used here. +fn decode_directly(stream: &crate::object::Stream) -> Result { + let filters = match stream.dict.get("Filter") { + Some(Object::Name(name)) => vec![name.clone()], + Some(Object::Array(items)) => items.iter().filter_map(Object::as_name).cloned().collect(), + _ => Vec::::new(), + }; + let parms = match stream.dict.get("DecodeParms") { + Some(Object::Dictionary(dict)) => vec![dict.clone()], + Some(Object::Array(items)) => items + .iter() + .map(|item| item.as_dict().cloned().unwrap_or_default()) + .collect(), + _ => Vec::new(), + }; + filter::decode(&stream.data, &filters, &parms) +} diff --git a/crates/mirsam-pdf/tests/corpus.rs b/crates/mirsam-pdf/tests/corpus.rs new file mode 100644 index 0000000..0ec2803 --- /dev/null +++ b/crates/mirsam-pdf/tests/corpus.rs @@ -0,0 +1,197 @@ +//! The fixture corpus, one file per generator family. +//! +//! `scripts/make-pdf-fixture.py` writes these byte by byte; `make pdfs` +//! regenerates them. Each asserts the *whole* of what the object layer owes a +//! caller for that family — the pages come back, the content decodes, the +//! Arabic survives, and nothing is silently missing — because a test that only +//! checked the file opened would pass on a reader that returned an empty +//! document. + +use mirsam_pdf::{Pdf, Reconstruction}; +use std::path::PathBuf; + +fn fixture(name: &str) -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests/pdfs") + .join(name) +} + +fn open(name: &str) -> Pdf { + Pdf::open(&fixture(name)).unwrap_or_else(|e| panic!("{name}: {e}")) +} + +/// How the fixtures write the Arabic: one byte per character, as a hexadecimal +/// string. Fourteen codes for the fourteen characters of `\u{627}\u{644}\u{62a}\u{642}\u{631}\u{64a}\u{631} +/// \u{627}\u{644}\u{633}\u{646}\u{648}\u{64a}`, which is the shape a generator that subset its font +/// produces and the shape PLAN \u{a7}6.3 hunts a reversal in. +const ARABIC_RUN: &[u8] = b"<4142434445464748494A4B4C4D4E>"; + +fn content(pdf: &Pdf) -> Vec { + pdf.pages()[0].content(pdf) +} + +#[test] +fn classic_table_uncompressed() { + let pdf = open("classic.pdf"); + assert_eq!(pdf.version(), "1.4"); + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.reconstructed().is_empty(), "nothing needed rebuilding"); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + let content = content(&pdf); + assert!(content.starts_with(b"BT\n")); + assert!( + content.contains_str(ARABIC_RUN), + "the Arabic run is not in the content stream" + ); +} + +#[test] +fn cross_reference_stream_and_object_stream() { + let pdf = open("compressed.pdf"); + assert_eq!(pdf.version(), "1.7"); + assert!(pdf.reconstructed().is_empty(), "{:?}", pdf.reconstructed()); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + // The catalog, the page tree node, the page and the font all came out of + // the object stream: a reader with no inflate finds none of them. + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.catalog().contains_key("Pages")); + let content = content(&pdf); + assert!( + content.starts_with(b"BT\n"), + "{:?}", + &content[..8.min(content.len())] + ); + assert!(content.contains_str(ARABIC_RUN)); +} + +#[test] +fn hybrid_file_prefers_the_stream() { + let pdf = open("hybrid.pdf"); + assert_eq!(pdf.pages().len(), 1); + // Object 6 is the ToUnicode CMap, and only the /XRefStm says where it is. + // A reader that stopped at the classic table has five objects and a font + // that says nothing about what it draws. + assert!( + pdf.object(mirsam_pdf::ObjectId::new(6, 0)).is_some(), + "the object only the /XRefStm names was not found" + ); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); +} + +#[test] +fn an_incremental_update_supersedes_what_it_replaces() { + let base = open("classic.pdf"); + let updated = open("incremental.pdf"); + + let before = content(&base); + let after = content(&updated); + assert_ne!(before, after, "the update changed nothing"); + assert!( + after.len() > before.len(), + "the newer content stream is the longer one, and this is not it" + ); + assert!(updated.reconstructed().is_empty()); +} + +#[test] +fn a_stale_startxref_is_recovered_from() { + let pdf = open("damaged.pdf"); + assert_eq!( + pdf.reconstructed(), + [Reconstruction::CrossReferenceTable], + "the rebuild has to be said out loud" + ); + assert_eq!(pdf.pages().len(), 1); + assert_eq!(content(&pdf), content(&open("classic.pdf"))); +} + +#[test] +fn an_encrypted_document_is_refused_by_name() { + let error = Pdf::open(&fixture("encrypted.pdf")) + .unwrap_err() + .to_string(); + assert!(error.contains("AES-256"), "{error}"); + assert!(error.contains("/Standard security handler"), "{error}"); + assert!(error.contains("V5 R6"), "{error}"); + assert!( + error.contains("unread rather than clean"), + "the refusal has to say what it is not claiming: {error}" + ); +} + +#[test] +fn the_ascii_filters_a_distiller_writes() { + let pdf = open("ascii-filters.pdf"); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + // /Contents is an array of three streams behind three different filters, + // joined at their boundaries. + let content = content(&pdf); + assert!( + content.starts_with(b"BT\n"), + "ASCII85 then Flate did not unwind" + ); + assert!(content.contains_str(b"% run length\n"), "RunLengthDecode"); + assert!(content.contains_str(b"% lzw\n"), "LZWDecode"); + + // The ToUnicode CMap is behind ASCIIHexDecode. + let cmap = pdf + .decoded(mirsam_pdf::ObjectId::new(6, 0)) + .unwrap() + .unwrap(); + assert!(cmap.contains_str(b"beginbfchar")); +} + +#[test] +fn a_scanned_page_is_named_rather_than_called_clean() { + let pdf = open("image-only.pdf"); + assert_eq!(pdf.pages().len(), 1); + + // The page's own content stream reads perfectly well; it draws an image. + let content = content(&pdf); + assert!(content.contains_str(b"/Im1 Do")); + + // Asking for the image is where the honesty is: the bytes are not text + // this crate failed to read, they are text that is not in the file. + let image = pdf.decoded(mirsam_pdf::ObjectId::new(5, 0)).unwrap(); + assert!(image.is_none(), "an image codec must not come back decoded"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 1, "{unread:?}"); + assert!(unread[0].starts_with("5 0 R — /DCTDecode"), "{unread:?}"); +} + +#[test] +fn every_fixture_states_its_own_version_and_one_page() { + for name in [ + "classic.pdf", + "compressed.pdf", + "hybrid.pdf", + "incremental.pdf", + "damaged.pdf", + "ascii-filters.pdf", + "image-only.pdf", + ] { + let pdf = open(name); + assert!(pdf.version().starts_with("1."), "{name}: {}", pdf.version()); + assert_eq!(pdf.pages().len(), 1, "{name}"); + // MediaBox is stated on the /Pages node in every fixture, so a page + // that has one proves the inheritable attributes came down the tree. + assert!( + pdf.pages()[0].dict.contains_key("MediaBox"), + "{name}: the page did not inherit /MediaBox" + ); + } +} + +/// `slice::contains` is for one element; this is for a subslice. +trait ContainsStr { + fn contains_str(&self, needle: &[u8]) -> bool; +} + +impl ContainsStr for Vec { + fn contains_str(&self, needle: &[u8]) -> bool { + self.windows(needle.len()).any(|w| w == needle) + } +} diff --git a/crates/mirsam-pdf/tests/document.rs b/crates/mirsam-pdf/tests/document.rs new file mode 100644 index 0000000..351477d --- /dev/null +++ b/crates/mirsam-pdf/tests/document.rs @@ -0,0 +1,357 @@ +//! The object layer's own behaviours, on documents built for one question +//! each — the fixture corpus proves the generator families, and these prove +//! the rules the loader applies to all of them. + +use mirsam_pdf::object::{Name, Object, ObjectId}; +use mirsam_pdf::{Pdf, Reconstruction}; + +/// Assemble a PDF from object bodies, with a correct classic table. +/// +/// Written here rather than taken from the fixture script so that a test can +/// state one situation in six lines. Object 1 is the catalog by convention. +fn build(objects: &[(u32, &[u8])], trailer_extra: &str) -> Vec { + let mut out = b"%PDF-1.4\n".to_vec(); + let mut offsets = Vec::new(); + for (number, body) in objects { + offsets.push((*number, out.len())); + out.extend_from_slice(format!("{number} 0 obj\n").as_bytes()); + out.extend_from_slice(body); + out.extend_from_slice(b"\nendobj\n"); + } + let size = objects.iter().map(|(n, _)| *n).max().unwrap_or(0) + 1; + let startxref = out.len(); + out.extend_from_slice(format!("xref\n0 {size}\n0000000000 65535 f \n").as_bytes()); + for number in 1..size { + match offsets.iter().find(|(n, _)| *n == number) { + Some((_, at)) => out.extend_from_slice(format!("{at:010} 00000 n \n").as_bytes()), + None => out.extend_from_slice(b"0000000000 65535 f \n"), + } + } + out.extend_from_slice( + format!("trailer\n<< /Size {size} /Root 1 0 R{trailer_extra} >>\nstartxref\n{startxref}\n%%EOF\n") + .as_bytes(), + ); + out +} + +fn open(objects: &[(u32, &[u8])]) -> Pdf { + Pdf::from_bytes(&build(objects, ""), "test").expect("open") +} + +const CATALOG: &[u8] = b"<< /Type /Catalog /Pages 2 0 R >>"; + +fn rfind(haystack: &[u8], needle: &[u8]) -> Option { + haystack.windows(needle.len()).rposition(|w| w == needle) +} + +/// The cross-reference table's own `xref`, and not the one inside the +/// `startxref` that follows it. +fn table_at(data: &[u8]) -> usize { + rfind(data, b"\nxref\n").expect("the table") + 1 +} + +#[test] +fn a_file_with_no_header_is_not_a_pdf() { + let error = Pdf::from_bytes(b"just some bytes", "test") + .unwrap_err() + .to_string(); + assert!(error.contains("not a PDF"), "{error}"); +} + +/// A `%PDF-` header preceded by junk is common enough that every viewer +/// tolerates it, and the stored offsets may be relative to either origin. +#[test] +fn a_prologue_before_the_header_is_read_through() { + let mut data = b"#!/bin/sh\n# a self-extracting wrapper\n".to_vec(); + let body = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + data.extend_from_slice(&body); + let pdf = Pdf::from_bytes(&data, "wrapped").expect("open"); + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.reconstructed().is_empty(), "{:?}", pdf.reconstructed()); +} + +/// §7.7.3.4's inheritable attributes come down the tree. `/Resources` is the +/// one that matters: a text layer reading it from the leaf alone would find no +/// fonts on the very common document that states them once at the root. +#[test] +fn a_page_inherits_resources_and_the_boxes() { + let pdf = open(&[ + (1, CATALOG), + ( + 2, + b"<< /Type /Pages /Kids [4 0 R] /Count 1 /MediaBox [0 0 595 842] \ + /Resources << /Font << /F1 9 0 R >> >> /Rotate 90 >>", + ), + (4, b"<< /Type /Pages /Kids [3 0 R] /Count 1 /Rotate 0 >>"), + (3, b"<< /Type /Page /Parent 4 0 R >>"), + ]); + let page = &pdf.pages()[0]; + assert!( + page.resources(&pdf).is_some(), + "the fonts did not come down" + ); + assert!(page.dict.contains_key("MediaBox")); + // The nearer ancestor's value wins over the further one's. + assert_eq!( + pdf.get(&page.dict, "Rotate").and_then(Object::as_i64), + Some(0) + ); +} + +/// A page's own value is never overwritten by what it inherits. +#[test] +fn a_page_outranks_what_it_inherits() { + let pdf = open(&[ + (1, CATALOG), + ( + 2, + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", + ), + ( + 3, + b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] >>", + ), + ]); + let media = pdf.get(&pdf.pages()[0].dict, "MediaBox").unwrap(); + assert_eq!(media.as_array().unwrap()[2].as_i64(), Some(100)); +} + +/// A `/Pages` node with no `/Type` is still an internal node: a great many +/// writers omit it, and reading the shape rather than the label is the only +/// way to walk those files. +#[test] +fn a_node_with_kids_is_an_internal_node_whatever_it_says() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ]); + assert_eq!(pdf.pages().len(), 1); +} + +/// `/Kids` pointing back up the tree is a file, not a document. It must be +/// read, not hung on. +#[test] +fn a_cycle_in_the_page_tree_terminates() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R 2 0 R] /Count 2 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ]); + assert_eq!(pdf.pages().len(), 1); +} + +/// So is a reference that points at itself. +#[test] +fn a_cycle_in_the_references_resolves_to_null() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R /Rotate 4 0 R >>"), + (4, b"5 0 R"), + (5, b"4 0 R"), + ]); + assert_eq!(pdf.get(&pdf.pages()[0].dict, "Rotate"), None); +} + +/// §7.3.9: a reference to an object the file does not have is null, and null +/// is indistinguishable from absent. +#[test] +fn a_dangling_reference_is_absent() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Rotate 99 0 R /CropBox null >>"), + ]); + let page = &pdf.pages()[0].dict; + assert_eq!(pdf.get(page, "Rotate"), None); + assert_eq!(pdf.get(page, "CropBox"), None); +} + +/// A trailer with no `/Root`, which a file assembled by hand often has. The +/// object that calls itself the catalog is taken instead. +#[test] +fn a_missing_root_is_found_by_type() { + let data = build( + &[ + (7, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + // /Root points at object 1, which does not exist. + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + assert!(pdf.catalog().contains_key("Pages")); + assert_eq!(pdf.pages().len(), 1); +} + +/// A document with a catalog nobody can find is unreadable, and says so. +#[test] +fn no_catalog_at_all_is_a_refusal() { + let data = build(&[(1, b"<< /Type /Page >>")], ""); + let error = Pdf::from_bytes(&data, "test").unwrap_err().to_string(); + assert!(error.contains("no document catalog"), "{error}"); +} + +/// An object at an offset that is not there is recovered from the body if it +/// is there at all — and only reported when it genuinely is not. +#[test] +fn an_object_the_table_misplaces_is_found_anyway() { + let mut data = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + // Break the entry for object 3 without touching the object itself. + let table = table_at(&data); + let entry = table + b"xref\n0 4\n".len() + 20 * 3; + data[entry..entry + 10].copy_from_slice(b"0000009999"); + + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + assert_eq!(pdf.pages().len(), 1); + assert_eq!(pdf.reconstructed(), [Reconstruction::CrossReferenceTable]); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); +} + +/// And when it really is not there, it is named — with the offset the table +/// claimed, so a reviewer can go and look. +#[test] +fn an_object_that_is_not_in_the_file_is_named() { + let mut data = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + (4, b"<< /Type /Nothing >>"), + ], + "", + ); + // Delete object 4's body, leaving the table naming it. + let at = rfind(&data, b"4 0 obj").expect("object 4"); + let end = table_at(&data); + data.splice(at..end, std::iter::repeat_n(b' ', end - at)); + + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 1, "{unread:?}"); + assert!(unread[0].starts_with("4 0 R —"), "{unread:?}"); + assert!(unread[0].contains("is not there"), "{unread:?}"); +} + +/// A dictionary key repeated in the file resolves to one entry, and the +/// document reads through it the same way. +#[test] +fn resolution_follows_a_chain_of_references() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Rotate 4 0 R >>"), + (4, b"5 0 R"), + (5, b"180"), + ]); + assert_eq!( + pdf.get(&pdf.pages()[0].dict, "Rotate") + .and_then(Object::as_i64), + Some(180) + ); +} + +#[test] +fn objects_are_matched_on_the_number_alone() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page >>"), + ]); + // A generation the table disagrees with is something writers produce and + // every reader ignores; refusing to follow it would lose the object. + assert!(pdf.object(ObjectId::new(3, 7)).is_some()); +} + +#[test] +fn a_document_reports_what_it_reconstructed_once() { + let pdf = Pdf::open( + &std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs/damaged.pdf"), + ) + .expect("open"); + assert_eq!(pdf.reconstructed(), [Reconstruction::CrossReferenceTable]); + assert_eq!( + Reconstruction::CrossReferenceTable.to_string(), + "the cross-reference table" + ); +} + +/// The text layer above this one has sources it cannot read too — a font with +/// no `ToUnicode` map is one — and they have to reach the same list. +#[test] +fn the_unread_list_is_open_to_the_layer_above() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page >>"), + ]); + pdf.note_unread( + Some(ObjectId::new(9, 0)), + "no /ToUnicode and no standard encoding", + ); + pdf.note_unread(None, "page 1 draws its Arabic as an image"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 2); + assert!( + unread + .iter() + .any(|u| u == "9 0 R — no /ToUnicode and no standard encoding") + ); + assert!( + unread + .iter() + .any(|u| u == "page 1 draws its Arabic as an image") + ); +} + +#[test] +fn a_name_answers_for_itself_and_nothing_else() { + let name = Name::from("FlateDecode"); + assert!(name.is("FlateDecode")); + assert!(!name.is("Flate")); + assert_eq!(name.as_str(), Some("FlateDecode")); +} + +/// mirsam is run on files people were sent. Every prefix and every +/// single-byte corruption of the fixture corpus has to come back as a document +/// or as an error, and never as a panic — the bounds in this crate are what +/// make that true, and this is what asserts they are all still there. +#[test] +fn no_prefix_or_corruption_of_the_corpus_panics() { + let dir = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs"); + let mut checked = 0usize; + for entry in std::fs::read_dir(&dir).expect("fixtures") { + let path = entry.expect("entry").path(); + if path.extension().is_none_or(|e| e != "pdf") { + continue; + } + let data = std::fs::read(&path).expect("read"); + checked += 1; + + for cut in (0..data.len()).step_by(7) { + let _ = Pdf::from_bytes(&data[..cut], "truncated"); + } + for at in (0..data.len()).step_by(13) { + let mut corrupted = data.clone(); + corrupted[at] = corrupted[at].wrapping_add(0x5b); + let _ = Pdf::from_bytes(&corrupted, "corrupted"); + } + } + assert!(checked >= 8, "the corpus lost fixtures: {checked}"); +} diff --git a/crates/mirsam-pdf/tests/encrypt.rs b/crates/mirsam-pdf/tests/encrypt.rs new file mode 100644 index 0000000..0845e2e --- /dev/null +++ b/crates/mirsam-pdf/tests/encrypt.rs @@ -0,0 +1,97 @@ +//! The refusal, and the sentence it gives. +//! +//! mirsam does not decrypt. What is being tested is that it says *what* it did +//! not decrypt: an encrypted document produces no text, and a reader that +//! stopped there would report a file with no defects. + +use mirsam_pdf::Pdf; +use mirsam_pdf::encrypt::describe; +use mirsam_pdf::lexer::Lexer; + +fn dict(source: &str) -> mirsam_pdf::Dictionary { + Lexer::new(source.as_bytes()) + .object() + .expect("parse") + .as_dict() + .expect("dictionary") + .clone() +} + +#[test] +fn the_algorithms_a_standard_handler_states() { + assert_eq!( + describe(&dict("<< /Filter /Standard /V 1 /R 2 >>")), + "RC4 40-bit (/Standard security handler, V1 R2)" + ); + assert_eq!( + describe(&dict("<< /Filter /Standard /V 2 /R 3 /Length 128 >>")), + "RC4 128-bit (/Standard security handler, V2 R3)" + ); + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 4 /R 4 /CF << /StdCF << /CFM /AESV2 >> >> /StmF /StdCF >>" + )), + "AES-128 (/Standard security handler, V4 R4)" + ); + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 5 /R 6 /CF << /StdCF << /CFM /AESV3 >> >> /StmF /StdCF >>" + )), + "AES-256 (/Standard security handler, V5 R6)" + ); +} + +/// `/StmF` names the filter streams actually use. A file whose stream filter +/// is not the one conventionally called `StdCF` would otherwise be described +/// by a dictionary entry nothing in it uses. +#[test] +fn the_filter_streams_use_is_the_one_reported() { + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 4 /R 4 /StmF /Custom \ + /CF << /StdCF << /CFM /AESV2 >> /Custom << /CFM /V2 /Length 16 >> >> >>" + )), + "RC4 128-bit (/Standard security handler, V4 R4)" + ); +} + +/// A third-party handler is named as itself. Claiming an algorithm it never +/// stated would be exactly the inference standing rule 4 forbids. +#[test] +fn a_handler_this_tool_does_not_know_is_named_rather_than_guessed_at() { + let answer = describe(&dict("<< /Filter /FOPN_foweb /V 4 /R 4 >>")); + assert!(answer.contains("/FOPN_foweb security handler"), "{answer}"); +} + +#[test] +fn an_encryption_that_states_nothing_says_so() { + let answer = describe(&dict("<< >>")); + assert_eq!( + answer, + "an undeclared algorithm (/Standard security handler)" + ); +} + +/// The `/Encrypt` dictionary is refused whether the trailer holds it inline or +/// points at it — §7.6.1 keeps it unencrypted precisely so a reader can find +/// out what it is up against. +#[test] +fn an_inline_encrypt_dictionary_is_refused_too() { + let mut out = b"%PDF-1.4\n".to_vec(); + let object = out.len(); + out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let table = out.len(); + out.extend_from_slice(b"xref\n0 2\n0000000000 65535 f \n"); + out.extend_from_slice(format!("{object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!( + "trailer\n<< /Size 2 /Root 1 0 R /Encrypt << /Filter /Standard /V 2 /R 3 \ + /Length 128 >> >>\nstartxref\n{table}\n%%EOF\n" + ) + .as_bytes(), + ); + + let error = Pdf::from_bytes(&out, "test").unwrap_err().to_string(); + assert!(error.contains("RC4 128-bit"), "{error}"); + assert!(error.contains("mirsam does not decrypt"), "{error}"); +} diff --git a/crates/mirsam-pdf/tests/filter.rs b/crates/mirsam-pdf/tests/filter.rs new file mode 100644 index 0000000..11eafda --- /dev/null +++ b/crates/mirsam-pdf/tests/filter.rs @@ -0,0 +1,189 @@ +//! Stream filters: §7.4, and the line between a filter that was not applied +//! and one this tool will never apply. + +use mirsam_pdf::filter::{Decoded, decode}; +use mirsam_pdf::object::{Dictionary, Name, Object}; + +fn bytes(data: &[u8], filters: &[&str], parms: &[Dictionary]) -> Vec { + let filters: Vec = filters.iter().copied().map(Name::from).collect(); + match decode(data, &filters, parms).expect("decode") { + Decoded::Bytes(bytes) => bytes, + Decoded::Stopped { filter } => panic!("stopped at {filter}"), + } +} + +fn stopped(data: &[u8], filters: &[&str]) -> Name { + let filters: Vec = filters.iter().copied().map(Name::from).collect(); + match decode(data, &filters, &[]).expect("decode") { + Decoded::Bytes(_) => panic!("expected to stop"), + Decoded::Stopped { filter } => filter, + } +} + +fn parms(entries: &[(&str, i64)]) -> Dictionary { + entries + .iter() + .map(|(key, value)| (Name::from(*key), Object::Integer(*value))) + .collect() +} + +fn deflate(data: &[u8]) -> Vec { + use flate2::Compression; + use flate2::write::ZlibEncoder; + use std::io::Write; + let mut encoder = ZlibEncoder::new(Vec::new(), Compression::best()); + encoder.write_all(data).unwrap(); + encoder.finish().unwrap() +} + +#[test] +fn no_filter_is_the_bytes_themselves() { + assert_eq!(bytes(b"plain", &[], &[]), b"plain"); +} + +#[test] +fn flate() { + let source = b"the annual report, at some length, so it actually compresses"; + assert_eq!(bytes(&deflate(source), &["FlateDecode"], &[]), source); + // Abbreviated names are §7.4's own, for inline images. + assert_eq!(bytes(&deflate(source), &["Fl"], &[]), source); +} + +/// A truncated stream keeps what it had. Discarding it would turn a document +/// with a damaged tail into a document with no text, which is the failure this +/// project treats as worse than a miss. +#[test] +fn a_truncated_flate_stream_keeps_what_decoded() { + let source = b"the annual report, at some length, so it actually compresses"; + let mut compressed = deflate(source); + compressed.truncate(compressed.len() - 4); + let out = bytes(&compressed, &["FlateDecode"], &[]); + assert!(!out.is_empty(), "a truncated stream decoded to nothing"); + assert!(source.starts_with(&out[..])); +} + +#[test] +fn ascii_hex() { + assert_eq!(bytes(b"48656C6C6F>", &["ASCIIHexDecode"], &[]), b"Hello"); + // White space between digits is ignored, and a missing final digit is a + // zero. + assert_eq!(bytes(b"48 65\n6C 6C 6F 9>", &["AHx"], &[]), b"Hello\x90"); +} + +#[test] +fn ascii85() { + assert_eq!( + bytes(b"87cURD]i,\"Ebo80~>", &["ASCII85Decode"], &[]), + b"Hello World!" + ); + // `z` is four zero bytes, and only at a group boundary. + assert_eq!(bytes(b"z~>", &["A85"], &[]), [0, 0, 0, 0]); + // The `<~` some writers open with. + assert_eq!(bytes(b"<~87cURDZ~>", &["ASCII85Decode"], &[]), b"Hello"); +} + +#[test] +fn run_length() { + // A length byte under 128 introduces that many literal bytes plus one; + // over 128 repeats the next byte 257 - length times; 128 ends the data. + let encoded = [4u8, b'H', b'e', b'l', b'l', b'o', 253, b'!', 128, b'?']; + assert_eq!(bytes(&encoded, &["RunLengthDecode"], &[]), b"Hello!!!!"); +} + +#[test] +fn lzw() { + // §7.4.4.2's own example: the codes for -----A---B, packed nine bits wide. + let encoded = [0x80u8, 0x0B, 0x60, 0x50, 0x22, 0x0C, 0x0C, 0x85, 0x01]; + assert_eq!(bytes(&encoded, &["LZWDecode"], &[]), b"-----A---B"); +} + +/// The PNG predictors, which a cross-reference stream always carries and +/// without which the table decodes to noise that looks like data. +#[test] +fn png_predictors() { + // Three rows of four bytes, each prefixed with its filter type: none, sub + // (each byte is a delta from its left neighbour) and up (from the row + // above). + let encoded = [ + 0u8, 1, 2, 3, 4, // 1 2 3 4 + 1, 1, 1, 1, 1, // 1 2 3 4 again, as deltas + 2, 1, 1, 1, 1, // 2 3 4 5 + ]; + let out = mirsam_pdf::filter::decode( + &deflate(&encoded), + &[Name::from("FlateDecode")], + &[parms(&[("Predictor", 12), ("Columns", 4)])], + ) + .expect("decode"); + let Decoded::Bytes(out) = out else { + panic!("stopped") + }; + assert_eq!(out, [1, 2, 3, 4, 1, 2, 3, 4, 2, 3, 4, 5]); +} + +#[test] +fn the_tiff_predictor() { + let encoded = deflate(&[1u8, 1, 1, 1, 5, 1, 1, 1]); + let out = mirsam_pdf::filter::decode( + &encoded, + &[Name::from("FlateDecode")], + &[parms(&[ + ("Predictor", 2), + ("Columns", 4), + ("BitsPerComponent", 8), + ])], + ) + .expect("decode"); + let Decoded::Bytes(out) = out else { + panic!("stopped") + }; + assert_eq!(out, [1, 2, 3, 4, 5, 6, 7, 8]); +} + +#[test] +fn a_chain_unwinds_in_order() { + let source = b"the annual report"; + let mut ascii85 = Vec::new(); + for chunk in deflate(source).chunks(4) { + let mut value = 0u32; + for i in 0..4 { + value = value * 256 + u32::from(chunk.get(i).copied().unwrap_or(0)); + } + let mut digits = [0u8; 5]; + for slot in digits.iter_mut().rev() { + *slot = 33 + (value % 85) as u8; + value /= 85; + } + ascii85.extend_from_slice(&digits[..chunk.len() + 1]); + } + ascii85.extend_from_slice(b"~>"); + assert_eq!( + bytes(&ascii85, &["ASCII85Decode", "FlateDecode"], &[]), + source + ); +} + +/// An image codec is named rather than decoded. The bytes behind one are not +/// text this crate failed to read; they are text that is not in the file, and +/// ADR 0009 says the difference has to reach the report. +#[test] +fn an_image_codec_stops_the_chain_by_name() { + for codec in ["DCTDecode", "JPXDecode", "CCITTFaxDecode", "JBIG2Decode"] { + assert!(stopped(b"\xff\xd8\xff", &[codec]).is(codec)); + } +} + +/// So does a filter this crate has simply never met, rather than an error that +/// would lose the rest of the document. +#[test] +fn an_unknown_filter_stops_the_chain_too() { + assert!(stopped(b"x", &["SomeVendorDecode"]).is("SomeVendorDecode")); +} + +/// `/Crypt` reaching this module means an encrypted document was not refused +/// at open, which is a bug in this crate rather than a fact about the file — +/// so it stops, and the name says which. +#[test] +fn crypt_stops_the_chain() { + assert!(stopped(b"x", &["Crypt"]).is("Crypt")); +} diff --git a/crates/mirsam-pdf/tests/lexer.rs b/crates/mirsam-pdf/tests/lexer.rs new file mode 100644 index 0000000..4dce385 --- /dev/null +++ b/crates/mirsam-pdf/tests/lexer.rs @@ -0,0 +1,202 @@ +//! Object syntax: §7.2 and §7.3, and the malformations every reader tolerates. + +use mirsam_pdf::lexer::Lexer; +use mirsam_pdf::object::{Name, Object, ObjectId}; + +fn parse(source: &str) -> Object { + Lexer::new(source.as_bytes()) + .object() + .unwrap_or_else(|e| panic!("{source:?}: {e}")) +} + +fn parse_bytes(source: &[u8]) -> Object { + Lexer::new(source).object().expect("parse") +} + +#[test] +fn the_atoms() { + assert_eq!(parse("null"), Object::Null); + assert_eq!(parse("true"), Object::Boolean(true)); + assert_eq!(parse("false"), Object::Boolean(false)); + assert_eq!(parse("42"), Object::Integer(42)); + assert_eq!(parse("-17"), Object::Integer(-17)); + assert_eq!(parse("+3"), Object::Integer(3)); + assert_eq!(parse("2.75"), Object::Real(2.75)); + assert_eq!(parse("-.002"), Object::Real(-0.002)); + // §7.3.3 spells this one out: a trailing point is legal. + assert_eq!(parse("4."), Object::Real(4.0)); +} + +/// Numbers a writer should not have produced and every reader accepts. A file +/// refused over its punctuation is a file this tool reported unreadable that +/// a viewer opens. +#[test] +fn numbers_no_writer_should_have_written() { + assert_eq!(parse("--5"), Object::Integer(0)); + assert_eq!(parse("6.-2"), Object::Real(6.0)); + assert_eq!(parse("."), Object::Integer(0)); +} + +#[test] +fn names_carry_their_escapes() { + assert_eq!(parse("/Type"), Object::Name(Name::from("Type"))); + // §7.3.5's own example: /A#42 is /AB, and a dictionary lookup depends on + // the two being one name. + assert_eq!(parse("/A#42"), Object::Name(Name::from("AB"))); + assert_eq!( + parse("/Lime#20Green"), + Object::Name(Name::from("Lime Green")) + ); + assert_eq!(parse("/"), Object::Name(Name::from(""))); + // A `#` a writer forgot to escape stays a `#` rather than failing the file. + assert_eq!(parse("/a#zz"), Object::Name(Name::from("a#zz"))); +} + +#[test] +fn a_name_prints_the_way_the_file_writes_it() { + assert_eq!(Name::from("Type").to_string(), "/Type"); + assert_eq!(Name::from("Lime Green").to_string(), "/Lime#20Green"); + assert_eq!(Name::from("A#B").to_string(), "/A#23B"); +} + +#[test] +fn literal_strings() { + assert_eq!(parse("(hello)").as_string(), Some(&b"hello"[..])); + // Balanced inner parentheses need no escaping, per §7.3.4.2. + assert_eq!(parse("(a (b) c)").as_string(), Some(&b"a (b) c"[..])); + assert_eq!(parse(r"(a\)b)").as_string(), Some(&b"a)b"[..])); + assert_eq!(parse(r"(a\nb)").as_string(), Some(&b"a\nb"[..])); + assert_eq!(parse(r"(\101)").as_string(), Some(&b"A"[..])); + // Octal beyond a byte drops the high bits, which the specification says + // in as many words. + assert_eq!(parse(r"(\400)").as_string(), Some(&b"\x00"[..])); + // A backslash before an end-of-line joins the lines. + assert_eq!(parse("(a\\\nb)").as_string(), Some(&b"ab"[..])); + // "\q" is q. + assert_eq!(parse(r"(\q)").as_string(), Some(&b"q"[..])); +} + +#[test] +fn hex_strings_pad_an_odd_final_digit() { + assert_eq!(parse("<48656C6C6F>").as_string(), Some(&b"Hello"[..])); + assert_eq!(parse("<48 65 6c>").as_string(), Some(&b"Hel"[..])); + // §7.3.4.3: a missing final digit is a zero. + assert_eq!(parse("<9>").as_string(), Some(&[0x90][..])); + assert_eq!(parse("<>").as_string(), Some(&[][..])); +} + +#[test] +fn arrays_and_dictionaries() { + let array = parse("[1 (two) /three [4]]"); + let items = array.as_array().expect("array"); + assert_eq!(items.len(), 4); + assert_eq!(items[0], Object::Integer(1)); + + let dict = parse("<< /Type /Page /Count 3 /Kids [1 0 R] >>"); + let dict = dict.as_dict().expect("dictionary"); + assert!(dict.get("Type").unwrap().as_name().unwrap().is("Page")); + assert_eq!(dict.get("Count").unwrap().as_i64(), Some(3)); + assert_eq!( + dict.get("Kids").unwrap().as_array().unwrap()[0], + Object::Reference(ObjectId::new(1, 0)) + ); +} + +/// A repeated key keeps the last value: the only way one arises in practice is +/// a writer appending an entry it meant to supersede an earlier one. +#[test] +fn a_repeated_key_keeps_the_last_value() { + let dict = parse("<< /N 1 /N 2 >>"); + let dict = dict.as_dict().expect("dictionary"); + assert_eq!(dict.len(), 1); + assert_eq!(dict.get("N").unwrap().as_i64(), Some(2)); +} + +#[test] +fn a_reference_is_two_integers_and_an_r() { + assert_eq!(parse("12 0 R"), Object::Reference(ObjectId::new(12, 0))); + // Without the R they are two numbers, and the lookahead has to rewind far + // enough to give both of them back. + assert_eq!(parse("[12 0]").as_array().unwrap().len(), 2); + assert_eq!(parse("[12 0 (R)]").as_array().unwrap().len(), 3); + // And a reference in the middle of an array is one item, not three. + let array = parse("[1 2 R 3]"); + let items = array.as_array().unwrap(); + assert_eq!(items.len(), 2); + assert_eq!(items[0], Object::Reference(ObjectId::new(1, 2))); + assert_eq!(items[1], Object::Integer(3)); +} + +#[test] +fn comments_are_white_space() { + assert_eq!(parse("% a comment\n42"), Object::Integer(42)); + let dict = parse("<< /A 1 % why\n/B 2 >>"); + assert_eq!(dict.as_dict().unwrap().len(), 2); +} + +#[test] +fn a_stream_takes_the_length_it_states() { + let source = b"<< /Length 5 >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// `/Length` may be an indirect reference, which the lexer cannot resolve: the +/// cross-reference table is itself parsed out of streams. The data runs to +/// `endstream` instead, and no caller has to hand over a resolver it has not +/// built yet. +#[test] +fn an_indirect_length_falls_back_to_endstream() { + let source = b"<< /Length 9 0 R >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// And a direct length that is simply wrong, which real writers produce. +#[test] +fn a_wrong_length_is_not_believed() { + let source = b"<< /Length 2 >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// A stream may end in a newline of its own, and only the one belonging to the +/// syntax is dropped. +#[test] +fn only_the_syntactic_newline_is_taken_off() { + let source = b"<< /Length 6 >>\nstream\nHELLO\n\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO\n"); +} + +#[test] +fn an_indirect_object() { + let mut lexer = Lexer::new(b"7 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let (id, object) = lexer.indirect().expect("indirect object"); + assert_eq!(id, ObjectId::new(7, 0)); + assert!(object.is_type("Catalog")); +} + +/// `endobj` is optional in practice: a writer that omits it produces a file +/// every reader opens, and the object is complete without it. +#[test] +fn endobj_is_not_required() { + let mut lexer = Lexer::new(b"7 0 obj\n<< /Type /Catalog >>\n8 0 obj\nnull\nendobj\n"); + assert_eq!(lexer.indirect().expect("first").0, ObjectId::new(7, 0)); + assert_eq!(lexer.indirect().expect("second").0, ObjectId::new(8, 0)); +} + +#[test] +fn a_parse_failure_says_where() { + let error = Lexer::new(b"<< /A 1").object().unwrap_err().to_string(); + assert!(error.contains("at byte 0"), "{error}"); + assert!(error.contains("no closing `>>`"), "{error}"); +} + +/// Depth is bounded: a hand-written file of ten thousand open brackets is not +/// a document, and must not be a stack overflow either. +#[test] +fn nesting_is_bounded() { + let source = "[".repeat(10_000); + assert!(Lexer::new(source.as_bytes()).object().is_err()); +} diff --git a/crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf b/crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf new file mode 100644 index 0000000000000000000000000000000000000000..8bc193184c6ecb2e916671921357aef5fca2b295 GIT binary patch literal 1947 zcmah~%Wm676h$`$N_5-(IEdV)g(D8Pp5lr(naNMt}!h0{fm zf9nSnMYmnGKhkrD6g83xlm=w-%)RH_*UU6dyYUWxD!s>ywgBgvJ z{p^8V`B+ksVl5fmIE3V}Nl5a!0Y#|~zO=w66eKCImi>+B7i#Bbmd*1a!lgW7*p@Gx zDQnkmL(tJAj)i7Co)ji$SIxbC=kTzlx@MS-P=|Pu6sG)u!m&y3ig%22O*FG2H_6mH zc=uy^p;G-+Z|}T0c-;@fQS0b>YkPD&x)zs%vx_i0p8Cnj?ts6MkH5a_z1%yyIhvpM zzkl?;GmWzC!=3Bo{fqCmzRj9k9&GPl4xhYwasNT=tLb!f);J0B-NF6!)}!5{w=cKy zSF-)9?L#GBb%at}BvZD~7NFUm4+`5+0?~>6WM;~h=VtP=DaOMDJ03z`vte3O4w1Zz)^U(MAtg?NVJL|B`RWB!LCIhLSlxR#QPFQ`Dyp!*xI492T${C zG#}!m`er)LXUUY^jvtD7Ze~xMgN=^G^-X&|Ps@F}wrjb&X}@f+e4et2J?6YmbO@}T zGIZ3gGatYHX^k^-&Vf%~bU**?kH5b9Y?JIEH@BY8xbOM(!?aRDv0GMS2+*#wbctyk z-SjvZmz#Fjj2m{?`5PD{*D=s-V#>!ow5rQhfQK)SD* chv{v`sGs&SofS!5*eYvoVk?Qi&4vF<=PrPiWAl4H9(~WI;=`qrjE~iYn4xvWwoJy+~(B z)o~mHvW7Enes4yc+^+H~{z0(GpTEC;GY&qqn`btgfmi(K6nIN%iSzyD##9 z8*)?YULLOU`o#=dL8Z3NYWD}-@ywk+720>QR9ZGW#b&;r!7R`1GM(ADa4wr|`6M-Q zi%>rfk__d*%0iGFI{lnv;q(E?k(EPEa_saLlbq;NlKsFZ%+%^#{JF#1_zQ;*3A1$i znB7oRZWcAf1y~lIoJ0oS%}apzo8K zmtK;gU}>meXUA1sQk0sQ%T;kp^pqo?gMk3+g}-x>rs=d?TRyeJWu@>LR+Wg3DJFN~ zch#NeoZ6j~8p&rg%Qt#s&Xiuon%74Q>)-u&rlTGjlQDZudy^aQ0!MDMK<*a^N_hYbW;zgxBH1?ZeyHMz-fi63v| z5|LTl8ka1t9@;b^)gv#yU!>=x)1%qu_Gk7ivX%8e=N9#+CGfw~lD(yB2Taa6GVeWf z*6I+eY1r}fDRs6Q{$1bJ{MjzmAeJtzX1b^Jo7{7TD-QEFPj6?}3OZA~Uyj!&E|cTb z-G(bYrd!_aYHKjox6~^>z4dy%)`X}}OQ(b;UDweO^6rdD``e(iS#QF9;ps9vPgSkD znft}oNSNnmvhn{7Gyl(g?ft~;wdg0q&vHMr=kAyJbVtJ9X>r*6FEUI!X58`lx4`(? z!K@wfwq;Ad^DC_@`Ggr{7Dz!BQdt0uMgOF%;F4SgeLn?LP&^kEmnayS0aowzpYTq+^eYcf2JAO@fly+}kw)?$R+)2^(T}>s< zT9r##UZw@={hX@p8g+54aDJxY`?j6ta^~r;o9!OojeGRQcZYB7KE|W_GI=mV))FIR zBZ5-X6!e2Lt5OvV4HWdl6`~ClOcab_74(Dh^Gg&Aff|Ao^gZ*^@)gV>Y<$_w!~!J^ zCnqp8{r^9+L4}=}xh1J3z^BKdOJ3~+&!gOyj}Z+v$J8Cp95|43`~U;PvS%!+n9eOO XNh~UJcjE#8pRuBa literal 0 HcmV?d00001 diff --git a/crates/mirsam-pdf/tests/pdfs/damaged.pdf b/crates/mirsam-pdf/tests/pdfs/damaged.pdf new file mode 100644 index 0000000000000000000000000000000000000000..bc61eed7136dd45d38ba905a1837ad7adcb252b3 GIT binary patch literal 1292 zcmZWp$!^;)5WVv&=3*c@q}r&(fFZz}(4b8lBsYdG^}do$wXc2!&jA2^%*`TOfP3&4kV^UP*5@X8;Z1aB#f*ta|IZpBXa5F+3X z@|?-0CLo>PIyjv7N!2|(Lz+Ii@(tLT+Og@IK3CXPzL-2+$iX#epqK z<`n@18I)U=hL9#clr_7UH_agSpk>!;a|z`$^WiE)RLy2Q;31DV=P^%s$}>LYbH3n9 zzT)egpzD$mZ)1v)K`QWV!?t|ye%Kg00h$9cw-CK_mXC7(T^dyhoKgF&QB_+@NX9<@ z4~EX?9r7SNY0+C=udW+qkXzkW_tL;tHMJg@gpJ&(hM`BTDl?v}t~!X0dCMEoeGv!T zkQ=3Yakwhfiy5?pT5g@y?hm@-IdJ||Xy1uSYSHW@oB4hYv!bxebY=^E|SKrw>VvtsDg;Cr)oM>4`og*$;ifOs(FZgPZZ-us{OUK?2*Uh#CBGKMny%E~j-r^|B*krQ)TCjgmH(Eph literal 0 HcmV?d00001 diff --git a/crates/mirsam-pdf/tests/pdfs/encrypted.pdf b/crates/mirsam-pdf/tests/pdfs/encrypted.pdf new file mode 100644 index 0000000000000000000000000000000000000000..c05c6cb5111bb7f3c078377416657ba3f33331b4 GIT binary patch literal 1726 zcmcgt$!^;)5WVXw<`SSixYR-|1`Gq2dgG;mnY{n2*zr_aSyJe*FCQ-SNPMa`oW!dcc;?l>{~tMr_Lsu({aC z8hj*NLBH=vl@pQ1uN4{|`&8sL+y+FyLo4N88SsEjWnPG3`2@EvPKDhN67Juf7$l9i z2FbmaXrl3t_LRY44V)YRyOMRe*BJ&#k*7G@01rY-==agZi5F69y#V_x)yCXIk`SRb zT8f=BTskcEz)N7cc48mm$c1I@yc;OB7hBM>Dz$kJ%Lm7Wj*qO|>2Z(yJm8#%JmN7= zc$W|OkdOG7Px?fiEFH>Q>!Qsd1^9ZwHhi|&uZ*n#)d4d%6t!$-KFaN*G({$`M&-3C zvND$twd;H-42{oM$UXmLiEMN^zEp+5-0Cu0NCWFa<+{xztmLLp4r=5@X4;XJWjj$h zY%~?s7qP<*nHIVhyG~j>nZ1^f%XKqr`v=u=>^1rnXkCd+YN0mL>A7wndTDB#>1@v9 zjmm1B-3v{*MXDS5aUZd7G3SJXMm`|S8@W$7v^ekxM~&Qa;uCp7*!5lFbgjIxKWO;Y z{;=T(#2Gd6i14_<31O@*#>UY;*0hZnkxkSgPuX5m2it7g>``V4B@YR&%8K;upI)!q zBtF)YrN!1r4QyeA%7xB>-C&=yD{O+7X7)qGqtxz#g~?kqN~gdElf_MdpS`)CJkp+G z<1(97QXP~7Y7_dQhh0iPaW-j#(&#~X-79Gx^ z;|1KMJ&pZByZuSyZgU}Vj@l4~VGzO^4l)mq9C|<WWx>u&Cp$|r6FMD@o>osD zeFY!D$MEby58`n}Ut_K6p2Y0#K*HD6U;Xq~Rh7d|ch`Fmvda1IZ{L{*E)>I4R6OF~Hb2J8H6G`{DFHY=IWHC}!2+rBF5X(;L1+k#2Bz4^ zLT;-pPh>vUClJSUKi5i%DQgXw)AhiMVK8Df5DN+eh`OH;c=)4MBK9GC<#tWVHerG?=G%Bl#q%Sp$vOHC# zjxdztENAGEW{ECO#+FRQj5+NCF?%YeI3EX@nv3afH(Tn3l90-gy)|z^cRYL6pX!{? zL?V^Q$CA}uw*mET*NoDYJC9qJ;V3x~iljxV8~N1+V&7oS2?y5RB+RYdCmb3ac!VQs zH=OE*Jtpk>E^%tcZsRvC-^6cOen6bIwMT?I7RQ9KxmX*=_}J32Vnjx$wNA-G(fk@U zje4X^mdmx^vtma6=0|^*r=+^kPCF$pdvXL$KYJ%32ImDFdAJL>NlvdQXw%GEbhC+; z3KwD`wCqSTDNZ>KL2Y9imcNo$H$Gfk+A|7>sH z`10xI`Srq<3LES~#N71 jGlF}+Qe#sB(y6bfP{WeUehZ~d(ViQ}tWw!~)@Afgc5M7H)v|K>q2h1f$SUdipNL7&PU`R{V&{P7c>S1GVQ&+J!U2jm^U&n!a zrE)27LGVM&jO&)9IkG<3JAU)t+qcsl4U*^Hi-323|GE0cJ#ax!PWk30h=Yr{0uf6q zXL<@^B&Vu^9s>5E-{-2#2uSCg|9-` z2zp0Lgw(N4Nb*J@#jQT_l@_-Z1S9}rUsZZxQZz`;KkZFURcf2>Vc|j8fckxuSyH0b zRtdy=RZi^*_#Oc&Ym_Yb87F__6FoNXwBUjW2bzE26hVt@!yxdg_d6$IyqMS~&tZW% zg7}zcD2IW(Q~C6yi6+)FZN_t%;_rd_oTuuokryDga$Aj*iFGm8C1tPyBFSe~8N|g0 z1C{CwTiO$NfxC2s`lkNHo@^(>ggFjl4nC~@$zs;|&sBe6k2}QYI8Ryql|3Wo)%B?U z!8)Y+nq6N({e3;JbEA&A3lVqk7ery`hj0w{H4jsz1?9h5gm?F#<^}7xUWoJj0O$KS oVRtoaWIj`-8TWC1sUX5p(weTL9rMacV;i55|B`pRZ}t-Y7nr2WmH+?% literal 0 HcmV?d00001 diff --git a/crates/mirsam-pdf/tests/pdfs/incremental.pdf b/crates/mirsam-pdf/tests/pdfs/incremental.pdf new file mode 100644 index 0000000000000000000000000000000000000000..7b0228bdf5c0d1e52abd6957d507ce39e159ad88 GIT binary patch literal 1576 zcmb_c&2F1O5WeqI%q3ECNMT{XShA4(hZ?nMqr^Q$Ijmvr5ZPeCx=QJ!f`Pn#ZQyX;hq7qlK4ka)(*5(nCgaNH- zwAzZvLU@@3nZRNt1_2CW9~Ol;A6M00ZJ^P+x-l29coIHb281fa2qgv>B4UIw4luzX zj&XuhoZ&oW=zJlVw=P7-ASd|xz&3sFZof3P0$c}TZlYT4cHZmFS8d8%QH|;^Rhid? zhPXTDp)kBYJt4}VcSLV`HM^=xL)D09=TtS;ZGo$#jD zYWGQPsY9;Ird8X^tb8{6Mnj=jZrAn?nbTT2drq|7sa!XzT5B=#{S-!7W}E3`Pogtf zuJT9KFt-Ht<6w}I99S8V<0_2>PEJ^+>QZh<{;8&2#w1yyjy&c2hCA4@Y1w0D%1R#z-_<*|w=aKQx5?n8 zp5CG~8%@m3vy3UB&`AF={R2M^C8uKlbhMZKFeH6UZ^HYjt$MMiTEO|%R*=eXE7)H2 zF3NL5aO2^~4UMvn Fn?J=opuGS9 literal 0 HcmV?d00001 diff --git a/crates/mirsam-pdf/tests/xref.rs b/crates/mirsam-pdf/tests/xref.rs new file mode 100644 index 0000000..13916ac --- /dev/null +++ b/crates/mirsam-pdf/tests/xref.rs @@ -0,0 +1,131 @@ +//! The cross-reference chain: newest wins, and a malformed chain terminates. + +use mirsam_pdf::xref::{Slot, read, recover}; + +/// A document with two sections, the newer one redefining object 3. +fn incremental(previous_marks_free: bool) -> Vec { + let mut out = b"%PDF-1.4\n".to_vec(); + let first_object = out.len(); + out.extend_from_slice(b"3 0 obj\n(old)\nendobj\n"); + let first_table = out.len(); + out.extend_from_slice(b"xref\n0 4\n0000000000 65535 f \n0000000000 65535 f \n"); + out.extend_from_slice(b"0000000000 65535 f \n"); + out.extend_from_slice(format!("{first_object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!("trailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{first_table}\n%%EOF\n").as_bytes(), + ); + + let second_object = out.len(); + out.extend_from_slice(b"3 0 obj\n(new)\nendobj\n"); + let second_table = out.len(); + let entry = if previous_marks_free { + "0000000000 65535 f \n".to_string() + } else { + format!("{second_object:010} 00000 n \n") + }; + out.extend_from_slice(format!("xref\n3 1\n{entry}").as_bytes()); + out.extend_from_slice( + format!( + "trailer\n<< /Size 4 /Root 1 0 R /Prev {first_table} >>\nstartxref\n{second_table}\n%%EOF\n" + ) + .as_bytes(), + ); + out +} + +#[test] +fn the_newer_section_wins() { + let data = incremental(false); + let xref = read(&data, 0).expect("read"); + let Some(Slot::InFile { offset, .. }) = xref.slots.get(&3) else { + panic!("object 3 is not in the table") + }; + assert_eq!( + &data[*offset as usize..*offset as usize + 13], + b"3 0 obj\n(new)" + ); +} + +/// A free entry in a newer section is what *hides* an object an older one +/// still names. Reading it as "no entry here, look further back" would +/// resurrect a deleted object. +#[test] +fn a_free_entry_in_a_newer_section_hides_the_older_one() { + let data = incremental(true); + let xref = read(&data, 0).expect("read"); + assert_eq!(xref.slots.get(&3), Some(&Slot::Free)); +} + +/// A `/Prev` pointing at the section that carried it. Two writers producing +/// the same offset is a loop, not a longer history. +#[test] +fn a_prev_loop_terminates() { + let mut out = b"%PDF-1.4\n".to_vec(); + let object = out.len(); + out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let table = out.len(); + out.extend_from_slice(b"xref\n0 2\n0000000000 65535 f \n"); + out.extend_from_slice(format!("{object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!("trailer\n<< /Size 2 /Root 1 0 R /Prev {table} >>\nstartxref\n{table}\n%%EOF\n") + .as_bytes(), + ); + let xref = read(&out, 0).expect("read"); + assert_eq!(xref.slots.len(), 2); +} + +/// A file with no `startxref` at all is not readable through the chain, and +/// says so rather than answering with an empty table. +#[test] +fn no_startxref_is_an_error() { + let error = read(b"%PDF-1.4\n1 0 obj\nnull\nendobj\n", 0) + .unwrap_err() + .to_string(); + assert!(error.contains("no `startxref`"), "{error}"); +} + +/// The scan takes the *last* definition of an object number, which is the same +/// rule the chain walk enforces from the other end: an update appends. +#[test] +fn recovery_takes_the_last_definition() { + let data = incremental(false); + let xref = recover(&data); + let Some(Slot::InFile { offset, .. }) = xref.slots.get(&3) else { + panic!("object 3 was not found") + }; + assert_eq!( + &data[*offset as usize..*offset as usize + 13], + b"3 0 obj\n(new)" + ); + // And the newest trailer, so a /Root moved by an update is the one used. + assert!(xref.trailer.contains_key("Root")); +} + +/// `objstm` is not `obj`. A scan that matched inside a longer token would +/// invent objects out of the middle of dictionaries. +#[test] +fn recovery_does_not_match_inside_a_longer_token() { + let data = b"%PDF-1.4\n1 0 objstm\n2 0 obj\nnull\nendobj\n"; + let xref = recover(data); + assert_eq!(xref.slots.keys().copied().collect::>(), [2]); +} + +/// The cross-reference streams in the fixture corpus, read through the same +/// entry point: a compressed object comes back as one, and the type-2 entry +/// names the stream that holds it. +#[test] +fn a_cross_reference_stream_names_object_streams() { + let path = + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs/compressed.pdf"); + let data = std::fs::read(path).expect("fixture"); + let xref = read(&data, 0).expect("read"); + assert_eq!( + xref.slots.get(&1), + Some(&Slot::InStream { + stream: 8, + index: 0 + }) + ); + assert!(matches!(xref.slots.get(&4), Some(Slot::InFile { .. }))); + assert!(xref.trailer.contains_key("Root")); +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 0c241f8..542daa6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -240,6 +240,43 @@ crates/ origin a browser gives it, and dir="auto" is Unset because "the renderer picks" is what auto asks for. DocumentReader only + mirsam-pdf/ adapter — the object layer of a format that is not a + package, and the first this project reads without being + able to write it. No DocumentWriter here or planned: a + broken Arabic PDF is rebuilt from its source. See + docs/adr/0010 for why the parser is written rather than + taken, and what is taken instead + object.rs PDF's eight types and two composites. A name is bytes, + because it is the one place a file stores a key and a + lossy decode would merge two entries the document kept + apart + lexer.rs bytes to objects, and the malformations every reader + tolerates: a wrong /Length, an unescaped `#`, `--5`. A + stream's length is verified against `endstream` rather + than believed, which covers indirect, absent and simply + wrong with one rule + filter.rs the general-purpose filters — Flate, LZW, the two ASCII + encodings, run-length, and the PNG and TIFF predictors a + cross-reference stream needs. The four image codecs are + *named* and never decoded: a stream behind DCTDecode is + not text this crate failed to read, it is text that is + not in the file + xref.rs both spellings of the cross-reference, the /Prev chain + newest-first, and the hybrid file where the stream + outranks the classic table beside it. Plus the scan that + rebuilds a table whose offsets have gone stale, because a + document reported unreadable while a viewer opens it is + the wrong half of standing rule 4 + encrypt.rs what the /Encrypt dictionary says, said precisely, so the + refusal names the algorithm. Nothing decrypts + document.rs the graph: objects loaded once, object streams expanded, + references resolved with a bound, and the list of what + could not be read — which is ADR 0009's port, reaching the + format that needs it most + page.rs the page tree and the four attributes §7.7.3.4 inherits + down it. /Resources is the one that matters: a text layer + reading it from the leaf alone finds no fonts on the very + common document that states them once at the root mirsam-conformance/ no library code at all: the crate exists so one suite can depend on every adapter at once. Living inside mirsam-ooxml, it would have made that crate depend on its diff --git a/docs/PLAN.md b/docs/PLAN.md index 4d56d4e..9cb230e 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -1621,7 +1621,7 @@ express — listed in `repairs.skipped`, never claimed. --- -## M6 — PDF `[ ]` +## M6 — PDF `[~]` PDF implements `DocumentReader` **only**. @@ -1637,27 +1637,104 @@ they should fire here without a line of `mirsam-core` changing. If either needs one, the abstraction was wrong and §3.5's instruction applies: fix the abstraction, not the rule. -### 6.1 The object layer `[ ]` +### 6.1 The object layer `[x]` `mirsam-pdf`: cross-reference table *and* cross-reference stream, object streams, `FlateDecode`, the page tree, the trailer. The package layer of a format that is not a package. -- [ ] **Decide the parser: own it or take one. This wants an ADR, not a +- [x] **Decide the parser: own it or take one. This wants an ADR, not a `Cargo.toml` line.** The constraints are pure Rust, no C, no network, a permissive licence, and no rasteriser — mirsam needs the object graph and the content streams, and most of what a PDF library carries is the rendering half this project has said it will never do. The precedent cuts both ways: `dom.rs` and `css.rs` were written here rather than taken, but they are hundreds of lines against a specification measured in thousands. -- [ ] Refuse an encrypted document with a message that names the encryption, +- [x] Refuse an encrypted document with a message that names the encryption, rather than extracting nothing and reporting a clean file. -- [ ] Fixture corpus: one PDF per generator family, since the defects this +- [x] Fixture corpus: one PDF per generator family, since the defects this milestone hunts are generator signatures rather than author mistakes. *A file the adapter could not open is not a file with no defects.* Whatever 6.1 cannot read has to reach the report as itself. +**The decision was to own the object layer and take the decompressor**, and +[ADR 0010](adr/0010-write-the-pdf-object-layer-take-the-decompressor.md) is the +record. What settled it was not the size of the specification but which half of +it mirsam asks for: the object graph and the content streams are §7.2 to §7.8, +and everything from §8 on is the rendering this project has said it will never +do. The second cost decided it — a library's object model becomes the adapter's +vocabulary, and with it every answer to *is a dangling reference null, is a +wrong `/Length` readable, is a stale `startxref` fatal*. Those are precisely the +honesty decisions §6.2 turns on, and inheriting them unexamined would have been +inheriting the wrong ones silently. `flate2` is taken for `FlateDecode`, held to +the `zlib-rs` backend `zip` already resolves, so no dependency entered the tree. + +**Every read of the file is checked against the file rather than believed.** A +`/Length` may be an indirect reference — the value is in an object the +cross-reference table has not been parsed yet to find — and plenty of writers +get the direct form wrong by a byte or two. So it is a hint, verified: taken +when `endstream` is where it says, and the data runs to the next `endstream` +otherwise. One rule covers indirect, absent and simply wrong, and no caller had +to be handed a resolver that cannot exist yet. + +**Newest wins, and it is the walk that enforces it.** An incremental update +appends a section and chains `/Prev` at the old one, so every insertion is +first-wins and an object an update redefined is found in the section that +redefined it. A free entry in a newer section therefore *hides* an object an +older one still names, which is what a deletion is. The trailers merge the same +way, which is why a `/Root` a signature moved is the one used. A hybrid file +(§7.5.8.4) carries a classic table *and* an `/XRefStm`, and the stream is read +first because the classic table beside it deliberately understates what the file +holds — a reader that took the classic table as the whole truth would draw a +blank page and report nothing wrong with it. + +**A stale `startxref` is recovered from, not refused.** Offsets go stale: a file +concatenated by a script, truncated by a transfer or edited by hand has one +pointing at nothing in particular, and every viewer opens it anyway by scanning +for `obj`. So does this. *A document reported unreadable while a reader opens it +is the wrong half of standing rule 4*, and `Reconstruction` is how the fact is +said out loud rather than hidden: the cross-reference table and the page tree +each report when they had to be rebuilt. + +**The refusals are the design.** An encrypted document is refused at +`Pdf::open` with the algorithm read out of the `/Encrypt` dictionary, which +§7.6.1 keeps in the clear precisely so a reader can say what it is up against — +`AES-256 (/Standard security handler, V5 R6)`, and a handler this tool does not +know is named rather than guessed at. Nothing decrypts, including the very +common permissions-only wrapper with an empty user password: a tool that opened +those would be one nobody could run on a file they were given. And the four +image codecs are *named*, never decoded, because a stream behind `DCTDecode` is +not text this crate failed to read — it is text that is not in the file. + +**`unread` is native here rather than bolted on.** ADR 0009 gave the port a way +to say what it could not read, and this is the format that needs it: an object +at an offset that is not there, an object stream behind a filter this crate does +not implement, a page drawn as a scan. Each is recorded where it happens and +named as the document names it — `5 0 R — /DCTDecode, a filter this tool does +not decode` — and `Pdf::note_unread` is open so that §6.2's own unread sources, +a font with no `ToUnicode` map among them, reach one list rather than two. + +*Acceptance:* met. Seventy tests over eight fixtures, one per generator +family, written byte by byte by `scripts/make-pdf-fixture.py` — because a +fixture built by the library under test proves only that the library agrees with +itself. A classic table with uncompressed content; a cross-reference stream with +its catalog, page tree, page and font inside an object stream; a hybrid file +whose `ToUnicode` map only the `/XRefStm` names; an incremental update that +supersedes the page's content; a file whose `startxref` points past its own end +and which reads identically to the one it was damaged from; an AES-256 document +refused by name; a distiller's `ASCII85Decode`/`FlateDecode` chain beside +`ASCIIHexDecode`, `RunLengthDecode` and `LZWDecode`; and a scanned page whose +image comes back named in `unread` rather than as a page with nothing wrong with +it. And every prefix and every single-byte corruption of all eight comes back +as a document or as an error and never as a panic, which is what the bounds on +nesting depth, reference-chain length, `/Prev` chain length and page-tree depth +are there for: this tool is run on files people were sent. + +No `DocumentReader` yet and no entry in the CLI's readable list: text +extraction is 6.2, and a `.pdf` the binary accepted before it could read one +would be a claim this milestone has not earned. + ### 6.2 Text extraction and its order `[~]` `BT`/`ET`, `Tj`/`TJ`/`'`/`"`, `Tf`/`Tm`/`Td`/`TD`/`T*`, and the `ToUnicode` diff --git a/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md b/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md new file mode 100644 index 0000000..969c3aa --- /dev/null +++ b/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md @@ -0,0 +1,130 @@ +# 10. Write the PDF object layer; take the decompressor + +Date: 2026-09-07 + +## Status + +Accepted. + +## Context + +`docs/PLAN.md` §6.1 asks the question directly: *"Decide the parser: own it or +take one. This wants an ADR, not a `Cargo.toml` line."* The constraints it sets +are pure Rust, no C, no network, a permissive licence, and no rasteriser. + +The precedent in this repository cuts both ways and says so. `dom.rs` and +`css.rs` were written here rather than taken, and the reasoning was specific: +`markup5ever_rcdom` describes itself as unsupported, and the part of CSS that +decides direction is a few hundred lines. But `html5ever` *was* taken, because +tree construction is a specification with observable, tested behaviour that a +hand-rolled nesting stack gets wrong in ways that move a node's ancestors — and +direction is inherited along ancestors. + +So the question is not "does this project write parsers". It is which half of +this particular specification mirsam needs, and what a dependency would carry +in with it. + +### What mirsam needs from a PDF + +The object graph and the content streams. Concretely: the two spellings of the +cross-reference table, `/Prev` chains, hybrid-reference files, object streams, +the general-purpose stream filters, the page tree with its four inheritable +attributes, and the bytes of the content streams. Above that, §6.2 needs the +text operators and `ToUnicode` CMaps, §6.4 the embedded font programs — and +`mirsam-fonts::sfnt` already parses those. + +### What a PDF library carries + +Most of a PDF library is the half this project has said it will never do. +Graphics state, path construction, colour spaces, transparency groups, shading +patterns, image decoding, annotation appearance streams, form field rendering — +none of it is reachable from anything mirsam asks. The read-write libraries add +a serialiser, an incremental-update writer and an encryption implementation, all +for a format this project has committed to never writing. + +There is a second cost, and it is the one that decided this. A library's object +model becomes the vocabulary of the adapter that uses it, and every subsequent +question — is a missing `/Type` an error, does a dangling reference resolve to +null, is a stream with a wrong `/Length` readable — is answered by that +library's judgement rather than by this project's. Those judgements are exactly +where a document reader is honest or is not. §6.2's whole difficulty is deciding +what the adapter is *entitled to claim* about a file that does not state its own +direction, and inheriting those decisions from a dependency would mean +inheriting them unexamined. + +### And what only a library should supply + +Inflate. `FlateDecode` is a codec, not a document format: it has one correct +answer, that answer is `zlib`'s, and a hand-written implementation would be a +liability with no upside. The same is true of nothing else in this crate. + +## Decision + +**`mirsam-pdf` owns the object layer.** The lexer, the object model, the +cross-reference table and stream, object streams, the page tree, the ASCII and +run-length and LZW filters and the PNG and TIFF predictors are written here, +against ISO 32000-1, with the section numbers in the comments. + +**`flate2` is taken for `FlateDecode`**, held to the `zlib-rs` backend the +workspace already resolves through `zip`. That is the same rule `ttf-parser` +is pinned under: two inflate implementations in one binary would be two answers +to what a stream decompresses to, and no new dependency enters the tree. + +**Three things are refused rather than implemented**, and the refusals are the +design rather than a gap: + +- **Decryption.** An encrypted document is refused at `Pdf::open` with the + algorithm named from the `/Encrypt` dictionary, which §7.6.1 keeps in the + clear. A tool that opened protected documents by trying the empty password + would be one nobody could run on a file they were given. +- **Rasterising.** `DCTDecode`, `JPXDecode`, `CCITTFaxDecode` and `JBIG2Decode` + are named, never decoded. A stream behind one is not text this crate failed + to read; it is text that is not in the file. +- **Writing.** There is no `DocumentWriter` here and none planned. A broken + Arabic PDF is rebuilt from its source document. + +## Consequences + +**The honesty decisions are this project's, and they are visible.** A wrong +`/Length` is verified rather than believed; a dangling reference is null, +because §7.3.9 says so; a reference cycle terminates and answers null rather +than hanging; a stale `startxref` is recovered from by scanning, because a +document reported unreadable while every viewer opens it is the wrong half of +standing rule 4. Each of those is a line in this crate with a test beside it, +and each would otherwise have been a dependency's default. + +**The unread list is native rather than bolted on.** ADR 0009 gave +`DocumentReader` a way to say what it could not read, and PDF is the format +that needs it most — a scanned page, an object at an offset that is not there, +a filter this crate does not implement. Because the loader is ours, each of +those is recorded where it happens, named as the document names it. Through a +library those cases are an `Err` or an empty `Vec`, and the difference between +"no text here" and "no text this tool can see" would have had to be +reconstructed from the outside. + +**It is more code than a dependency line, and less than a PDF library.** Two +and a half thousand lines, comments included, against a specification of some +hundreds of pages — because the pages that matter here are §7.2 through §7.8, +and everything from §8 on is rendering. That ratio is the decision, and it +would not survive the day mirsam needed a graphics state. + +**The bound is stated so it can be tested.** Nesting depth, reference-chain +length, `/Prev` chain length and page-tree depth all have explicit limits, and +a file that exceeds one is malformed rather than a stack overflow. mirsam is +run on files people were sent, which is the threat model a document tool has +whether or not it says so. + +**The cost is real: this crate now owns every PDF bug it has.** A file that +opens in a viewer and not here is this repository's to fix, and the fixture +corpus is the answer to that — one file per generator family, written byte by +byte by `scripts/make-pdf-fixture.py` rather than by a library, so a fixture +can never prove only that the parser agrees with itself. + +## Related + +- Standing rule 4, `AGENTS.md` — "Report only what was verified." +- [ADR 0002](0002-rust-and-token-preserving-xml.md) — the same shape one format + earlier: parse what is needed, keep the bytes, do not adopt a model. +- [ADR 0009](0009-a-source-the-adapter-could-not-read-is-part-of-the-report.md) + — the port this crate's `unread` list feeds. +- `docs/PLAN.md` §6.1, which asked for this record. diff --git a/scripts/make-pdf-fixture.py b/scripts/make-pdf-fixture.py new file mode 100644 index 0000000..99478ce --- /dev/null +++ b/scripts/make-pdf-fixture.py @@ -0,0 +1,546 @@ +#!/usr/bin/env python3 +"""Generate the PDF fixture corpus the object layer is proved against. + +Writes into crates/mirsam-pdf/tests/pdfs/. One file per *generator family*, +because the things PLAN §6 hunts are generator signatures rather than author +mistakes: how a producer assembles a file decides what a reader has to be able +to do before it can see a single character. + + classic.pdf + A cross-reference *table*, uncompressed content, no object streams — + PDF 1.4 as anything predating 2003 writes it, and as a great many + report generators still do. The baseline: if this does not read, + nothing else will. + + compressed.pdf + A cross-reference *stream* and an object stream, PDF 1.7. This is what + LibreOffice, modern Word and Chrome's print-to-PDF emit, and every + object but the stream containers themselves is inside a FlateDecode + stream. A reader with no inflate sees an empty document. + + hybrid.pdf + A classic table *and* an /XRefStm beside it (ISO 32000-1 §7.5.8.4) — + Acrobat's compatibility output, where a 1.4 reader is shown a subset of + what a 1.5 reader is shown. The page's real content object is reachable + only through the stream, so a reader that took the classic table as the + whole truth draws a blank page and reports no defects. + + incremental.pdf + classic.pdf plus an appended update chained by /Prev, replacing the + page's content. Every form filler, signer and annotation tool writes + this shape. Newest-wins is the whole test: read the first definition + and the document says the wrong thing. + + damaged.pdf + Correct objects, a startxref pointing into the middle of nothing. What + a truncated download, a naive concatenation or a hand edit leaves + behind. Every viewer opens it by scanning for `obj`, so mirsam must + too: a document reported unreadable while a reader opens it is the + wrong half of standing rule 4. + + encrypted.pdf + /Encrypt with the AES-256 crypt filter, V5 R6. Must be refused *by + name*. The streams inside are not real ciphertext and this file is not + a cryptographic fixture — nothing here decrypts, and what is being + proved is that the refusal reads the /Encrypt dictionary, which + §7.6.1 keeps in the clear precisely so a reader can say what it is up + against. + + ascii-filters.pdf + ASCIIHexDecode, ASCII85Decode, RunLengthDecode and LZWDecode, one per + stream, plus an ASCII85+Flate chain. The distiller family: dvips and + pdfTeX emitted ASCII-safe streams for decades so that a PDF survived a + mail gateway, and the files are still in circulation. + + image-only.pdf + A page whose entire content is a DCTDecode image. The scanner family, + and the one this crate must be *unable* to read: the text is not in the + file. It has to come back named in `unread` rather than as a page with + nothing wrong with it. + +Every one of them is written byte by byte here, with no third-party module, +for the reason the OOXML generators avoid python-pptx: a fixture built by the +library under test proves the library agrees with itself. + +Regenerate with `make pdfs`. +""" + +from __future__ import annotations + +import zlib +from pathlib import Path + +OUT = Path(__file__).resolve().parent.parent / "crates" / "mirsam-pdf" / "tests" / "pdfs" + +# The Arabic every fixture carries, so §6.2 has something to grow into and so a +# reader that silently drops non-ASCII fails here rather than on a user's file. +ARABIC = "التقرير السنوي" + + +def utf16be(text: str) -> bytes: + """A PDF text string in UTF-16BE with the byte-order mark §7.9.2.2 wants.""" + return b"\xfe\xff" + text.encode("utf-16-be") + + +def hex_string(data: bytes) -> bytes: + return b"<" + data.hex().upper().encode("ascii") + b">" + + +def tounicode(codes: dict[int, str]) -> bytes: + """A ToUnicode CMap mapping single-byte codes to the characters they draw. + + This is how a generator says what its glyph codes mean, and the only thing + standing between an extractor and a page of numbers. + """ + pairs = b"".join( + b"<%02X> <%s>\n" % (code, char.encode("utf-16-be").hex().upper().encode("ascii")) + for code, char in sorted(codes.items()) + ) + return ( + b"/CIDInit /ProcSet findresource begin\n" + b"12 dict begin\nbegincmap\n" + b"/CMapName /Mirsam-Fixture def\n/CMapType 2 def\n" + b"1 begincodespacerange\n<00> \nendcodespacerange\n" + + b"%d beginbfchar\n" % len(codes) + + pairs + + b"endbfchar\n" + b"endcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n" + ) + + +def arabic_codes(text: str, first: int = 0x41) -> tuple[bytes, dict[int, str]]: + """One byte per character, plus the map back. The shape a generator that + subsets a font produces, and the shape §6.3 hunts a reversal in.""" + codes: dict[int, str] = {} + out = bytearray() + for index, char in enumerate(text): + code = first + index + codes[code] = char + out.append(code) + return bytes(out), codes + + +class Builder: + """Objects in, PDF out. Holds the offsets a cross-reference table needs.""" + + def __init__(self, version: str = "1.4") -> None: + self.version = version + self.objects: dict[int, bytes] = {} + self.next = 1 + + def add(self, body: bytes, number: int | None = None) -> int: + if number is None: + number = self.next + self.next = max(self.next, number + 1) + self.objects[number] = body + return number + + def stream(self, dict_entries: bytes, data: bytes, number: int | None = None) -> int: + body = b"<< " + dict_entries + b" /Length %d >>\nstream\n" % len(data) + data + b"\nendstream" + return self.add(body, number) + + def serialise(self, root: int, info: int | None = None, extra_trailer: bytes = b"") -> bytes: + out = bytearray(b"%%PDF-%s\n%%\xe2\xe3\xcf\xd3\n" % self.version.encode("ascii")) + offsets: dict[int, int] = {} + for number in sorted(self.objects): + offsets[number] = len(out) + out += b"%d 0 obj\n" % number + self.objects[number] + b"\nendobj\n" + + size = max(self.objects) + 1 + startxref = len(out) + out += b"xref\n0 %d\n" % size + out += b"0000000000 65535 f \n" + for number in range(1, size): + if number in offsets: + out += b"%010d 00000 n \n" % offsets[number] + else: + out += b"0000000000 65535 f \n" + trailer = b"<< /Size %d /Root %d 0 R" % (size, root) + if info is not None: + trailer += b" /Info %d 0 R" % info + trailer += extra_trailer + b" >>" + out += b"trailer\n" + trailer + b"\nstartxref\n%d\n%%%%EOF\n" % startxref + return bytes(out) + + +def page_document(builder: Builder, content: bytes, *, font_extra: bytes = b"") -> tuple[int, int]: + """The four objects every one-page fixture shares. Returns (catalog, content).""" + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + 3, + ) + stream = builder.stream(b"", content, 4) + builder.add( + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica" + font_extra + b" >>", + 5, + ) + return catalog, stream + + +def content_for(text: str, *, first: int = 0x41) -> tuple[bytes, dict[int, str]]: + codes, mapping = arabic_codes(text, first) + stream = ( + b"BT\n/F1 18 Tf\n72 760 Td\n(Annual report) Tj\n" + b"0 -28 Td\n" + hex_string(codes) + b" Tj\nET\n" + ) + return stream, mapping + + +def write(name: str, data: bytes) -> None: + OUT.mkdir(parents=True, exist_ok=True) + (OUT / name).write_bytes(data) + print(f" {name:22} {len(data):7} bytes") + + +# -------------------------------------------------------------------------- +# classic.pdf +# -------------------------------------------------------------------------- +def classic() -> bytes: + builder = Builder("1.4") + content, mapping = content_for(ARABIC) + catalog, _ = page_document(builder, content, font_extra=b" /ToUnicode 6 0 R") + builder.stream(b"", tounicode(mapping), 6) + info = builder.add( + b"<< /Title " + hex_string(utf16be(ARABIC)) + b" /Producer (mirsam fixtures) >>", 7 + ) + return builder.serialise(catalog, info) + + +# -------------------------------------------------------------------------- +# compressed.pdf — cross-reference stream and object stream +# -------------------------------------------------------------------------- +def compressed() -> bytes: + content, mapping = content_for(ARABIC) + content_z = zlib.compress(content, 9) + cmap_z = zlib.compress(tounicode(mapping), 9) + + # 1..3 and 7 go inside the object stream; 4, 5 and 6 are streams, which + # §7.5.7 forbids an object stream from holding. + inside = { + 1: b"<< /Type /Catalog /Pages 2 0 R >>", + 2: b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", + 3: ( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>" + ), + 5: b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>", + 7: b"<< /Title " + hex_string(utf16be(ARABIC)) + b" /Producer (mirsam fixtures) >>", + } + pairs = bytearray() + bodies = bytearray() + for number, body in sorted(inside.items()): + pairs += b"%d %d " % (number, len(bodies)) + bodies += body + b" " + objstm_plain = bytes(pairs) + bytes(bodies) + first = len(pairs) + objstm_z = zlib.compress(objstm_plain, 9) + + out = bytearray(b"%PDF-1.7\n%\xe2\xe3\xcf\xd3\n") + offsets: dict[int, int] = {} + + def emit(number: int, body: bytes) -> None: + offsets[number] = len(out) + out.extend(b"%d 0 obj\n" % number + body + b"\nendobj\n") + + emit( + 4, + b"<< /Filter /FlateDecode /Length %d >>\nstream\n" % len(content_z) + + content_z + + b"\nendstream", + ) + emit( + 6, + b"<< /Filter /FlateDecode /Length %d >>\nstream\n" % len(cmap_z) + cmap_z + b"\nendstream", + ) + emit( + 8, + b"<< /Type /ObjStm /N %d /First %d /Filter /FlateDecode /Length %d >>\nstream\n" + % (len(inside), first, len(objstm_z)) + + objstm_z + + b"\nendstream", + ) + + # 9 is the cross-reference stream itself: three fields, 1/4/2 bytes wide. + size = 10 + rows = bytearray() + startxref = len(out) + for number in range(size): + if number == 0: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + elif number in offsets: + rows += bytes([1]) + offsets[number].to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number == 9: + rows += bytes([1]) + startxref.to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number in inside: + index = sorted(inside).index(number) + rows += bytes([2]) + (8).to_bytes(4, "big") + index.to_bytes(2, "big") + else: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + rows_z = zlib.compress(bytes(rows), 9) + out.extend( + b"9 0 obj\n<< /Type /XRef /Size %d /W [1 4 2] /Root 1 0 R /Info 7 0 R" + b" /Filter /FlateDecode /Length %d >>\nstream\n" % (size, len(rows_z)) + + rows_z + + b"\nendstream\nendobj\n" + ) + out.extend(b"startxref\n%d\n%%%%EOF\n" % startxref) + return bytes(out) + + +# -------------------------------------------------------------------------- +# hybrid.pdf — a classic table with /XRefStm beside it +# -------------------------------------------------------------------------- +def hybrid() -> bytes: + content, mapping = content_for(ARABIC) + cmap = tounicode(mapping) + + out = bytearray(b"%PDF-1.5\n%\xe2\xe3\xcf\xd3\n") + offsets: dict[int, int] = {} + + def emit(number: int, body: bytes) -> None: + offsets[number] = len(out) + out.extend(b"%d 0 obj\n" % number + body + b"\nendobj\n") + + emit(1, b"<< /Type /Catalog /Pages 2 0 R >>") + emit(2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>") + emit( + 3, + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + ) + emit(4, b"<< /Length %d >>\nstream\n" % len(content) + content + b"\nendstream") + emit(5, b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>") + # 6 is the object the classic table deliberately does not name: only the + # /XRefStm knows where it is, which is the whole point of a hybrid file. + emit(6, b"<< /Length %d >>\nstream\n" % len(cmap) + cmap + b"\nendstream") + + size = 8 + rows = bytearray() + xrefstm_at = len(out) + for number in range(size): + if number in offsets: + rows += bytes([1]) + offsets[number].to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number == 7: + rows += bytes([1]) + xrefstm_at.to_bytes(4, "big") + (0).to_bytes(2, "big") + else: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + rows_z = zlib.compress(bytes(rows), 9) + out.extend( + b"7 0 obj\n<< /Type /XRef /Size %d /W [1 4 2] /Root 1 0 R" + b" /Filter /FlateDecode /Length %d >>\nstream\n" % (size, len(rows_z)) + + rows_z + + b"\nendstream\nendobj\n" + ) + + startxref = len(out) + out.extend(b"xref\n0 6\n0000000000 65535 f \n") + for number in range(1, 6): + out.extend(b"%010d 00000 n \n" % offsets[number]) + out.extend( + b"trailer\n<< /Size %d /Root 1 0 R /XRefStm %d >>\nstartxref\n%d\n%%%%EOF\n" + % (size, xrefstm_at, startxref) + ) + return bytes(out) + + +# -------------------------------------------------------------------------- +# incremental.pdf — classic.pdf, then an update that supersedes the content +# -------------------------------------------------------------------------- +def incremental() -> bytes: + base = classic() + first_startxref = int(base.rsplit(b"startxref\n", 1)[1].split(b"\n", 1)[0]) + + revised, _ = content_for("التقرير السنوي المعدل") + out = bytearray(base) + offset = len(out) + out.extend( + b"4 0 obj\n<< /Length %d >>\nstream\n" % len(revised) + revised + b"\nendstream\nendobj\n" + ) + startxref = len(out) + out.extend( + b"xref\n0 1\n0000000000 65535 f \n4 1\n%010d 00000 n \n" % offset + + b"trailer\n<< /Size 8 /Root 1 0 R /Info 7 0 R /Prev %d >>\nstartxref\n%d\n%%%%EOF\n" + % (first_startxref, startxref) + ) + return bytes(out) + + +# -------------------------------------------------------------------------- +# damaged.pdf — good objects, a startxref pointing nowhere useful +# -------------------------------------------------------------------------- +def damaged() -> bytes: + base = classic() + head, _ = base.rsplit(b"startxref\n", 1) + # Past the end of the file, which is what a truncated transfer leaves and + # what a reader must survive by scanning the body instead. + return head + b"startxref\n%d\n%%%%EOF\n" % (len(base) + 4096) + + +# -------------------------------------------------------------------------- +# encrypted.pdf — refused by name, never decrypted +# -------------------------------------------------------------------------- +def encrypted() -> bytes: + builder = Builder("1.7") + content, mapping = content_for(ARABIC) + catalog, _ = page_document(builder, content, font_extra=b" /ToUnicode 6 0 R") + builder.stream(b"", tounicode(mapping), 6) + encrypt = builder.add( + b"<< /Filter /Standard /V 5 /R 6 /Length 256" + b" /CF << /StdCF << /CFM /AESV3 /AuthEvent /DocOpen /Length 32 >> >>" + b" /StmF /StdCF /StrF /StdCF" + b" /O <" + b"00" * 48 + b"> /U <" + b"00" * 48 + b">" + b" /OE <" + b"00" * 32 + b"> /UE <" + b"00" * 32 + b">" + b" /Perms <" + b"00" * 16 + b"> /P -1340 >>", + 7, + ) + return builder.serialise(catalog, extra_trailer=b" /Encrypt %d 0 R" % encrypt) + + +# -------------------------------------------------------------------------- +# ascii-filters.pdf — the distiller family +# -------------------------------------------------------------------------- +def ascii_hex(data: bytes) -> bytes: + return data.hex().upper().encode("ascii") + b">" + + +def ascii85(data: bytes) -> bytes: + out = bytearray() + for start in range(0, len(data), 4): + group = data[start : start + 4] + pad = 4 - len(group) + value = int.from_bytes(group + b"\0" * pad, "big") + if value == 0 and pad == 0: + out += b"z" + continue + digits = bytearray() + for _ in range(5): + digits.insert(0, 33 + value % 85) + value //= 85 + out += bytes(digits)[: 5 - pad] + return bytes(out) + b"~>" + + +def run_length(data: bytes) -> bytes: + """Literal runs only. A decoder that mishandles the length byte produces + plausible rubbish, which is what this is here to catch.""" + out = bytearray() + for start in range(0, len(data), 128): + chunk = data[start : start + 128] + out.append(len(chunk) - 1) + out += chunk + return bytes(out) + b"\x80" + + +def lzw(data: bytes) -> bytes: + """Variable-code LZW with early change, as §7.4.4.2 defines it.""" + table = {bytes([b]): b for b in range(256)} + nxt = 258 + width = 9 + bits = 0 + held = 0 + out = bytearray() + + def push(code: int) -> None: + nonlocal bits, held + bits = (bits << width) | code + held += width + while held >= 8: + out.append((bits >> (held - 8)) & 0xFF) + held -= 8 + + push(256) + current = b"" + for byte in data: + candidate = current + bytes([byte]) + if candidate in table: + current = candidate + continue + push(table[current]) + table[candidate] = nxt + nxt += 1 + # Early change: the width grows one code before the table is full. + if nxt + 1 > (1 << width) and width < 12: + width += 1 + current = bytes([byte]) + if current: + push(table[current]) + push(257) + if held: + out.append((bits << (8 - held)) & 0xFF) + return bytes(out) + + +def ascii_filters() -> bytes: + builder = Builder("1.4") + content, mapping = content_for(ARABIC) + cmap = tounicode(mapping) + + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents [4 0 R 8 0 R 9 0 R]" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + 3, + ) + # The page's own content, through ASCII85 and Flate in that order — the + # chain a distiller writes, and the one a reader must unwind in reverse. + chained = ascii85(zlib.compress(content, 9)) + builder.stream(b"/Filter [/ASCII85Decode /FlateDecode]", chained, 4) + builder.add(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>", 5) + builder.stream(b"/Filter /ASCIIHexDecode", ascii_hex(cmap), 6) + builder.add(b"<< /Producer (mirsam fixtures) >>", 7) + builder.stream(b"/Filter /RunLengthDecode", run_length(b"% run length\n"), 8) + builder.stream(b"/Filter /LZWDecode", lzw(b"% lzw\n"), 9) + return builder.serialise(catalog, 7) + + +# -------------------------------------------------------------------------- +# image-only.pdf — the scanner family, which has no text to find +# -------------------------------------------------------------------------- +# The smallest thing a JPEG decoder would accept, and this one never runs: what +# matters is that the filter is named and the bytes are left alone. +JPEG = bytes.fromhex( + "ffd8ffe000104a46494600010100000100010000ffdb004300" + + "08" * 64 + + "ffc0000b080001000101011100ffc40014000100000000000000000000000000000000" + "03ffda0008010100003f00d2cf20ffd9" +) + + +def image_only() -> bytes: + builder = Builder("1.4") + content = b"q\n595 0 0 842 0 0 cm\n/Im1 Do\nQ\n" + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /XObject << /Im1 5 0 R >> >> >>", + 3, + ) + builder.stream(b"", content, 4) + builder.stream( + b"/Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray" + b" /BitsPerComponent 8 /Filter /DCTDecode", + JPEG, + 5, + ) + return builder.serialise(catalog) + + +def main() -> None: + print(f"writing PDF fixtures into {OUT}") + write("classic.pdf", classic()) + write("compressed.pdf", compressed()) + write("hybrid.pdf", hybrid()) + write("incremental.pdf", incremental()) + write("damaged.pdf", damaged()) + write("encrypted.pdf", encrypted()) + write("ascii-filters.pdf", ascii_filters()) + write("image-only.pdf", image_only()) + + +if __name__ == "__main__": + main()