diff --git a/AGENTS.md b/AGENTS.md index cc18992..5626349 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -460,6 +460,19 @@ all of them against the published ECMA-376 schemas. A document the application offers to repair cannot answer "does it open the repaired file without a prompt", which is the M1 application check. +**The PDF fixtures are not corpus documents, and are generated too.** The eight +under `crates/mirsam-pdf/tests/pdfs/` come from `scripts/make-pdf-fixture.py` +and regenerate with `make pdfs` — one file per *generator family*, because what +PDF work hunts is a generator signature rather than an author mistake: a classic +cross-reference table, a cross-reference stream with object streams, a hybrid +file, an incremental update, a stale `startxref`, an AES-256 document, a +distiller's ASCII filter chain, and a page that is a scan. They live beside the +crate rather than in `tests/fixtures/` on purpose: the golden corpus is what the +*binary* does to a document, and `mirsam audit` does not read a `.pdf` yet. When +a `DocumentReader` lands, two of them join the corpus and the exit code `repair` +gives a readable format with no writer is recorded on a real PDF, the way +`quarterly-page.html` records it now. + **The two HTML corpus documents are the exception, and edited by hand.** A page *is* its own source: there is no package to build, so `make fixtures` has nothing to generate and `scripts/validate-ooxml.py` has nothing to validate. diff --git a/CHANGELOG.md b/CHANGELOG.md index f73537e..760e08e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,56 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Milestones M1 through M5 have landed since `v0.1.0`: byte-preserving `repair`, inheritance resolution, the DOCX and XLSX adapters with their writers, the HTML -reader, and shaping and font-coverage checks behind `--fonts`. Next is M6, the -read-only PDF adapter. See [`docs/PLAN.md`](docs/PLAN.md) and +reader, and shaping and font-coverage checks behind `--fonts`. M6, the +read-only PDF adapter, is under way: its object layer has landed and text +extraction is next. See [`docs/PLAN.md`](docs/PLAN.md) and [`docs/ROADMAP.md`](docs/ROADMAP.md). ### Added +- **`mirsam-pdf`, the PDF object layer.** The package layer of a format that is + not a package: both spellings of the cross-reference table, `/Prev` chains, + hybrid-reference files, object streams, the general-purpose stream filters, + the page tree with the four attributes it inherits, and content streams + decoded and joined. Nothing is wired into the CLI yet — `mirsam audit` + still turns a `.pdf` away — because text extraction is PLAN §6.2, and a + binary that accepted a `.pdf` before it could read one would be making a + claim this milestone has not earned. + + **The parser is written here and the decompressor is taken**, which PLAN + §6.1 asked for as an ADR rather than a `Cargo.toml` line. What settled it was + which half of ISO 32000-1 mirsam asks for: the object graph and the content + streams are §7.2 to §7.8, and everything from §8 on is the rendering this + project has said it will never do. The deciding cost was the second one — a + library's object model becomes the adapter's vocabulary, and with it every + answer to *is a dangling reference null, is a wrong `/Length` readable, is a + stale `startxref` fatal*, which are exactly the honesty decisions §6.2 turns + on. `flate2` is taken for `FlateDecode`, pinned to the `zlib-rs` backend + `zip` already resolves, so no dependency entered the tree. See + [ADR 0010](docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md). + + **An encrypted document is refused by name.** `AES-256 (/Standard security + handler, V5 R6)`, read out of the `/Encrypt` dictionary that §7.6.1 keeps in + the clear — including the permissions-only wrapper with an empty user + password, because a tool that opened those is one nobody could run on a file + they were given. The alternative was extracting nothing and reporting a clean + file, which is the failure the whole milestone is arranged against. + + **What could not be read is named, and so is what had to be rebuilt.** The + four image codecs are named rather than decoded — a stream behind `DCTDecode` + is not text this crate failed to read, it is text that is not in the file — + and a scanned page therefore reaches `unread` as `5 0 R — /DCTDecode, a + filter this tool does not decode`. That is ADR 0009 reaching the format that + needs it most. Separately, a file whose `startxref` has gone stale is + recovered from by scanning for `obj`, the way every viewer opens it, and says + so: a document reported unreadable while a reader opens it is the wrong half + of standing rule 4. + + Eight fixtures, one per generator family, are written byte by byte by + `scripts/make-pdf-fixture.py` and regenerate with `make pdfs` — a fixture + built by the library under test proves only that the library agrees with + itself. + - **The DOCX writer.** `mirsam repair report.docx fixed.docx` works, and `repair` stops turning a Word document away as a readable format without a writer. A repair writes `w:pPr/w:bidi`, `w:pPr/w:jc`, `w:rPr/w:lang@w:bidi`, diff --git a/Cargo.lock b/Cargo.lock index 7b2a07b..287ab84 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -315,6 +315,14 @@ dependencies = [ "zip", ] +[[package]] +name = "mirsam-pdf" +version = "0.1.0" +dependencies = [ + "flate2", + "mirsam-core", +] + [[package]] name = "new_debug_unreachable" version = "1.0.6" diff --git a/Cargo.toml b/Cargo.toml index f24ccf0..479cad0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,6 +4,7 @@ members = [ "crates/mirsam-core", "crates/mirsam-ooxml", "crates/mirsam-html", + "crates/mirsam-pdf", "crates/mirsam-conformance", "crates/mirsam-fonts", "crates/mirsam-cli", @@ -24,6 +25,7 @@ categories = ["command-line-utilities", "text-processing", "internationalization mirsam-core = { path = "crates/mirsam-core", version = "0.1.0" } mirsam-ooxml = { path = "crates/mirsam-ooxml", version = "0.1.0" } mirsam-html = { path = "crates/mirsam-html", version = "0.1.0" } +mirsam-pdf = { path = "crates/mirsam-pdf", version = "0.1.0" } mirsam-fonts = { path = "crates/mirsam-fonts", version = "0.1.0" } unicode-bidi = "0.3.18" unicode-normalization = "0.1.25" @@ -36,6 +38,10 @@ quick-xml = "0.42" # than taking `markup5ever_rcdom`, which names itself unsupported. html5ever = "0.39" zip = { version = "8.6", default-features = false, features = ["deflate"] } +# FlateDecode. Held to the backend `zip` already pulls in: two inflate +# implementations in one binary would be two answers to "what do these bytes +# decompress to". +flate2 = { version = "1.1", default-features = false, features = ["zlib-rs"] } serde = { version = "1", features = ["derive"] } serde_json = "1" thiserror = "2" diff --git a/Makefile b/Makefile index ba0d880..b584de5 100644 --- a/Makefile +++ b/Makefile @@ -76,8 +76,12 @@ fixtures: ## Regenerate the hand-built corpus documents, then their reports @python3 scripts/make-torture-fixture.py @python3 scripts/make-word-fixture.py @python3 scripts/make-excel-fixture.py + @$(MAKE) --no-print-directory pdfs @$(MAKE) --no-print-directory golden +pdfs: ## Regenerate the PDF fixture corpus (one file per generator family) + @python3 scripts/make-pdf-fixture.py + fonts: ## Regenerate the hand-built fonts the shaping tests run against @python3 scripts/make-shaping-fixture.py @@ -113,5 +117,5 @@ install: ## Install mirsam into ~/.cargo/bin @cargo install --path crates/mirsam-cli .PHONY: help build test fmt fmt-check lint doc clean version check-version \ - codename release-name msrv audit-deps golden fixtures fonts names \ + codename release-name msrv audit-deps golden fixtures pdfs fonts names \ validate-fixtures corpus verify pre-push hooks-install install diff --git a/crates/mirsam-pdf/Cargo.toml b/crates/mirsam-pdf/Cargo.toml new file mode 100644 index 0000000..977d1d3 --- /dev/null +++ b/crates/mirsam-pdf/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "mirsam-pdf" +description = "PDF object layer for mirsam: cross-reference tables and streams, object streams, stream filters and the page tree. Reads; never writes." +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true + +[dependencies] +mirsam-core.workspace = true +# The one thing not written here. See docs/adr/0010. Pinned to the backend the +# workspace already resolves through `zip`, so the binary carries one inflate +# implementation rather than two. +flate2 = { workspace = true } diff --git a/crates/mirsam-pdf/src/document.rs b/crates/mirsam-pdf/src/document.rs new file mode 100644 index 0000000..89f2c24 --- /dev/null +++ b/crates/mirsam-pdf/src/document.rs @@ -0,0 +1,606 @@ +//! The document: the object graph, resolved, and the page tree over it. +//! +//! This is the package layer of a format that is not a package. `Package` in +//! `mirsam-ooxml` answers "give me the part named `ppt/slides/slide1.xml`"; +//! [`Pdf`] answers "give me object `12 0 R`, wherever the file put it" — in the +//! body, inside a compressed object stream, or in a position only a scan +//! could find. +//! +//! ## Everything is loaded at open, and the file bytes are then dropped +//! +//! An object stream has to be decompressed before the objects inside it exist +//! at all, and a page's `/Resources` may be inherited from a node three levels +//! up the tree. Lazy loading would mean either interior mutability threaded +//! through every accessor or a resolver argument on every call, and it would +//! buy nothing: this tool reads the whole document. +//! +//! Streams are kept *filtered*, so what is held is about the size of the file, +//! and the file's own bytes are released once the graph is built. +//! +//! ## What could not be read is answered for +//! +//! A page whose content is a JPEG, an object at an offset that is not there, an +//! object stream behind a filter this crate does not implement — each is +//! recorded and named by [`Pdf::unread`]. ADR 0009 is the reason: a source the +//! adapter could not read is part of the report, and a document that produced +//! no text must never be indistinguishable from one with no defects. + +use mirsam_core::error::{Error, Result}; +use std::cell::RefCell; +use std::collections::{BTreeMap, BTreeSet}; +use std::fmt; +use std::path::Path; + +use crate::filter::{self, Decoded}; +use crate::lexer::{Lexer, find}; +use crate::object::{Dictionary, Name, Object, ObjectId, Stream}; +use crate::xref::{self, Slot, Xref}; +use crate::{encrypt, page}; + +/// How far into the file `%PDF-` is looked for. §7.5.2 puts it at byte zero; +/// a kilobyte of tolerance covers the shell scripts and mail gateways that put +/// something in front of it, and stops well short of finding the string inside +/// an embedded document. +const HEADER_WINDOW: usize = 1024; + +/// How many references are followed before a chain is called circular. +/// `12 0 R` pointing at `13 0 R` pointing back is a file, not a document. +const MAX_INDIRECTION: usize = 32; + +/// Something the file did not state usably and the loader had to work out. +/// +/// Not a defect and not a finding: a fact about how the document was read, +/// which a caller may want to say out loud and which the tests assert on. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Reconstruction { + /// The cross-reference chain was unusable, so the body was scanned for + /// ` obj`. + CrossReferenceTable, + /// `/Root /Pages` did not yield a walkable tree, so the pages were taken + /// from the objects that call themselves `/Type /Page`. + PageTree, +} + +impl fmt::Display for Reconstruction { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::CrossReferenceTable => f.write_str("the cross-reference table"), + Self::PageTree => f.write_str("the page tree"), + } + } +} + +/// Something in the document this crate did not read, and why. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct Unread { + /// The object it was, where there is one to name. + pub object: Option, + pub reason: String, +} + +/// Named as the document names it, so a reader can find the thing being talked +/// about: `12 0 R — /DCTDecode, an image codec this tool does not decode`. +impl fmt::Display for Unread { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.object { + Some(id) => write!(f, "{id} — {}", self.reason), + None => f.write_str(&self.reason), + } + } +} + +/// One entry of the object graph. +struct Entry { + generation: u16, + object: Object, +} + +/// A PDF, read. +pub struct Pdf { + name: String, + version: String, + objects: BTreeMap, + trailer: Dictionary, + catalog: ObjectId, + pages: Vec, + reconstructed: Vec, + /// Grown while reading, because a stream's filter is only discovered when + /// somebody asks for its bytes. A `RefCell` rather than `&mut self` on + /// every accessor: recording what was not read is bookkeeping, and making + /// it change the shape of the read API would put it in every caller. + unread: RefCell>, +} + +/// A summary rather than the graph: printing every object of a document is +/// not a thing a test failure or a `dbg!` wants, and the pieces are all +/// reachable through the accessors. +impl fmt::Debug for Pdf { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Pdf") + .field("name", &self.name) + .field("version", &self.version) + .field("objects", &self.objects.len()) + .field("pages", &self.pages.len()) + .field("reconstructed", &self.reconstructed) + .field("unread", &self.unread.borrow().len()) + .finish() + } +} + +impl Pdf { + /// Open and read a PDF. + pub fn open(path: &Path) -> Result { + if !path.exists() { + return Err(Error::NotFound); + } + let data = std::fs::read(path)?; + Self::from_bytes(&data, &path.display().to_string()) + } + + /// Read a PDF already in memory. `name` is used only in messages. + pub fn from_bytes(data: &[u8], name: &str) -> Result { + let (shift, version) = header(data)?; + + let mut reconstructed = Vec::new(); + let xref = match xref::read(data, shift) { + Ok(xref) if !xref.slots.is_empty() => xref, + _ => { + reconstructed.push(Reconstruction::CrossReferenceTable); + xref::recover(data) + } + }; + + refuse_encryption(data, &xref, shift)?; + + let mut unread = Vec::new(); + let mut objects = load(data, &xref, shift, &mut unread); + + // An offset that pointed at nothing is worth one scan of the body + // before it is reported: a file with a stale table is a file every + // viewer opens, and the objects are all still there. + let missing = xref + .slots + .iter() + .filter(|(number, slot)| { + matches!(slot, Slot::InFile { .. }) && !objects.contains_key(number) + }) + .count(); + if missing > 0 { + let rebuilt = xref::recover(data); + let mut recovered = load(data, &rebuilt, shift, &mut Vec::new()); + let mut gained = false; + for (number, entry) in std::mem::take(&mut recovered) { + if let std::collections::btree_map::Entry::Vacant(slot) = objects.entry(number) { + slot.insert(entry); + gained = true; + } + } + if gained { + reconstructed.push(Reconstruction::CrossReferenceTable); + unread.retain(|entry| { + entry + .object + .is_none_or(|id| !objects.contains_key(&id.number)) + }); + } + } + + if objects.is_empty() { + return Err(Error::Format(format!( + "{name}: no readable objects; the file states a cross-reference table and has \ + nothing where it points" + ))); + } + + let mut pdf = Self { + name: name.to_string(), + version, + objects, + trailer: xref.trailer.clone(), + catalog: ObjectId::default(), + pages: Vec::new(), + reconstructed, + unread: RefCell::new(unread), + }; + pdf.expand_object_streams(&xref); + pdf.catalog = pdf.find_catalog()?; + let (pages, rebuilt) = page::tree(&pdf); + pdf.pages = pages; + if rebuilt { + pdf.reconstructed.push(Reconstruction::PageTree); + } + pdf.reconstructed.sort_unstable(); + pdf.reconstructed.dedup(); + Ok(pdf) + } + + /// The path or label this document was opened under. + pub fn name(&self) -> &str { + &self.name + } + + /// The version in the `%PDF-` header: `1.7`, `2.0`. + /// + /// The header alone. A catalog may override it with `/Version`, and + /// [`Pdf::catalog`] is where a caller asks for that — reporting one number + /// for two different claims would hide a disagreement between them. + pub fn version(&self) -> &str { + &self.version + } + + pub fn trailer(&self) -> &Dictionary { + &self.trailer + } + + /// The document catalog, `/Root`. + pub fn catalog(&self) -> &Dictionary { + self.object(self.catalog) + .and_then(Object::as_dict) + .unwrap_or(&EMPTY) + } + + /// The pages, in the order the page tree gives them. + pub fn pages(&self) -> &[page::Page] { + &self.pages + } + + /// How many objects were loaded. + pub fn object_count(&self) -> usize { + self.objects.len() + } + + /// What the loader had to work out because the file did not state it + /// usably. Empty for a well-formed document. + pub fn reconstructed(&self) -> &[Reconstruction] { + &self.reconstructed + } + + /// What this crate did not read, in object order, each named as the + /// document names it. + /// + /// This is what a `DocumentReader` over PDF answers `unread_sources` with. + pub fn unread(&self) -> Vec { + let mut entries = self.unread.borrow().clone(); + entries.sort(); + entries.dedup(); + entries.iter().map(ToString::to_string).collect() + } + + /// Record something that was not read. Public because the text layer above + /// this one has its own reasons — a font with no `ToUnicode` map is a + /// source it could not read, and the list has to be one list. + pub fn note_unread(&self, object: Option, reason: impl Into) { + self.unread.borrow_mut().push(Unread { + object, + reason: reason.into(), + }); + } + + /// The object at `id`, or `Null` if the document has none. + /// + /// **Matched on the object number alone.** A reference carries a + /// generation, and matching on it would be more faithful to §7.3.10 — but + /// a generation that disagrees with the table is something writers produce + /// and every reader ignores, and refusing to follow such a reference would + /// lose an object the file plainly holds. + pub fn object(&self, id: ObjectId) -> Option<&Object> { + self.objects.get(&id.number).map(|entry| &entry.object) + } + + /// The generation the file stored `number` under. + pub fn generation(&self, number: u32) -> Option { + self.objects.get(&number).map(|entry| entry.generation) + } + + /// Every object, in object-number order — which is the order the file + /// wrote them, and the only order available to a document whose page tree + /// has to be reconstructed. + pub fn objects(&self) -> impl Iterator { + self.objects + .iter() + .map(|(number, entry)| (ObjectId::new(*number, entry.generation), &entry.object)) + } + + /// Follow references until something else is reached. + pub fn resolve<'a>(&'a self, object: &'a Object) -> &'a Object { + let mut current = object; + for _ in 0..MAX_INDIRECTION { + let Object::Reference(id) = current else { + return current; + }; + match self.object(*id) { + Some(next) => current = next, + None => return &Object::Null, + } + } + // A cycle. `Null` is the honest answer: §7.3.9 already says an + // undefined reference is null, and a document whose objects point at + // each other has no value there to give. + &Object::Null + } + + /// A dictionary entry, resolved. Absent and explicitly null are one + /// answer, as §7.3.7 says they are. + pub fn get<'a>(&'a self, dict: &'a Dictionary, key: &str) -> Option<&'a Object> { + match dict.get(key).map(|value| self.resolve(value)) { + Some(Object::Null) | None => None, + Some(value) => Some(value), + } + } + + /// The bytes of a stream, decoded. + /// + /// `Ok(None)` means the stream is behind a filter this crate does not + /// decode — an image codec, all but always — and the fact has been + /// recorded in [`Pdf::unread`]. It is not an error: a scanned page is a + /// page whose text is not in the file, which is a true thing to report and + /// not a failure to read one. + pub fn decoded(&self, id: ObjectId) -> Result>> { + let Some(stream) = self.object(id).and_then(Object::as_stream) else { + return Ok(None); + }; + self.decode_stream(Some(id), stream) + } + + fn decode_stream(&self, id: Option, stream: &Stream) -> Result>> { + let filters = self.filters(&stream.dict); + let parms = self.decode_parms(&stream.dict, filters.len()); + match filter::decode(&stream.data, &filters, &parms)? { + Decoded::Bytes(bytes) => Ok(Some(bytes)), + Decoded::Stopped { filter } => { + self.note_unread(id, format!("{filter}, a filter this tool does not decode")); + Ok(None) + } + } + } + + /// `/Filter`, which is a name or an array of them, either of which may be + /// written indirectly. + fn filters(&self, dict: &Dictionary) -> Vec { + let Some(filter) = self.get(dict, "Filter").or_else(|| self.get(dict, "F")) else { + return Vec::new(); + }; + match filter { + Object::Name(name) => vec![name.clone()], + Object::Array(items) => items + .iter() + .filter_map(|item| self.resolve(item).as_name()) + .cloned() + .collect(), + _ => Vec::new(), + } + } + + /// `/DecodeParms`, padded to the length of the filter chain so index `i` + /// belongs to filter `i` whatever shape the file wrote it in. + fn decode_parms(&self, dict: &Dictionary, filters: usize) -> Vec { + let mut parms = match self + .get(dict, "DecodeParms") + .or_else(|| self.get(dict, "DP")) + { + Some(Object::Dictionary(one)) => vec![one.clone()], + Some(Object::Array(items)) => items + .iter() + .map(|item| self.resolve(item).as_dict().cloned().unwrap_or_default()) + .collect(), + _ => Vec::new(), + }; + parms.resize(filters, Dictionary::new()); + parms + } + + /// Decompress every object stream the table pointed into, and add the + /// objects it holds. + /// + /// Objects already loaded from the body win: a compressed object is only + /// reachable through the table that named it, so a body definition of the + /// same number came from a section that superseded it. + fn expand_object_streams(&mut self, xref: &Xref) { + for (container, members) in xref.in_streams() { + let id = ObjectId::new(container, self.generation(container).unwrap_or(0)); + let Some(stream) = self.object(id).and_then(Object::as_stream) else { + self.note_unread(Some(id), "an object stream the file does not contain"); + continue; + }; + if !stream + .dict + .get("Type") + .is_none_or(|t| t.as_name().is_some_and(|name| name.is("ObjStm"))) + { + self.note_unread(Some(id), "referenced as an object stream, and is not one"); + continue; + } + let count = self + .get(&stream.dict, "N") + .and_then(Object::as_i64) + .unwrap_or(0); + let first = self + .get(&stream.dict, "First") + .and_then(Object::as_i64) + .unwrap_or(0); + let bytes = match self.decode_stream(Some(id), stream) { + Ok(Some(bytes)) => bytes, + Ok(None) => continue, + Err(e) => { + self.note_unread(Some(id), format!("object stream: {e}")); + continue; + } + }; + + let Ok(first) = usize::try_from(first) else { + self.note_unread(Some(id), "object stream: /First is not an offset"); + continue; + }; + // The header is `N` pairs of ` `, each + // offset relative to /First. + let mut header = Lexer::new(&bytes); + let mut offsets = Vec::new(); + for _ in 0..count.max(0) { + let (Ok(Some(number)), Ok(Some(offset))) = ( + header.object().map(|o| o.as_i64()), + header.object().map(|o| o.as_i64()), + ) else { + break; + }; + match (u32::try_from(number), usize::try_from(offset)) { + (Ok(number), Ok(offset)) => offsets.push((number, offset)), + _ => break, + } + } + + let wanted: BTreeSet = members.iter().map(|(number, _)| *number).collect(); + for (number, offset) in offsets { + if !wanted.contains(&number) || self.objects.contains_key(&number) { + continue; + } + let Some(at) = first.checked_add(offset).filter(|at| *at < bytes.len()) else { + self.note_unread( + Some(ObjectId::new(number, 0)), + format!("object stream {container}: its offset is past the end"), + ); + continue; + }; + match Lexer::at(&bytes, at).object() { + // §7.5.7: every object in an object stream has generation 0. + Ok(object) => { + self.objects.insert( + number, + Entry { + generation: 0, + object, + }, + ); + } + Err(e) => self.note_unread( + Some(ObjectId::new(number, 0)), + format!("in object stream {container}: {e}"), + ), + } + } + } + } + + /// `/Root`, or the object that calls itself the catalog when the trailer + /// does not say. + fn find_catalog(&self) -> Result { + if let Some(id) = self.trailer.get("Root").and_then(Object::as_reference) + && self.object(id).is_some_and(|root| root.is_type("Catalog")) + { + return Ok(id); + } + let found = self + .objects + .iter() + .find(|(_, entry)| entry.object.is_type("Catalog")) + .map(|(number, entry)| ObjectId::new(*number, entry.generation)); + found.ok_or_else(|| { + Error::Format(format!( + "{}: no document catalog; the trailer names none and no object calls itself one", + self.name + )) + }) + } +} + +static EMPTY: Dictionary = Dictionary::empty(); + +/// `%PDF-`, and where in the file it starts. +fn header(data: &[u8]) -> Result<(usize, String)> { + let window = &data[..data.len().min(HEADER_WINDOW)]; + let Some(at) = find(window, b"%PDF-") else { + return Err(Error::Format( + "not a PDF: no `%PDF-` header in the first kilobyte".into(), + )); + }; + let version: String = data[at + 5..] + .iter() + .take(8) + .take_while(|b| b.is_ascii_digit() || **b == b'.') + .map(|b| *b as char) + .collect(); + if version.is_empty() { + return Err(Error::Format( + "not a PDF: the `%PDF-` header states no version".into(), + )); + } + Ok((at, version)) +} + +/// Refuse an encrypted document, naming the encryption. +/// +/// The `/Encrypt` dictionary is read on its own, before the body: §7.6.1 keeps +/// it in the clear precisely so a reader can find out what it is up against +/// without being able to decrypt anything. +fn refuse_encryption(data: &[u8], xref: &Xref, shift: usize) -> Result<()> { + let Some(entry) = xref.trailer.get("Encrypt") else { + return Ok(()); + }; + let dict = match entry { + Object::Dictionary(dict) => Some(dict.clone()), + Object::Reference(id) => match xref.slots.get(&id.number) { + Some(Slot::InFile { offset, .. }) => usize::try_from(*offset) + .ok() + .and_then(|at| at.checked_add(shift)) + .filter(|at| *at < data.len()) + .and_then(|at| Lexer::at(data, at).indirect().ok()) + .and_then(|(_, object)| object.as_dict().cloned()), + _ => None, + }, + _ => None, + }; + + let what = dict + .as_ref() + .map(encrypt::describe) + .unwrap_or_else(|| "an encryption the file does not describe".into()); + Err(Error::Format(format!( + "encrypted with {what}; mirsam does not decrypt, so this document is unread rather than \ + clean. Rebuild it without protection, or remove the protection in the application that \ + wrote it" + ))) +} + +/// Parse every object the table places in the body. +fn load(data: &[u8], xref: &Xref, shift: usize, unread: &mut Vec) -> BTreeMap { + let mut objects = BTreeMap::new(); + for (&number, slot) in &xref.slots { + let Slot::InFile { offset, generation } = *slot else { + continue; + }; + let Ok(offset) = usize::try_from(offset) else { + continue; + }; + let id = ObjectId::new(number, generation); + + // The stored offset, then the same offset past a prologue. A file with + // bytes before its header may have been written either way round. + let mut parsed = None; + for at in [Some(offset), offset.checked_add(shift)] + .into_iter() + .flatten() + { + if at >= data.len() { + continue; + } + match Lexer::at(data, at).indirect() { + Ok((found, object)) if found.number == number => { + parsed = Some((found.generation, object)); + break; + } + _ => continue, + } + } + match parsed { + Some((generation, object)) => { + objects.insert(number, Entry { generation, object }); + } + None => unread.push(Unread { + object: Some(id), + reason: format!( + "the cross-reference table puts it at byte {offset}, and it is not there" + ), + }), + } + } + objects +} diff --git a/crates/mirsam-pdf/src/encrypt.rs b/crates/mirsam-pdf/src/encrypt.rs new file mode 100644 index 0000000..f38559e --- /dev/null +++ b/crates/mirsam-pdf/src/encrypt.rs @@ -0,0 +1,88 @@ +//! Encryption: what it is, said precisely, so the document can be refused. +//! +//! mirsam does not decrypt. That is not a gap to be filled later — a tool that +//! opened protected documents by trying the empty password would be a tool +//! nobody could run on a file they were given, and one that asked for a +//! password would have to hold it. +//! +//! What matters is what the refusal *says*. An encrypted PDF whose strings and +//! streams cannot be read produces no text, and a reader that stopped there +//! would report a document with no defects — the exact failure ADR 0009 exists +//! to prevent, arriving as a whole file rather than one source inside it. So +//! the refusal names the algorithm, from the `/Encrypt` dictionary alone, which +//! is never itself encrypted (§7.6.1). A person reading it knows whether they +//! are looking at a permissions-only wrapper they can strip or a document they +//! need a password for, without opening anything. + +use crate::object::{Dictionary, Object}; + +/// The `/Encrypt` dictionary in one line: the algorithm, the handler and the +/// version the file states. +/// +/// Nothing here is inferred. Where the dictionary does not say, the answer says +/// so — an undeclared algorithm is reported as undeclared, never guessed at, +/// because the point of the message is that a reader can trust it. +pub fn describe(encrypt: &Dictionary) -> String { + let handler = encrypt + .get("Filter") + .and_then(Object::as_name) + .map(ToString::to_string) + .unwrap_or_else(|| "/Standard".into()); + let v = encrypt.get("V").and_then(Object::as_i64); + let r = encrypt.get("R").and_then(Object::as_i64); + let bits = encrypt.get("Length").and_then(Object::as_i64).unwrap_or(40); + + let algorithm = match v { + Some(1) => "RC4 40-bit".to_string(), + Some(2) | Some(3) => format!("RC4 {bits}-bit"), + Some(4) | Some(5) => crypt_filter(encrypt).unwrap_or_else(|| match v { + Some(5) => "AES-256".into(), + _ => format!("RC4 {bits}-bit"), + }), + Some(0) => "an algorithm the document does not state".to_string(), + _ => "an undeclared algorithm".to_string(), + }; + + let version = match (v, r) { + (Some(v), Some(r)) => format!(", V{v} R{r}"), + (Some(v), None) => format!(", V{v}"), + (None, Some(r)) => format!(", R{r}"), + (None, None) => String::new(), + }; + format!("{algorithm} ({handler} security handler{version})") +} + +/// §7.6.5's crypt filters, where PDF 1.5 and later actually state the method. +/// +/// `/StdCF` is the name every writer uses, but the *default* filter for streams +/// is whatever `/StmF` names, so that is asked first: a file whose stream +/// filter is not the one called `StdCF` would otherwise be described by a +/// dictionary entry nothing in it uses. +fn crypt_filter(encrypt: &Dictionary) -> Option { + let filters = encrypt.get("CF")?.as_dict()?; + let named = encrypt + .get("StmF") + .and_then(Object::as_name) + .and_then(|name| name.as_str().map(str::to_string)) + .unwrap_or_else(|| "StdCF".into()); + let filter = filters + .get(&named) + .or_else(|| filters.get("StdCF")) + .and_then(Object::as_dict)?; + + let method = filter.get("CFM").and_then(Object::as_name)?; + let bits = filter + .get("Length") + .and_then(Object::as_i64) + // §7.6.5 lets a crypt filter state its length in bytes; the trailer + // dictionary states it in bits. Both spellings appear. + .map(|n| if n <= 64 { n * 8 } else { n }); + + Some(match method { + m if m.is("AESV2") => "AES-128".into(), + m if m.is("AESV3") => "AES-256".into(), + m if m.is("V2") => format!("RC4 {}-bit", bits.unwrap_or(128)), + m if m.is("None") => "no stream encryption".into(), + other => format!("{other}"), + }) +} diff --git a/crates/mirsam-pdf/src/filter.rs b/crates/mirsam-pdf/src/filter.rs new file mode 100644 index 0000000..2e4e117 --- /dev/null +++ b/crates/mirsam-pdf/src/filter.rs @@ -0,0 +1,413 @@ +//! Stream filters: §7.4, the half of it that carries text. +//! +//! ## Which filters are implemented, and why the rest are named rather than +//! failed +//! +//! `FlateDecode`, `LZWDecode`, `ASCIIHexDecode`, `ASCII85Decode` and +//! `RunLengthDecode` are general-purpose: a content stream, a `ToUnicode` CMap, +//! an object stream or a cross-reference stream may arrive through any of them, +//! and a document whose text is behind one is a document this tool must read. +//! +//! `DCTDecode`, `JPXDecode`, `CCITTFaxDecode` and `JBIG2Decode` are image +//! codecs. Decoding one produces a raster, and mirsam has said it will never +//! rasterise — so a stream behind one is not text this tool failed to read, it +//! is text that is not in the file at all. [`Decoded::Stopped`] carries the +//! filter's name so the document can say which, which is ADR 0009's shape: a +//! source the adapter could not read is part of the report, and a scanned page +//! must never come back as a page with no defects. +//! +//! `Crypt` is the same shape for a different reason — a document reaching this +//! module through it was not refused at [`crate::Pdf::open`], which would be a +//! bug rather than a file. +//! +//! ## A truncated stream keeps what it had +//! +//! Real files are truncated, and inflate reports that as an error after +//! producing perfectly good bytes. Discarding them would turn a document with +//! a damaged tail into a document with no text — the failure this project +//! treats as worse than a miss. So a decode that produced output keeps it, and +//! only one that produced nothing is an error. + +use flate2::read::{DeflateDecoder, ZlibDecoder}; +use mirsam_core::error::{Error, Result}; +use std::io::Read; + +use crate::object::{Dictionary, Name, Object}; + +/// What came out of a filter chain. +pub enum Decoded { + /// Every filter in the chain was applied. + Bytes(Vec), + /// The chain reached a filter this crate does not decode. Named, never + /// silently dropped. + Stopped { filter: Name }, +} + +/// The filters that end a chain rather than continue it: an image codec +/// produces pixels, and mirsam does not rasterise. +fn is_image_codec(name: &Name) -> bool { + ["DCTDecode", "JPXDecode", "CCITTFaxDecode", "JBIG2Decode"] + .iter() + .any(|codec| name.is(codec)) +} + +/// Run `data` through `filters` in order, with `parms[i]` the parameters of +/// `filters[i]`. +/// +/// A missing entry in `parms` is an empty dictionary, which is what an absent +/// `/DecodeParms` means. +pub fn decode(data: &[u8], filters: &[Name], parms: &[Dictionary]) -> Result { + let mut bytes = data.to_vec(); + let empty = Dictionary::new(); + for (index, filter) in filters.iter().enumerate() { + let parms = parms.get(index).unwrap_or(&empty); + bytes = match filter { + f if f.is("FlateDecode") || f.is("Fl") => predict(inflate(&bytes)?, parms)?, + f if f.is("LZWDecode") || f.is("LZW") => { + let early = parms + .get("EarlyChange") + .and_then(Object::as_i64) + .unwrap_or(1); + predict(lzw(&bytes, early != 0)?, parms)? + } + f if f.is("ASCIIHexDecode") || f.is("AHx") => ascii_hex(&bytes), + f if f.is("ASCII85Decode") || f.is("A85") => ascii85(&bytes)?, + f if f.is("RunLengthDecode") || f.is("RL") => run_length(&bytes), + // §7.4.1: an identity filter, which some writers spell explicitly. + f if f.is("Identity") => bytes, + f if is_image_codec(f) || f.is("Crypt") => { + return Ok(Decoded::Stopped { + filter: filter.clone(), + }); + } + _ => { + return Ok(Decoded::Stopped { + filter: filter.clone(), + }); + } + }; + } + Ok(Decoded::Bytes(bytes)) +} + +/// zlib, then raw deflate. +/// +/// Both spellings are in the wild: §7.4.4 requires the zlib wrapper, and a +/// long tail of writers emits the bare deflate stream. Trying the wrapper +/// first keeps the conformant case exact. +fn inflate(data: &[u8]) -> Result> { + let trimmed = { + let start = data + .iter() + .position(|b| !crate::object::is_whitespace(*b)) + .unwrap_or(data.len()); + &data[start..] + }; + + let mut out = Vec::new(); + let zlib = ZlibDecoder::new(trimmed).read_to_end(&mut out); + if zlib.is_ok() || !out.is_empty() { + return Ok(out); + } + + out.clear(); + let raw = DeflateDecoder::new(trimmed).read_to_end(&mut out); + if raw.is_ok() || !out.is_empty() { + return Ok(out); + } + Err(Error::Format(format!( + "FlateDecode: {}", + zlib.err() + .map(|e| e.to_string()) + .unwrap_or_else(|| "not a deflate stream".into()) + ))) +} + +/// §7.4.4.4's predictors, which a stream carries when it holds rows of equal +/// width — a cross-reference stream always does, and an image often does. +fn predict(data: Vec, parms: &Dictionary) -> Result> { + let int = |key: &str, default: i64| parms.get(key).and_then(Object::as_i64).unwrap_or(default); + let predictor = int("Predictor", 1); + if predictor < 2 { + return Ok(data); + } + let colors = int("Colors", 1).clamp(1, 32) as usize; + let bpc = int("BitsPerComponent", 8).clamp(1, 32) as usize; + let columns = int("Columns", 1).max(1) as usize; + + let bpp = (colors * bpc).div_ceil(8).max(1); + let row = (colors * bpc * columns).div_ceil(8); + + if predictor == 2 { + return Ok(tiff_predictor(data, colors, bpc, row)); + } + + // PNG: each row is preceded by its filter type. + let mut out = Vec::with_capacity(data.len()); + let mut previous = vec![0u8; row]; + for chunk in data.chunks(row + 1) { + // A truncated final row is dropped rather than un-filtered against + // bytes that are not there. + if chunk.len() < 2 { + break; + } + let (kind, encoded) = (chunk[0], &chunk[1..]); + let mut current = encoded.to_vec(); + for i in 0..current.len() { + let left = if i >= bpp { current[i - bpp] } else { 0 }; + let up = *previous.get(i).unwrap_or(&0); + let up_left = if i >= bpp { + *previous.get(i - bpp).unwrap_or(&0) + } else { + 0 + }; + current[i] = match kind { + 0 => current[i], + 1 => current[i].wrapping_add(left), + 2 => current[i].wrapping_add(up), + 3 => current[i].wrapping_add(((u16::from(left) + u16::from(up)) / 2) as u8), + 4 => current[i].wrapping_add(paeth(left, up, up_left)), + other => { + return Err(Error::Format(format!( + "unknown PNG predictor filter type {other}" + ))); + } + }; + } + out.extend_from_slice(¤t); + previous = current; + previous.resize(row, 0); + } + Ok(out) +} + +fn tiff_predictor(mut data: Vec, colors: usize, bpc: usize, row: usize) -> Vec { + // Only the byte-aligned case is undone. Sub-byte components appear in + // images and never in a stream this crate reads for text, and guessing at + // one would corrupt bytes rather than leave them alone. + if bpc != 8 || row == 0 { + return data; + } + for start in (0..data.len()).step_by(row) { + let end = (start + row).min(data.len()); + for i in (start + colors)..end { + data[i] = data[i].wrapping_add(data[i - colors]); + } + } + data +} + +fn paeth(left: u8, up: u8, up_left: u8) -> u8 { + let p = i16::from(left) + i16::from(up) - i16::from(up_left); + let (dl, du, dul) = ( + (p - i16::from(left)).abs(), + (p - i16::from(up)).abs(), + (p - i16::from(up_left)).abs(), + ); + if dl <= du && dl <= dul { + left + } else if du <= dul { + up + } else { + up_left + } +} + +/// §7.4.2. White space is ignored, `>` ends the data, and an odd final digit +/// is padded with a zero. +fn ascii_hex(data: &[u8]) -> Vec { + let mut out = Vec::with_capacity(data.len() / 2); + let mut half: Option = None; + for &b in data { + match b { + b'>' => break, + b if b.is_ascii_hexdigit() => { + let value = match b { + b'0'..=b'9' => b - b'0', + b'a'..=b'f' => b - b'a' + 10, + _ => b - b'A' + 10, + }; + match half.take() { + Some(hi) => out.push(hi * 16 + value), + None => half = Some(value), + } + } + _ => {} + } + } + if let Some(hi) = half { + out.push(hi * 16); + } + out +} + +/// §7.4.3, base-85. `z` is four zero bytes, `~>` ends the data, and a partial +/// final group encodes fewer than four bytes. +fn ascii85(data: &[u8]) -> Result> { + let data = data.strip_prefix(b"<~").unwrap_or(data); + let mut out = Vec::with_capacity(data.len() * 4 / 5); + let mut group = [0u8; 5]; + let mut filled = 0usize; + + let flush = |group: &[u8; 5], filled: usize, out: &mut Vec| { + if filled == 0 { + return; + } + let mut value: u32 = 0; + for (i, digit) in group.iter().enumerate() { + // A short final group is padded with the highest digit, `u`. + let digit = if i < filled { *digit } else { 84 }; + value = value.wrapping_mul(85).wrapping_add(u32::from(digit)); + } + out.extend_from_slice(&value.to_be_bytes()[..filled - 1]); + }; + + for b in data.iter().copied() { + match b { + b'~' => break, + b'z' if filled == 0 => out.extend_from_slice(&[0, 0, 0, 0]), + b'z' => return Err(Error::Format("ASCII85Decode: `z` inside a group".into())), + b'!'..=b'u' => { + group[filled] = b - b'!'; + filled += 1; + if filled == 5 { + flush(&group, 5, &mut out); + filled = 0; + } + } + b if crate::object::is_whitespace(b) => {} + other => { + return Err(Error::Format(format!( + "ASCII85Decode: `{}` is not a base-85 digit", + other as char + ))); + } + } + } + if filled == 1 { + return Err(Error::Format( + "ASCII85Decode: a final group of one digit encodes no bytes".into(), + )); + } + flush(&group, filled, &mut out); + Ok(out) +} + +/// §7.4.5. A length byte under 128 introduces that many literal bytes plus +/// one; over 128 it repeats the next byte `257 - length` times; 128 ends the +/// data. +fn run_length(data: &[u8]) -> Vec { + let mut out = Vec::new(); + let mut i = 0usize; + while i < data.len() { + let length = data[i]; + i += 1; + match length { + 128 => break, + 0..=127 => { + let count = usize::from(length) + 1; + let end = (i + count).min(data.len()); + out.extend_from_slice(&data[i..end]); + i = end; + } + _ => { + let Some(&byte) = data.get(i) else { break }; + out.extend(std::iter::repeat_n(byte, 257 - usize::from(length))); + i += 1; + } + } + } + out +} + +/// §7.4.4.2's variable-code LZW, which is TIFF's rather than GIF's: codes grow +/// from nine bits, 256 clears the table and 257 ends the data. +/// +/// `early` is `/EarlyChange`, and it is not cosmetic — it decides whether the +/// code width grows one entry before the table is full. A decoder on the wrong +/// side of it produces plausible-looking rubbish rather than an error. +fn lzw(data: &[u8], early: bool) -> Result> { + const CLEAR: u16 = 256; + const END: u16 = 257; + + let mut table: Vec> = Vec::new(); + let reset = |table: &mut Vec>| { + table.clear(); + table.extend((0u16..=255).map(|b| vec![b as u8])); + table.push(Vec::new()); // 256, clear + table.push(Vec::new()); // 257, end + }; + reset(&mut table); + + let mut out = Vec::new(); + let mut width = 9u32; + let mut previous: Option = None; + let mut bits: u32 = 0; + let mut held: u32 = 0; + + for &byte in data { + bits = (bits << 8) | u32::from(byte); + held += 8; + while held >= width { + let code = ((bits >> (held - width)) & ((1 << width) - 1)) as u16; + held -= width; + + match code { + CLEAR => { + reset(&mut table); + width = 9; + previous = None; + } + END => return Ok(out), + _ => { + let entry = match table.get(usize::from(code)) { + Some(entry) => entry.clone(), + None => { + // The one legal forward reference: the code being + // defined by this very step. + let Some(previous) = previous.and_then(|p| table.get(usize::from(p))) + else { + return Err(Error::Format(format!( + "LZWDecode: code {code} is not in the table" + ))); + }; + let mut entry = previous.clone(); + let Some(&first) = previous.first() else { + return Err(Error::Format( + "LZWDecode: a code expanding to nothing".into(), + )); + }; + entry.push(first); + entry + } + }; + out.extend_from_slice(&entry); + if let Some(previous) = previous + && let Some(prefix) = table.get(usize::from(previous)) + { + let mut new = prefix.clone(); + new.push(entry[0]); + table.push(new); + } + previous = Some(code); + + let limit = table.len() as u32 + u32::from(early); + width = match limit { + ..512 => 9, + 512..1024 => 10, + 1024..2048 => 11, + _ => 12, + }; + if table.len() >= 4096 { + reset(&mut table); + width = 9; + previous = None; + } + } + } + } + } + // Data that ran out before its end-of-data code: keep what decoded, for the + // reason the module note gives. + Ok(out) +} diff --git a/crates/mirsam-pdf/src/lexer.rs b/crates/mirsam-pdf/src/lexer.rs new file mode 100644 index 0000000..6cf7b9c --- /dev/null +++ b/crates/mirsam-pdf/src/lexer.rs @@ -0,0 +1,485 @@ +//! Bytes to objects: the syntax half of ISO 32000-1 §7.2 and §7.3. +//! +//! A borrowing scanner over the whole file, so an object can be read at an +//! offset the cross-reference table names without copying anything before it. +//! +//! ## A stream's length is checked, not believed +//! +//! `/Length` may be an indirect reference — `<< /Length 12 0 R >>` — which is +//! a value the lexer cannot resolve, because resolving it means consulting the +//! cross-reference table that is itself being parsed out of streams. Worse, +//! plenty of real writers get the direct form *wrong* by a byte or two. +//! +//! So the length is treated as a hint and verified: if `/Length` is a direct +//! integer and the bytes it names are followed by `endstream`, it is taken. In +//! every other case — indirect, absent, or simply wrong — the data runs to the +//! next `endstream`. One rule covers all three, and no caller has to hand the +//! lexer a resolver it cannot have yet. + +use mirsam_core::error::{Error, Result}; + +use crate::object::{Dictionary, Name, Object, ObjectId, Stream, is_regular, is_whitespace}; + +/// How deep a nesting of arrays and dictionaries is read before the file is +/// called malformed. Well past anything a document contains, and short of the +/// depth at which a hand-written adversarial file would exhaust the stack. +const MAX_DEPTH: usize = 128; + +pub struct Lexer<'a> { + data: &'a [u8], + pos: usize, +} + +/// A parse failure, always carrying the byte offset it happened at: an object +/// layer that reported "malformed dictionary" without saying where would be +/// unusable against a document nobody can open in an editor. +fn malformed(at: usize, what: impl std::fmt::Display) -> Error { + Error::Format(format!("at byte {at}: {what}")) +} + +impl<'a> Lexer<'a> { + pub fn new(data: &'a [u8]) -> Self { + Self { data, pos: 0 } + } + + /// A lexer positioned at `pos`, which is what a cross-reference entry + /// hands over. + pub fn at(data: &'a [u8], pos: usize) -> Self { + Self { data, pos } + } + + pub fn pos(&self) -> usize { + self.pos + } + + pub fn seek(&mut self, pos: usize) { + self.pos = pos.min(self.data.len()); + } + + fn peek(&self) -> Option { + self.data.get(self.pos).copied() + } + + fn peek_at(&self, ahead: usize) -> Option { + self.data.get(self.pos + ahead).copied() + } + + /// White space and comments, which §7.2.4 says are equivalent to a single + /// space anywhere a token boundary is allowed. + pub fn skip_space(&mut self) { + loop { + match self.peek() { + Some(b) if is_whitespace(b) => self.pos += 1, + Some(b'%') => { + while let Some(b) = self.peek() { + if b == b'\r' || b == b'\n' { + break; + } + self.pos += 1; + } + } + _ => return, + } + } + } + + /// Consume `word` if it is the next token, and answer whether it was. + /// + /// The token has to *end* where the keyword does: without that check, + /// `endstream` would match a lexer looking for `end`, and `obj` would + /// match inside `objstm`. + pub fn keyword(&mut self, word: &str) -> bool { + self.skip_space(); + let end = self.pos + word.len(); + if self.data.get(self.pos..end) != Some(word.as_bytes()) { + return false; + } + if self.data.get(end).copied().is_some_and(is_regular) { + return false; + } + self.pos = end; + true + } + + /// The run of regular characters at the cursor: a keyword, a number, or + /// whatever a malformed file has there instead. + fn token(&mut self) -> &'a [u8] { + self.skip_space(); + let start = self.pos; + while self.peek().is_some_and(is_regular) { + self.pos += 1; + } + &self.data[start..self.pos] + } + + /// One object, following a reference no further than recognising it. + pub fn object(&mut self) -> Result { + self.object_at_depth(0) + } + + fn object_at_depth(&mut self, depth: usize) -> Result { + if depth > MAX_DEPTH { + return Err(malformed(self.pos, "objects nested past any real document")); + } + self.skip_space(); + let at = self.pos; + match self.peek() { + None => Err(malformed(at, "expected an object, found end of file")), + Some(b'/') => Ok(Object::Name(self.name()?)), + Some(b'(') => Ok(Object::String(self.literal_string()?)), + Some(b'[') => { + self.pos += 1; + let mut items = Vec::new(); + loop { + self.skip_space(); + match self.peek() { + Some(b']') => { + self.pos += 1; + return Ok(Object::Array(items)); + } + None => return Err(malformed(at, "array with no closing `]`")), + _ => items.push(self.object_at_depth(depth + 1)?), + } + } + } + Some(b'<') if self.peek_at(1) == Some(b'<') => self.dictionary_or_stream(depth), + Some(b'<') => Ok(Object::String(self.hex_string()?)), + Some(b) if b.is_ascii_digit() || matches!(b, b'+' | b'-' | b'.') => self.numeric(), + Some(stray @ (b']' | b'>' | b')' | b'}')) => { + Err(malformed(at, format!("stray `{}`", stray as char))) + } + Some(_) => match self.token() { + b"true" => Ok(Object::Boolean(true)), + b"false" => Ok(Object::Boolean(false)), + b"null" => Ok(Object::Null), + [] => Err(malformed(at, "expected an object, found a delimiter")), + other => Err(malformed( + at, + format!("unknown keyword {:?}", String::from_utf8_lossy(other)), + )), + }, + } + } + + /// A number, or the `n g R` reference two of them may begin. + /// + /// Reference recognition is lookahead with a rewind rather than a + /// post-pass over a token list: `1 0 R` and `1 0` differ only in what + /// follows, and every context that admits one admits the other. + fn numeric(&mut self) -> Result { + let at = self.pos; + let first = number(self.token(), at)?; + + if let Object::Integer(number) = first + && let Ok(number) = u32::try_from(number) + { + let rewind = self.pos; + let generation = self.token(); + if !generation.is_empty() + && generation.iter().all(u8::is_ascii_digit) + && let Ok(generation) = String::from_utf8_lossy(generation).parse::() + && self.keyword("R") + { + return Ok(Object::Reference(ObjectId::new(number, generation))); + } + self.pos = rewind; + } + Ok(first) + } + + fn name(&mut self) -> Result { + let at = self.pos; + self.pos += 1; // the `/` + let mut bytes = Vec::new(); + while let Some(b) = self.peek() { + if !is_regular(b) { + break; + } + self.pos += 1; + if b != b'#' { + bytes.push(b); + continue; + } + // `#xx`. A `#` not followed by two hex digits is kept literally: + // the alternative is failing a whole document over one byte a + // writer forgot to escape, and every reader in the world opens it. + match (self.peek(), self.peek_at(1)) { + (Some(hi), Some(lo)) if hi.is_ascii_hexdigit() && lo.is_ascii_hexdigit() => { + bytes.push(hex(hi) * 16 + hex(lo)); + self.pos += 2; + } + _ => bytes.push(b'#'), + } + } + if bytes.is_empty() && self.pos == at + 1 { + // `/` alone is the empty name, which is legal. + } + Ok(Name::new(bytes)) + } + + /// `( ... )`, with §7.3.4.2's escapes and balanced inner parentheses. + fn literal_string(&mut self) -> Result> { + let at = self.pos; + self.pos += 1; // the `(` + let mut bytes = Vec::new(); + let mut depth = 1usize; + while let Some(b) = self.peek() { + self.pos += 1; + match b { + b'(' => { + depth += 1; + bytes.push(b); + } + b')' => { + depth -= 1; + if depth == 0 { + return Ok(bytes); + } + bytes.push(b); + } + b'\\' => { + let Some(escape) = self.peek() else { break }; + self.pos += 1; + match escape { + b'n' => bytes.push(b'\n'), + b'r' => bytes.push(b'\r'), + b't' => bytes.push(b'\t'), + b'b' => bytes.push(0x08), + b'f' => bytes.push(0x0c), + b'(' | b')' | b'\\' => bytes.push(escape), + // A backslash before an end-of-line joins the lines and + // contributes nothing. + b'\n' => {} + b'\r' => { + if self.peek() == Some(b'\n') { + self.pos += 1; + } + } + b'0'..=b'7' => { + let mut value = u32::from(escape - b'0'); + for _ in 0..2 { + match self.peek() { + Some(d @ b'0'..=b'7') => { + value = value * 8 + u32::from(d - b'0'); + self.pos += 1; + } + _ => break, + } + } + // §7.3.4.2: high-order overflow is ignored. + bytes.push((value & 0xff) as u8); + } + // "\q" is q. The backslash goes, the byte stays. + other => bytes.push(other), + } + } + _ => bytes.push(b), + } + } + Err(malformed(at, "string with no closing `)`")) + } + + /// `<...>`, hexadecimal. White space between digits is ignored and an odd + /// final digit is padded with a zero, both per §7.3.4.3. + fn hex_string(&mut self) -> Result> { + let at = self.pos; + self.pos += 1; // the `<` + let mut bytes = Vec::new(); + let mut half: Option = None; + while let Some(b) = self.peek() { + self.pos += 1; + match b { + b'>' => { + if let Some(hi) = half { + bytes.push(hi * 16); + } + return Ok(bytes); + } + b if b.is_ascii_hexdigit() => match half.take() { + Some(hi) => bytes.push(hi * 16 + hex(b)), + None => half = Some(hex(b)), + }, + b if is_whitespace(b) => {} + other => { + return Err(malformed( + self.pos - 1, + format!("`{}` in a hexadecimal string", other as char), + )); + } + } + } + Err(malformed(at, "hexadecimal string with no closing `>`")) + } + + fn dictionary_or_stream(&mut self, depth: usize) -> Result { + let at = self.pos; + self.pos += 2; // the `<<` + let mut dict = Dictionary::new(); + loop { + self.skip_space(); + match self.peek() { + Some(b'>') if self.peek_at(1) == Some(b'>') => { + self.pos += 2; + break; + } + Some(b'/') => { + let key = self.name()?; + let value = self.object_at_depth(depth + 1)?; + dict.insert(key, value); + } + None => return Err(malformed(at, "dictionary with no closing `>>`")), + Some(other) => { + return Err(malformed( + self.pos, + format!("expected a key, found `{}`", other as char), + )); + } + } + } + + let rewind = self.pos; + if !self.keyword("stream") { + self.pos = rewind; + return Ok(Object::Dictionary(dict)); + } + // §7.3.8.1: CRLF or LF after the keyword, never CR alone. A file that + // uses CR alone is still read: dropping it costs nothing, and treating + // it as data would shift every byte of the stream. + match (self.peek(), self.peek_at(1)) { + (Some(b'\r'), Some(b'\n')) => self.pos += 2, + (Some(b'\n'), _) | (Some(b'\r'), _) => self.pos += 1, + _ => {} + } + let data = self.stream_data(&dict)?; + Ok(Object::Stream(Stream { dict, data })) + } + + /// The bytes between `stream` and `endstream`, leaving the cursor past the + /// closing keyword. See the module note on why `/Length` is verified. + fn stream_data(&mut self, dict: &Dictionary) -> Result> { + let start = self.pos; + + if let Some(length) = dict.get("Length").and_then(Object::as_i64) + && let Ok(length) = usize::try_from(length) + && let Some(end) = start.checked_add(length) + && end <= self.data.len() + { + let mut probe = Lexer::at(self.data, end); + if probe.keyword("endstream") { + self.pos = probe.pos; + return Ok(self.data[start..end].to_vec()); + } + } + + let Some(found) = find(&self.data[start..], b"endstream") else { + return Err(malformed(start, "stream with no `endstream`")); + }; + let mut end = start + found; + self.pos = end + b"endstream".len(); + // The end-of-line before `endstream` belongs to the syntax, not to the + // data, and only that one: a stream may legitimately end in a newline. + if end > start && self.data[end - 1] == b'\n' { + end -= 1; + } + if end > start && self.data[end - 1] == b'\r' { + end -= 1; + } + Ok(self.data[start..end].to_vec()) + } + + /// `n g obj endobj`, at the cursor. + /// + /// `endobj` is not required: a writer that omits it produces a file every + /// reader opens, and the object is complete without it. + pub fn indirect(&mut self) -> Result<(ObjectId, Object)> { + self.skip_space(); + let at = self.pos; + let number = self.token(); + let generation = self.token(); + let header = ( + String::from_utf8_lossy(number).parse::(), + String::from_utf8_lossy(generation).parse::(), + ); + let (Ok(number), Ok(generation)) = header else { + return Err(malformed(at, "expected ` obj`")); + }; + if !self.keyword("obj") { + return Err(malformed(at, "expected `obj` after the object number")); + } + let object = self.object()?; + let rewind = self.pos; + if !self.keyword("endobj") { + self.pos = rewind; + } + Ok((ObjectId::new(number, generation), object)) + } +} + +fn hex(b: u8) -> u8 { + match b { + b'0'..=b'9' => b - b'0', + b'a'..=b'f' => b - b'a' + 10, + _ => b - b'A' + 10, + } +} + +/// A number token, which §7.3.3 spells more loosely than it looks: `4.`, `-.2` +/// and `34.5` are all legal, and writers emit `--5` and `6.-2` often enough +/// that every reader tolerates them. +fn number(token: &[u8], at: usize) -> Result { + if token.is_empty() { + return Err(malformed(at, "expected a number")); + } + let text = String::from_utf8_lossy(token); + if !token.contains(&b'.') + && let Ok(value) = text.parse::() + { + return Ok(Object::Integer(value)); + } + if let Ok(value) = text.parse::() { + return Ok(Object::Real(value)); + } + // Salvage the leading well-formed number, which is what the malformed + // spellings above amount to. `--5` is 0 with a stray sign, not a failure + // to read the file. + let mut cleaned = String::new(); + for (i, c) in text.chars().enumerate() { + match c { + '+' | '-' if i == 0 => cleaned.push(c), + '.' if !cleaned.contains('.') => cleaned.push(c), + '0'..='9' => cleaned.push(c), + _ => break, + } + } + match cleaned.trim_start_matches(['+', '-']) { + "" | "." => Ok(Object::Integer(0)), + _ => cleaned + .parse::() + .map(Object::Real) + .map_err(|_| malformed(at, format!("unreadable number {text:?}"))), + } +} + +/// The first offset of `needle` in `haystack`. +/// +/// A plain scan: the needles here are short keywords and the streams they are +/// looked for in are the exception rather than the rule, so a search structure +/// would cost more to build than it saves. +pub(crate) fn find(haystack: &[u8], needle: &[u8]) -> Option { + if needle.is_empty() || haystack.len() < needle.len() { + return None; + } + haystack + .windows(needle.len()) + .position(|window| window == needle) +} + +/// The last offset of `needle` in `haystack`. +pub(crate) fn rfind(haystack: &[u8], needle: &[u8]) -> Option { + if needle.is_empty() || haystack.len() < needle.len() { + return None; + } + haystack + .windows(needle.len()) + .rposition(|window| window == needle) +} diff --git a/crates/mirsam-pdf/src/lib.rs b/crates/mirsam-pdf/src/lib.rs new file mode 100644 index 0000000..6bdf23c --- /dev/null +++ b/crates/mirsam-pdf/src/lib.rs @@ -0,0 +1,54 @@ +//! PDF object layer for mirsam. +//! +//! The package layer of a format that is not a package. `mirsam-ooxml` opens a +//! ZIP and asks for a part by name; this crate opens a PDF and asks for an +//! object by number — from the body, from inside a compressed object stream, +//! or from a scan of the file when the cross-reference table has gone stale. +//! +//! What it does: classic cross-reference tables and cross-reference streams, +//! `/Prev` chains, hybrid-reference files, object streams, the general-purpose +//! stream filters, the page tree with its inheritable attributes, and content +//! streams decoded and joined. +//! +//! What it deliberately does not do: **decrypt**, **rasterise**, and **write**. +//! An encrypted document is refused by name at [`Pdf::open`], an image codec +//! is named rather than decoded, and there is no writer here or planned — a +//! broken Arabic PDF is rebuilt from its source document, not edited in place. +//! `DocumentWriter` is not implemented, and the absence is the design. +//! +//! ```no_run +//! use mirsam_pdf::Pdf; +//! +//! let pdf = Pdf::open(std::path::Path::new("report.pdf"))?; +//! for page in pdf.pages() { +//! println!("page {} — {} bytes of content", page.index + 1, page.content(&pdf).len()); +//! } +//! for source in pdf.unread() { +//! println!("not read: {source}"); +//! } +//! # Ok::<(), mirsam_core::error::Error>(()) +//! ``` +//! +//! Text extraction sits on top of this and is PLAN §6.2: what this crate hands +//! over is the object graph and the bytes of the content streams, never a +//! string. A PDF stores glyphs with positions, and turning those back into +//! text is a reconstruction with its own rules about what may honestly be +//! claimed. +//! +//! See [`docs/adr/0010`] for why the parser is written here rather than taken. +//! +//! [`docs/adr/0010`]: https://github.com/aenawi/mirsam/blob/main/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md + +#![forbid(unsafe_code)] + +pub mod document; +pub mod encrypt; +pub mod filter; +pub mod lexer; +pub mod object; +pub mod page; +pub mod xref; + +pub use document::{Pdf, Reconstruction, Unread}; +pub use object::{Dictionary, Name, Object, ObjectId, Stream}; +pub use page::Page; diff --git a/crates/mirsam-pdf/src/object.rs b/crates/mirsam-pdf/src/object.rs new file mode 100644 index 0000000..cdabfe4 --- /dev/null +++ b/crates/mirsam-pdf/src/object.rs @@ -0,0 +1,353 @@ +//! PDF's object types: what everything in the file is made of. +//! +//! ISO 32000-1 §7.3 gives eight basic types and two composites. This module is +//! the model alone — [`crate::lexer`] builds it out of bytes, [`crate::document`] +//! resolves the references between the pieces. +//! +//! ## A name is bytes, not a string +//! +//! §7.3.5 defines a name as a sequence of bytes with `#xx` escapes, and says +//! nothing about their encoding. A well-behaved writer emits ASCII, but a name +//! is also the one place a PDF stores a *key*, so decoding one lossily would +//! silently merge two dictionary entries that the file kept apart. [`Name`] +//! therefore holds the decoded bytes and compares on them; [`Name::as_str`] +//! offers the text form to whoever wants it and answers `None` rather than +//! guessing. + +use std::fmt; + +/// A PDF name: `/Type`, `/FlateDecode`, `/Root`. +/// +/// Holds the *decoded* bytes, so `/A#42` and `/AB` are one name, which is what +/// §7.3.5 requires and what a dictionary lookup depends on. +#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Default)] +pub struct Name(Vec); + +impl Name { + pub fn new(bytes: impl Into>) -> Self { + Self(bytes.into()) + } + + pub fn as_bytes(&self) -> &[u8] { + &self.0 + } + + /// The name as text, or `None` when it is not UTF-8. + /// + /// Never lossy: a caller comparing names compares bytes, and one printing + /// a name gets [`fmt::Display`], which escapes rather than replaces. + pub fn as_str(&self) -> Option<&str> { + std::str::from_utf8(&self.0).ok() + } + + /// Whether this is the name `other` spells. + pub fn is(&self, other: &str) -> bool { + self.0 == other.as_bytes() + } +} + +impl From<&str> for Name { + fn from(s: &str) -> Self { + Self(s.as_bytes().to_vec()) + } +} + +impl PartialEq for Name { + fn eq(&self, other: &str) -> bool { + self.is(other) + } +} + +/// Written the way the file writes it, with the escapes a PDF requires — so a +/// name in a diagnostic can be searched for in the document it came from. +impl fmt::Display for Name { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("/")?; + for &b in &self.0 { + match b { + b'!'..=b'~' if !is_delimiter(b) && b != b'#' => { + f.write_str(std::str::from_utf8(&[b]).unwrap_or("?"))? + } + _ => write!(f, "#{b:02X}")?, + } + } + Ok(()) + } +} + +impl fmt::Debug for Name { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{self}") + } +} + +/// The characters §7.2.2 reserves as delimiters. Everything else outside +/// whitespace is a regular character, and may appear in a name unescaped. +pub(crate) fn is_delimiter(b: u8) -> bool { + matches!( + b, + b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%' + ) +} + +/// The six bytes §7.2.2 calls white space. The null is one of them, which is +/// why a byte-level scanner cannot use `u8::is_ascii_whitespace` here. +pub(crate) fn is_whitespace(b: u8) -> bool { + matches!(b, 0 | b'\t' | b'\n' | 0x0c | b'\r' | b' ') +} + +/// Neither white space nor a delimiter: the bytes a name, a number or a +/// keyword is made of. +pub(crate) fn is_regular(b: u8) -> bool { + !is_whitespace(b) && !is_delimiter(b) +} + +/// The address of an indirect object: `12 0 R`. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Debug, Default)] +pub struct ObjectId { + pub number: u32, + pub generation: u16, +} + +impl ObjectId { + pub fn new(number: u32, generation: u16) -> Self { + Self { number, generation } + } +} + +/// Printed as the file writes a reference to it, so an `unread` entry names an +/// object in the words a person would use to find it in the file. +impl fmt::Display for ObjectId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} {} R", self.number, self.generation) + } +} + +/// A dictionary: an ordered list of name/value pairs. +/// +/// A `Vec` rather than a map, for two reasons. A PDF dictionary is small — +/// tens of entries at the outside — so a linear scan beats hashing, and file +/// order is preserved, which keeps a dictionary printed back in a diagnostic +/// looking like the one in the document. +/// +/// **A repeated key keeps the last value.** §7.3.7 leaves the case undefined. +/// Last wins because the only way a duplicate arises in practice is a writer +/// appending an entry it meant to supersede an earlier one. +#[derive(Clone, PartialEq, Default)] +pub struct Dictionary(Vec<(Name, Object)>); + +impl Dictionary { + pub fn new() -> Self { + Self::default() + } + + /// A dictionary with no entries, in a `const` context — what a document + /// hands back for a dictionary it does not have, so no caller has to + /// distinguish "no `/Resources`" from "an empty one". + pub const fn empty() -> Self { + Self(Vec::new()) + } + + /// The value stored under `key`, unresolved: a `/Length` that the file + /// wrote as `12 0 R` comes back as [`Object::Reference`]. Use + /// [`crate::Pdf::get`] to follow it. + pub fn get(&self, key: &str) -> Option<&Object> { + self.0 + .iter() + .find(|(name, _)| name.is(key)) + .map(|(_, value)| value) + } + + pub fn contains_key(&self, key: &str) -> bool { + self.get(key).is_some() + } + + pub fn insert(&mut self, name: Name, value: Object) { + match self.0.iter_mut().find(|(existing, _)| *existing == name) { + Some(slot) => slot.1 = value, + None => self.0.push((name, value)), + } + } + + pub fn iter(&self) -> impl Iterator { + self.0.iter().map(|(name, value)| (name, value)) + } + + pub fn len(&self) -> usize { + self.0.len() + } + + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl FromIterator<(Name, Object)> for Dictionary { + fn from_iter>(iter: I) -> Self { + let mut dict = Self::new(); + for (name, value) in iter { + dict.insert(name, value); + } + dict + } +} + +impl fmt::Debug for Dictionary { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("<<")?; + for (name, value) in &self.0 { + write!(f, " {name} {value:?}")?; + } + f.write_str(" >>") + } +} + +/// A stream: a dictionary and the bytes after its `stream` keyword. +/// +/// **`data` is what the file holds, still filtered.** Decoding is +/// [`crate::Pdf::decoded`]'s, and it is deliberately not done here: a stream +/// whose filter this crate does not implement has to be *named* rather than +/// failed, and only the document knows how to record that. +#[derive(Clone, PartialEq)] +pub struct Stream { + pub dict: Dictionary, + pub data: Vec, +} + +impl fmt::Debug for Stream { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{:?} stream({} bytes)", self.dict, self.data.len()) + } +} + +/// Any PDF object. +#[derive(Clone, PartialEq)] +pub enum Object { + Null, + Boolean(bool), + Integer(i64), + Real(f64), + /// A literal or hexadecimal string, decoded to the bytes it denotes. + /// Still bytes: which of PDFDocEncoding, UTF-16BE or a font's own codes + /// they are depends on where the string was found. + String(Vec), + Name(Name), + Array(Vec), + Dictionary(Dictionary), + Stream(Stream), + Reference(ObjectId), +} + +impl Object { + pub fn is_null(&self) -> bool { + matches!(self, Self::Null) + } + + pub fn as_bool(&self) -> Option { + match self { + Self::Boolean(b) => Some(*b), + _ => None, + } + } + + /// An integer, or a real that names one exactly. A PDF writer is free to + /// spell a count `3.0`, and a reader that refused it would reject the file + /// over its punctuation. + pub fn as_i64(&self) -> Option { + match self { + Self::Integer(n) => Some(*n), + Self::Real(r) if r.fract() == 0.0 && r.is_finite() => Some(*r as i64), + _ => None, + } + } + + pub fn as_f64(&self) -> Option { + match self { + Self::Integer(n) => Some(*n as f64), + Self::Real(r) => Some(*r), + _ => None, + } + } + + pub fn as_string(&self) -> Option<&[u8]> { + match self { + Self::String(bytes) => Some(bytes), + _ => None, + } + } + + pub fn as_name(&self) -> Option<&Name> { + match self { + Self::Name(name) => Some(name), + _ => None, + } + } + + pub fn as_array(&self) -> Option<&[Object]> { + match self { + Self::Array(items) => Some(items), + _ => None, + } + } + + /// The dictionary of a dictionary *or* of a stream. + /// + /// A stream is a dictionary with bytes attached, and every caller that + /// wants `/Type` or `/Filter` wants the same answer from both. + pub fn as_dict(&self) -> Option<&Dictionary> { + match self { + Self::Dictionary(dict) => Some(dict), + Self::Stream(stream) => Some(&stream.dict), + _ => None, + } + } + + pub fn as_stream(&self) -> Option<&Stream> { + match self { + Self::Stream(stream) => Some(stream), + _ => None, + } + } + + pub fn as_reference(&self) -> Option { + match self { + Self::Reference(id) => Some(*id), + _ => None, + } + } + + /// Whether this object's `/Type` is `name`. + /// + /// Answers `false` for a dictionary with no `/Type` at all: several are + /// optional in PDF, so absence is not disagreement, and a caller that + /// wants to infer a type from an object's shape has to say so. + pub fn is_type(&self, name: &str) -> bool { + self.as_dict() + .and_then(|dict| dict.get("Type")) + .and_then(Object::as_name) + .is_some_and(|found| found.is(name)) + } +} + +impl fmt::Debug for Object { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Null => f.write_str("null"), + Self::Boolean(b) => write!(f, "{b}"), + Self::Integer(n) => write!(f, "{n}"), + Self::Real(r) => write!(f, "{r}"), + Self::String(bytes) => write!(f, "({} bytes)", bytes.len()), + Self::Name(name) => write!(f, "{name}"), + Self::Array(items) => { + f.write_str("[")?; + for item in items { + write!(f, " {item:?}")?; + } + f.write_str(" ]") + } + Self::Dictionary(dict) => write!(f, "{dict:?}"), + Self::Stream(stream) => write!(f, "{stream:?}"), + Self::Reference(id) => write!(f, "{id}"), + } + } +} diff --git a/crates/mirsam-pdf/src/page.rs b/crates/mirsam-pdf/src/page.rs new file mode 100644 index 0000000..ea1aa79 --- /dev/null +++ b/crates/mirsam-pdf/src/page.rs @@ -0,0 +1,184 @@ +//! The page tree: §7.7.3, and the four attributes a page inherits down it. +//! +//! A PDF's pages are the leaves of a tree of `/Pages` nodes, and four +//! attributes — `/Resources`, `/MediaBox`, `/CropBox` and `/Rotate` — may be +//! stated on any node and are inherited by everything below it. `/Resources` is +//! the one that matters here: it is where a page's fonts are, and a text layer +//! that read it from the leaf alone would find no fonts at all on the very +//! common document that states them once at the root. +//! +//! So the walk carries them down and a [`Page`] is handed over with them +//! already filled in. A page's own value always wins; an inherited one is only +//! ever an answer to a question the page did not answer itself. +//! +//! ## The tree is walked, and scanned for only if that fails +//! +//! `/Kids` can point at anything, including back up at itself. The walk keeps +//! a set of the nodes it has been through and stops at a repeat, which is the +//! difference between reading a malformed document and hanging on one. If it +//! yields nothing at all — no catalog `/Pages`, or a root whose kids are all +//! missing — the objects are scanned for `/Type /Page` instead, and the +//! document says so through [`crate::Reconstruction::PageTree`]. + +use std::collections::BTreeSet; + +use crate::document::Pdf; +use crate::object::{Dictionary, Name, Object, ObjectId}; + +/// §7.7.3.4's inheritable page attributes, in the order the specification +/// tabulates them. +const INHERITED: [&str; 4] = ["Resources", "MediaBox", "CropBox", "Rotate"]; + +/// How deep the tree is walked. A page tree is balanced by every writer that +/// exists; this is a bound on malice, not on documents. +const MAX_DEPTH: usize = 64; + +/// One page, with what it inherited already on it. +#[derive(Clone, Debug)] +pub struct Page { + /// Position in the document, from zero — which is the number a person + /// counts pages by, and not the `/PageLabels` the document may print. + pub index: usize, + pub id: ObjectId, + /// The page dictionary, plus any of the four inheritable attributes an + /// ancestor supplied and the page did not state. + pub dict: Dictionary, +} + +impl Page { + /// The page's resource dictionary, where its fonts are named. + pub fn resources<'a>(&'a self, pdf: &'a Pdf) -> Option<&'a Dictionary> { + pdf.get(&self.dict, "Resources").and_then(Object::as_dict) + } + + /// The page's content streams, decoded and joined. + /// + /// `/Contents` is one stream or an array of them, and §7.8.2 says an array + /// is to be treated as a single stream *concatenated at the boundaries* — + /// so the join is a newline, without which the last operator of one stream + /// and the first of the next would run together into a token neither of + /// them wrote. + /// + /// A part behind a filter this crate does not decode contributes nothing + /// and is recorded in [`Pdf::unread`]: a page drawn as a scanned image + /// comes back empty *and named*, never merely empty. + /// + /// [`Pdf::unread`]: crate::Pdf::unread + pub fn content(&self, pdf: &Pdf) -> Vec { + let Some(contents) = self.dict.get("Contents") else { + return Vec::new(); + }; + let parts: Vec<&Object> = match pdf.resolve(contents) { + Object::Array(items) => items.iter().collect(), + _ => vec![contents], + }; + + let mut out = Vec::new(); + for part in parts { + let Some(id) = part.as_reference() else { + // §7.8.2 requires a content stream to be indirect. A direct + // one is not something to fail over, but there is nothing to + // decode: a stream object cannot appear inline. + continue; + }; + match pdf.decoded(id) { + Ok(Some(bytes)) => { + out.extend_from_slice(&bytes); + out.push(b'\n'); + } + Ok(None) => {} + Err(e) => pdf.note_unread(Some(id), format!("content stream: {e}")), + } + } + out + } +} + +/// Every page, in reading order, and whether the tree had to be reconstructed. +pub(crate) fn tree(pdf: &Pdf) -> (Vec, bool) { + let mut pages = Vec::new(); + let mut seen = BTreeSet::new(); + + if let Some(root) = pdf.catalog().get("Pages") { + let inherited = Dictionary::new(); + descend(pdf, root, &inherited, 0, &mut seen, &mut pages); + } + if !pages.is_empty() { + return (pages, false); + } + + // Nothing walkable. Take the objects that call themselves pages, in object + // order — which is the order they were written, and the closest thing to + // reading order a file with no tree has. + let mut scanned = Vec::new(); + for (id, object) in pdf.objects() { + if !object.is_type("Page") { + continue; + } + let Some(dict) = object.as_dict() else { + continue; + }; + scanned.push(Page { + index: scanned.len(), + id, + dict: dict.clone(), + }); + } + let rebuilt = !scanned.is_empty(); + (scanned, rebuilt) +} + +fn descend( + pdf: &Pdf, + node: &Object, + inherited: &Dictionary, + depth: usize, + seen: &mut BTreeSet, + pages: &mut Vec, +) { + if depth > MAX_DEPTH { + return; + } + if let Some(id) = node.as_reference() + && !seen.insert(id) + { + return; + } + let id = node.as_reference().unwrap_or_default(); + let Some(dict) = pdf.resolve(node).as_dict() else { + return; + }; + + // What this node passes down: what it was given, overridden by what it + // states itself. + let mut passing = inherited.clone(); + for key in INHERITED { + if let Some(value) = dict.get(key) { + passing.insert(Name::from(key), value.clone()); + } + } + + // A node with `/Kids` is an internal node whatever its `/Type` says, and a + // great many writers omit `/Type` entirely. Reading the shape rather than + // the label is the only way to walk those files. + if let Some(kids) = pdf.get(dict, "Kids").and_then(Object::as_array) { + for kid in kids { + descend(pdf, kid, &passing, depth + 1, seen, pages); + } + return; + } + + let mut page = dict.clone(); + for key in INHERITED { + if !page.contains_key(key) + && let Some(value) = passing.get(key) + { + page.insert(Name::from(key), value.clone()); + } + } + pages.push(Page { + index: pages.len(), + id, + dict: page, + }); +} diff --git a/crates/mirsam-pdf/src/xref.rs b/crates/mirsam-pdf/src/xref.rs new file mode 100644 index 0000000..6b63b09 --- /dev/null +++ b/crates/mirsam-pdf/src/xref.rs @@ -0,0 +1,486 @@ +//! The cross-reference: where each object is, and how to find that out when +//! the file lies about it. +//! +//! §7.5.4 gives two spellings and every real corpus contains both. A classic +//! *table* is plain text — `xref`, subsection headers, twenty-byte entries — +//! and is what anything writing PDF 1.4 or earlier emits. A cross-reference +//! *stream* is a binary table inside a `FlateDecode`d stream, introduced with +//! PDF 1.5 so that the table itself could be compressed alongside the object +//! streams it indexes. Both may be chained by `/Prev` into an arbitrarily long +//! history of incremental updates, and a *hybrid* file (§7.5.8.4) carries one +//! of each so that a 1.4 reader sees a subset of the objects a 1.5 reader sees. +//! +//! ## Newest wins, and it is the walk that enforces it +//! +//! An incremental update appends a new section and points its `/Prev` at the +//! old one, so the chain runs newest to oldest. Every insertion here is +//! therefore *first wins*: an object redefined by an update is found in the +//! section that redefined it, and the older entry is never consulted. The same +//! rule merges the trailers, which is why a `/Root` moved by an update is the +//! one that is used. +//! +//! ## And when the table is wrong +//! +//! Offsets go stale. A file concatenated by a script, truncated by a transfer +//! or edited by hand has a `startxref` pointing at nothing in particular, and +//! every viewer in the world opens it anyway by scanning for `obj`. So does +//! [`recover`] — because a document this tool refused while a reader opened it +//! would be a document reported as unreadable that is not, which is the worse +//! half of standing rule 4. + +use mirsam_core::error::{Error, Result}; +use std::collections::{BTreeMap, BTreeSet}; + +use crate::filter::{self, Decoded}; +use crate::lexer::{Lexer, find, rfind}; +use crate::object::{Dictionary, Name, Object, is_regular, is_whitespace}; + +/// How long a `/Prev` chain is followed. An incremental update per save is +/// normal; ten thousand of them is a file built by a loop. +const MAX_SECTIONS: usize = 4096; + +/// How far back from the end of the file `startxref` is looked for. §7.5.5 +/// puts it in the last line; a kilobyte covers every writer that appends +/// something after it. +const TAIL: usize = 2048; + +/// Where an object is. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Slot { + /// At a byte offset in the file. + InFile { offset: u64, generation: u16 }, + /// Inside an object stream, at `index` among the objects it holds. + InStream { stream: u32, index: u32 }, + /// Deleted. Kept rather than dropped, because a free entry in a newer + /// section is what *hides* an object an older section still names. + Free, +} + +/// A resolved cross-reference: every object's location, and the merged trailer. +#[derive(Debug, Default)] +pub struct Xref { + pub slots: BTreeMap, + pub trailer: Dictionary, +} + +impl Xref { + /// Record `slot` for `number` unless a newer section already did. + fn note(&mut self, number: u32, slot: Slot) { + self.slots.entry(number).or_insert(slot); + } + + /// Merge a section's trailer under the same first-wins rule. + fn merge_trailer(&mut self, trailer: &Dictionary) { + for (key, value) in trailer.iter() { + // The chain's own links are a property of the section that carried + // them, never of the document. + if key.is("Prev") || key.is("XRefStm") { + continue; + } + if !self.trailer.contains_key(key.as_str().unwrap_or_default()) { + self.trailer.insert(key.clone(), value.clone()); + } + } + } + + /// The objects stored in object streams, grouped by the stream holding + /// them: loading one stream answers for every object in it. + pub fn in_streams(&self) -> BTreeMap> { + let mut grouped: BTreeMap> = BTreeMap::new(); + for (&number, slot) in &self.slots { + if let Slot::InStream { stream, index } = *slot { + grouped.entry(stream).or_default().push((number, index)); + } + } + grouped + } +} + +/// Follow the chain from `startxref` and answer with every object's location. +/// +/// `shift` is the offset of `%PDF-` within the file, which §7.5.5 makes the +/// origin every stored offset is relative to. It is zero for a well-formed +/// document and non-zero for one with a prologue bolted on the front. +pub fn read(data: &[u8], shift: usize) -> Result { + let Some(start) = startxref(data) else { + return Err(Error::Format( + "no `startxref`: the file does not say where its cross-reference table is".into(), + )); + }; + + let mut xref = Xref::default(); + let mut next = Some(start); + let mut seen = BTreeSet::new(); + + for _ in 0..MAX_SECTIONS { + let Some(offset) = next.take() else { break }; + let Some(at) = locate(data, offset, shift) else { + return Err(Error::Format(format!( + "cross-reference section at byte {offset} is not there" + ))); + }; + if !seen.insert(at) { + // A `/Prev` pointing into a section already walked. Two writers + // producing the same offset is a loop, not a longer history. + break; + } + + let section = section(data, at)?; + + // §7.5.8.4: in a hybrid file the stream is the authority for the + // section, and the classic table beside it deliberately understates + // what the file holds. Reading it first makes it win. + if let Some(hybrid) = section.hybrid + && let Some(hybrid_at) = locate(data, hybrid, shift) + && seen.insert(hybrid_at) + && let Ok(stream_section) = self::section(data, hybrid_at) + { + for (number, slot) in stream_section.slots { + xref.note(number, slot); + } + xref.merge_trailer(&stream_section.trailer); + } + + for (number, slot) in section.slots { + xref.note(number, slot); + } + xref.merge_trailer(§ion.trailer); + next = section.previous; + } + + if xref.slots.is_empty() { + return Err(Error::Format( + "the cross-reference table names no objects".into(), + )); + } + Ok(xref) +} + +/// Rebuild the table by scanning for ` obj`. +/// +/// Later definitions win, which is the same rule the chain walk enforces from +/// the other end: an incremental update appends, so the last definition of an +/// object number in the file is the current one. +pub fn recover(data: &[u8]) -> Xref { + let mut xref = Xref::default(); + let mut at = 0usize; + + while let Some(found) = find(&data[at..], b"obj") { + let keyword = at + found; + at = keyword + 3; + // `obj` has to be a token of its own: `objstm` is not one. + if data.get(at).copied().is_some_and(is_regular) { + continue; + } + let Some((number, generation, start)) = header_before(data, keyword) else { + continue; + }; + xref.slots.insert( + number, + Slot::InFile { + offset: start as u64, + generation, + }, + ); + } + + // Every `trailer` in the file, oldest first, so the newest wins. + let mut at = 0usize; + while let Some(found) = find(&data[at..], b"trailer") { + let keyword = at + found; + at = keyword + b"trailer".len(); + let mut lexer = Lexer::at(data, at); + if let Ok(Object::Dictionary(dict)) = lexer.object() { + for (key, value) in dict.iter() { + if key.is("Prev") || key.is("XRefStm") { + continue; + } + xref.trailer.insert(key.clone(), value.clone()); + } + } + } + xref +} + +/// Walk back from an `obj` keyword over ` `, answering the two numbers +/// and where the indirect object begins. +fn header_before(data: &[u8], keyword: usize) -> Option<(u32, u16, usize)> { + let mut end = keyword; + let back = |end: &mut usize| -> Option<(usize, usize)> { + while *end > 0 && is_whitespace(data[*end - 1]) { + *end -= 1; + } + let stop = *end; + while *end > 0 && data[*end - 1].is_ascii_digit() { + *end -= 1; + } + (*end < stop).then_some((*end, stop)) + }; + + let (generation_start, generation_end) = back(&mut end)?; + let (number_start, number_end) = back(&mut end)?; + // The number must start a token: `x12 0 obj` is not an object header. + if number_start > 0 && is_regular(data[number_start - 1]) { + return None; + } + let number = std::str::from_utf8(&data[number_start..number_end]) + .ok()? + .parse::() + .ok()?; + let generation = std::str::from_utf8(&data[generation_start..generation_end]) + .ok()? + .parse::() + .ok()?; + Some((number, generation, number_start)) +} + +/// The offset of the last `startxref` value in the file. +fn startxref(data: &[u8]) -> Option { + let from = data.len().saturating_sub(TAIL); + let found = rfind(&data[from..], b"startxref")? + from; + let mut lexer = Lexer::at(data, found + b"startxref".len()); + lexer.object().ok()?.as_i64()?.try_into().ok() +} + +/// Turn a stored offset into a position in this file. +/// +/// A file with bytes before its `%PDF-` header stores offsets relative to the +/// header, and one whose prologue was added *after* it was written stores them +/// relative to the file. Both are common; both are tried, and the one that +/// lands on something that begins a cross-reference section wins. +fn locate(data: &[u8], offset: u64, shift: usize) -> Option { + let candidates = [ + usize::try_from(offset).ok()?.checked_add(shift)?, + usize::try_from(offset).ok()?, + ]; + for at in candidates { + if at >= data.len() { + continue; + } + let mut lexer = Lexer::at(data, at); + let rewind = lexer.pos(); + if lexer.keyword("xref") { + return Some(at); + } + lexer.seek(rewind); + if lexer.indirect().is_ok() { + return Some(at); + } + } + None +} + +/// One section of the chain, before it is merged into the whole. +struct Section { + slots: Vec<(u32, Slot)>, + trailer: Dictionary, + previous: Option, + /// `/XRefStm`, the cross-reference stream a hybrid file hides beside its + /// classic table. + hybrid: Option, +} + +fn section(data: &[u8], at: usize) -> Result
{ + let mut lexer = Lexer::at(data, at); + if lexer.keyword("xref") { + return classic(data, lexer); + } + lexer.seek(at); + let (_, object) = lexer.indirect()?; + let Some(stream) = object.as_stream() else { + return Err(Error::Format(format!( + "byte {at} begins neither an `xref` table nor a cross-reference stream" + ))); + }; + stream_section(stream) +} + +/// The plain-text table: subsection headers, then twenty-byte entries. +fn classic(data: &[u8], mut lexer: Lexer<'_>) -> Result
{ + let mut slots = Vec::new(); + loop { + let rewind = lexer.pos(); + if lexer.keyword("trailer") { + break; + } + lexer.seek(rewind); + let Ok(Some(first)) = lexer.object().map(|o| o.as_i64()) else { + lexer.seek(rewind); + break; + }; + let Ok(Some(count)) = lexer.object().map(|o| o.as_i64()) else { + return Err(Error::Format(format!( + "cross-reference subsection at byte {rewind} has no entry count" + ))); + }; + let Ok(first) = u32::try_from(first) else { + return Err(Error::Format(format!( + "cross-reference subsection at byte {rewind} starts at object {first}" + ))); + }; + + for index in 0..count.max(0) { + let offset = lexer.object()?.as_i64().unwrap_or(-1); + let generation = lexer.object()?.as_i64().unwrap_or(0); + lexer.skip_space(); + let kind = data.get(lexer.pos()).copied().unwrap_or(b'n'); + lexer.seek(lexer.pos() + 1); + + let Some(number) = first.checked_add(index as u32) else { + break; + }; + let slot = match kind { + b'f' => Slot::Free, + _ => match (u64::try_from(offset), u16::try_from(generation)) { + (Ok(offset), Ok(generation)) => Slot::InFile { offset, generation }, + // An entry a reader cannot use is a free one: it names no + // place in the file, and treating it as one would send the + // object loader to byte zero. + _ => Slot::Free, + }, + }; + slots.push((number, slot)); + } + } + + let trailer = match lexer.object()? { + Object::Dictionary(dict) => dict, + other => { + return Err(Error::Format(format!( + "expected a trailer dictionary, found {other:?}" + ))); + } + }; + Ok(Section { + previous: trailer + .get("Prev") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + hybrid: trailer + .get("XRefStm") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + slots, + trailer, + }) +} + +/// The binary table: `/W` gives each field's width in bytes, `/Index` which +/// object numbers the rows are for. +fn stream_section(stream: &crate::object::Stream) -> Result
{ + let dict = &stream.dict; + let bytes = match decode_directly(stream)? { + Decoded::Bytes(bytes) => bytes, + Decoded::Stopped { filter } => { + return Err(Error::Format(format!( + "cross-reference stream is behind {filter}, which is not a filter this tool decodes" + ))); + } + }; + + let widths: Vec = dict + .get("W") + .and_then(Object::as_array) + .ok_or_else(|| Error::Format("cross-reference stream has no /W".into()))? + .iter() + .map(|w| w.as_i64().unwrap_or(0).clamp(0, 8) as usize) + .collect(); + if widths.len() < 3 { + return Err(Error::Format(format!( + "cross-reference stream /W has {} fields, and the format has three", + widths.len() + ))); + } + let row = widths.iter().sum::(); + if row == 0 { + return Err(Error::Format( + "cross-reference stream /W gives every field a width of zero".into(), + )); + } + + // §7.5.8.2: /Index defaults to the whole of /Size. + let size = dict.get("Size").and_then(Object::as_i64).unwrap_or(0); + let index: Vec = match dict.get("Index").and_then(Object::as_array) { + Some(pairs) => pairs.iter().map(|p| p.as_i64().unwrap_or(0)).collect(), + None => vec![0, size], + }; + + let mut slots = Vec::new(); + let mut cursor = 0usize; + for pair in index.chunks(2) { + let (&first, &count) = match pair { + [first, count] => (first, count), + _ => break, + }; + for i in 0..count.max(0) { + if cursor + row > bytes.len() { + break; + } + let mut fields = [0u64; 3]; + let mut at = cursor; + for (field, &width) in fields.iter_mut().zip(&widths) { + for _ in 0..width { + *field = (*field << 8) | u64::from(bytes[at]); + at += 1; + } + } + // A zero-width first field means type 1, per §7.5.8.2. + if widths[0] == 0 { + fields[0] = 1; + } + cursor += row; + + let Ok(number) = u32::try_from(first + i) else { + continue; + }; + let slot = match fields[0] { + 0 => Slot::Free, + 1 => Slot::InFile { + offset: fields[1], + generation: u16::try_from(fields[2]).unwrap_or(0), + }, + 2 => match (u32::try_from(fields[1]), u32::try_from(fields[2])) { + (Ok(stream), Ok(index)) => Slot::InStream { stream, index }, + _ => continue, + }, + // §7.5.8.3: a type this reader does not know is to be treated + // as free rather than guessed at. + _ => Slot::Free, + }; + slots.push((number, slot)); + } + } + + Ok(Section { + previous: dict + .get("Prev") + .and_then(Object::as_i64) + .and_then(|p| u64::try_from(p).ok()), + hybrid: None, + slots, + trailer: dict.clone(), + }) +} + +/// Decode a stream whose dictionary cannot contain indirect references. +/// +/// A cross-reference stream is the one stream that has to be read before the +/// table exists, so §7.5.8.2 requires its `/Filter`, `/W` and `/Index` to be +/// direct. That is what makes this function possible, and why the general path +/// through [`crate::Pdf::decoded`] is not used here. +fn decode_directly(stream: &crate::object::Stream) -> Result { + let filters = match stream.dict.get("Filter") { + Some(Object::Name(name)) => vec![name.clone()], + Some(Object::Array(items)) => items.iter().filter_map(Object::as_name).cloned().collect(), + _ => Vec::::new(), + }; + let parms = match stream.dict.get("DecodeParms") { + Some(Object::Dictionary(dict)) => vec![dict.clone()], + Some(Object::Array(items)) => items + .iter() + .map(|item| item.as_dict().cloned().unwrap_or_default()) + .collect(), + _ => Vec::new(), + }; + filter::decode(&stream.data, &filters, &parms) +} diff --git a/crates/mirsam-pdf/tests/corpus.rs b/crates/mirsam-pdf/tests/corpus.rs new file mode 100644 index 0000000..0ec2803 --- /dev/null +++ b/crates/mirsam-pdf/tests/corpus.rs @@ -0,0 +1,197 @@ +//! The fixture corpus, one file per generator family. +//! +//! `scripts/make-pdf-fixture.py` writes these byte by byte; `make pdfs` +//! regenerates them. Each asserts the *whole* of what the object layer owes a +//! caller for that family — the pages come back, the content decodes, the +//! Arabic survives, and nothing is silently missing — because a test that only +//! checked the file opened would pass on a reader that returned an empty +//! document. + +use mirsam_pdf::{Pdf, Reconstruction}; +use std::path::PathBuf; + +fn fixture(name: &str) -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests/pdfs") + .join(name) +} + +fn open(name: &str) -> Pdf { + Pdf::open(&fixture(name)).unwrap_or_else(|e| panic!("{name}: {e}")) +} + +/// How the fixtures write the Arabic: one byte per character, as a hexadecimal +/// string. Fourteen codes for the fourteen characters of `\u{627}\u{644}\u{62a}\u{642}\u{631}\u{64a}\u{631} +/// \u{627}\u{644}\u{633}\u{646}\u{648}\u{64a}`, which is the shape a generator that subset its font +/// produces and the shape PLAN \u{a7}6.3 hunts a reversal in. +const ARABIC_RUN: &[u8] = b"<4142434445464748494A4B4C4D4E>"; + +fn content(pdf: &Pdf) -> Vec { + pdf.pages()[0].content(pdf) +} + +#[test] +fn classic_table_uncompressed() { + let pdf = open("classic.pdf"); + assert_eq!(pdf.version(), "1.4"); + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.reconstructed().is_empty(), "nothing needed rebuilding"); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + let content = content(&pdf); + assert!(content.starts_with(b"BT\n")); + assert!( + content.contains_str(ARABIC_RUN), + "the Arabic run is not in the content stream" + ); +} + +#[test] +fn cross_reference_stream_and_object_stream() { + let pdf = open("compressed.pdf"); + assert_eq!(pdf.version(), "1.7"); + assert!(pdf.reconstructed().is_empty(), "{:?}", pdf.reconstructed()); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + // The catalog, the page tree node, the page and the font all came out of + // the object stream: a reader with no inflate finds none of them. + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.catalog().contains_key("Pages")); + let content = content(&pdf); + assert!( + content.starts_with(b"BT\n"), + "{:?}", + &content[..8.min(content.len())] + ); + assert!(content.contains_str(ARABIC_RUN)); +} + +#[test] +fn hybrid_file_prefers_the_stream() { + let pdf = open("hybrid.pdf"); + assert_eq!(pdf.pages().len(), 1); + // Object 6 is the ToUnicode CMap, and only the /XRefStm says where it is. + // A reader that stopped at the classic table has five objects and a font + // that says nothing about what it draws. + assert!( + pdf.object(mirsam_pdf::ObjectId::new(6, 0)).is_some(), + "the object only the /XRefStm names was not found" + ); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); +} + +#[test] +fn an_incremental_update_supersedes_what_it_replaces() { + let base = open("classic.pdf"); + let updated = open("incremental.pdf"); + + let before = content(&base); + let after = content(&updated); + assert_ne!(before, after, "the update changed nothing"); + assert!( + after.len() > before.len(), + "the newer content stream is the longer one, and this is not it" + ); + assert!(updated.reconstructed().is_empty()); +} + +#[test] +fn a_stale_startxref_is_recovered_from() { + let pdf = open("damaged.pdf"); + assert_eq!( + pdf.reconstructed(), + [Reconstruction::CrossReferenceTable], + "the rebuild has to be said out loud" + ); + assert_eq!(pdf.pages().len(), 1); + assert_eq!(content(&pdf), content(&open("classic.pdf"))); +} + +#[test] +fn an_encrypted_document_is_refused_by_name() { + let error = Pdf::open(&fixture("encrypted.pdf")) + .unwrap_err() + .to_string(); + assert!(error.contains("AES-256"), "{error}"); + assert!(error.contains("/Standard security handler"), "{error}"); + assert!(error.contains("V5 R6"), "{error}"); + assert!( + error.contains("unread rather than clean"), + "the refusal has to say what it is not claiming: {error}" + ); +} + +#[test] +fn the_ascii_filters_a_distiller_writes() { + let pdf = open("ascii-filters.pdf"); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); + + // /Contents is an array of three streams behind three different filters, + // joined at their boundaries. + let content = content(&pdf); + assert!( + content.starts_with(b"BT\n"), + "ASCII85 then Flate did not unwind" + ); + assert!(content.contains_str(b"% run length\n"), "RunLengthDecode"); + assert!(content.contains_str(b"% lzw\n"), "LZWDecode"); + + // The ToUnicode CMap is behind ASCIIHexDecode. + let cmap = pdf + .decoded(mirsam_pdf::ObjectId::new(6, 0)) + .unwrap() + .unwrap(); + assert!(cmap.contains_str(b"beginbfchar")); +} + +#[test] +fn a_scanned_page_is_named_rather_than_called_clean() { + let pdf = open("image-only.pdf"); + assert_eq!(pdf.pages().len(), 1); + + // The page's own content stream reads perfectly well; it draws an image. + let content = content(&pdf); + assert!(content.contains_str(b"/Im1 Do")); + + // Asking for the image is where the honesty is: the bytes are not text + // this crate failed to read, they are text that is not in the file. + let image = pdf.decoded(mirsam_pdf::ObjectId::new(5, 0)).unwrap(); + assert!(image.is_none(), "an image codec must not come back decoded"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 1, "{unread:?}"); + assert!(unread[0].starts_with("5 0 R — /DCTDecode"), "{unread:?}"); +} + +#[test] +fn every_fixture_states_its_own_version_and_one_page() { + for name in [ + "classic.pdf", + "compressed.pdf", + "hybrid.pdf", + "incremental.pdf", + "damaged.pdf", + "ascii-filters.pdf", + "image-only.pdf", + ] { + let pdf = open(name); + assert!(pdf.version().starts_with("1."), "{name}: {}", pdf.version()); + assert_eq!(pdf.pages().len(), 1, "{name}"); + // MediaBox is stated on the /Pages node in every fixture, so a page + // that has one proves the inheritable attributes came down the tree. + assert!( + pdf.pages()[0].dict.contains_key("MediaBox"), + "{name}: the page did not inherit /MediaBox" + ); + } +} + +/// `slice::contains` is for one element; this is for a subslice. +trait ContainsStr { + fn contains_str(&self, needle: &[u8]) -> bool; +} + +impl ContainsStr for Vec { + fn contains_str(&self, needle: &[u8]) -> bool { + self.windows(needle.len()).any(|w| w == needle) + } +} diff --git a/crates/mirsam-pdf/tests/document.rs b/crates/mirsam-pdf/tests/document.rs new file mode 100644 index 0000000..351477d --- /dev/null +++ b/crates/mirsam-pdf/tests/document.rs @@ -0,0 +1,357 @@ +//! The object layer's own behaviours, on documents built for one question +//! each — the fixture corpus proves the generator families, and these prove +//! the rules the loader applies to all of them. + +use mirsam_pdf::object::{Name, Object, ObjectId}; +use mirsam_pdf::{Pdf, Reconstruction}; + +/// Assemble a PDF from object bodies, with a correct classic table. +/// +/// Written here rather than taken from the fixture script so that a test can +/// state one situation in six lines. Object 1 is the catalog by convention. +fn build(objects: &[(u32, &[u8])], trailer_extra: &str) -> Vec { + let mut out = b"%PDF-1.4\n".to_vec(); + let mut offsets = Vec::new(); + for (number, body) in objects { + offsets.push((*number, out.len())); + out.extend_from_slice(format!("{number} 0 obj\n").as_bytes()); + out.extend_from_slice(body); + out.extend_from_slice(b"\nendobj\n"); + } + let size = objects.iter().map(|(n, _)| *n).max().unwrap_or(0) + 1; + let startxref = out.len(); + out.extend_from_slice(format!("xref\n0 {size}\n0000000000 65535 f \n").as_bytes()); + for number in 1..size { + match offsets.iter().find(|(n, _)| *n == number) { + Some((_, at)) => out.extend_from_slice(format!("{at:010} 00000 n \n").as_bytes()), + None => out.extend_from_slice(b"0000000000 65535 f \n"), + } + } + out.extend_from_slice( + format!("trailer\n<< /Size {size} /Root 1 0 R{trailer_extra} >>\nstartxref\n{startxref}\n%%EOF\n") + .as_bytes(), + ); + out +} + +fn open(objects: &[(u32, &[u8])]) -> Pdf { + Pdf::from_bytes(&build(objects, ""), "test").expect("open") +} + +const CATALOG: &[u8] = b"<< /Type /Catalog /Pages 2 0 R >>"; + +fn rfind(haystack: &[u8], needle: &[u8]) -> Option { + haystack.windows(needle.len()).rposition(|w| w == needle) +} + +/// The cross-reference table's own `xref`, and not the one inside the +/// `startxref` that follows it. +fn table_at(data: &[u8]) -> usize { + rfind(data, b"\nxref\n").expect("the table") + 1 +} + +#[test] +fn a_file_with_no_header_is_not_a_pdf() { + let error = Pdf::from_bytes(b"just some bytes", "test") + .unwrap_err() + .to_string(); + assert!(error.contains("not a PDF"), "{error}"); +} + +/// A `%PDF-` header preceded by junk is common enough that every viewer +/// tolerates it, and the stored offsets may be relative to either origin. +#[test] +fn a_prologue_before_the_header_is_read_through() { + let mut data = b"#!/bin/sh\n# a self-extracting wrapper\n".to_vec(); + let body = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + data.extend_from_slice(&body); + let pdf = Pdf::from_bytes(&data, "wrapped").expect("open"); + assert_eq!(pdf.pages().len(), 1); + assert!(pdf.reconstructed().is_empty(), "{:?}", pdf.reconstructed()); +} + +/// §7.7.3.4's inheritable attributes come down the tree. `/Resources` is the +/// one that matters: a text layer reading it from the leaf alone would find no +/// fonts on the very common document that states them once at the root. +#[test] +fn a_page_inherits_resources_and_the_boxes() { + let pdf = open(&[ + (1, CATALOG), + ( + 2, + b"<< /Type /Pages /Kids [4 0 R] /Count 1 /MediaBox [0 0 595 842] \ + /Resources << /Font << /F1 9 0 R >> >> /Rotate 90 >>", + ), + (4, b"<< /Type /Pages /Kids [3 0 R] /Count 1 /Rotate 0 >>"), + (3, b"<< /Type /Page /Parent 4 0 R >>"), + ]); + let page = &pdf.pages()[0]; + assert!( + page.resources(&pdf).is_some(), + "the fonts did not come down" + ); + assert!(page.dict.contains_key("MediaBox")); + // The nearer ancestor's value wins over the further one's. + assert_eq!( + pdf.get(&page.dict, "Rotate").and_then(Object::as_i64), + Some(0) + ); +} + +/// A page's own value is never overwritten by what it inherits. +#[test] +fn a_page_outranks_what_it_inherits() { + let pdf = open(&[ + (1, CATALOG), + ( + 2, + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", + ), + ( + 3, + b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] >>", + ), + ]); + let media = pdf.get(&pdf.pages()[0].dict, "MediaBox").unwrap(); + assert_eq!(media.as_array().unwrap()[2].as_i64(), Some(100)); +} + +/// A `/Pages` node with no `/Type` is still an internal node: a great many +/// writers omit it, and reading the shape rather than the label is the only +/// way to walk those files. +#[test] +fn a_node_with_kids_is_an_internal_node_whatever_it_says() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ]); + assert_eq!(pdf.pages().len(), 1); +} + +/// `/Kids` pointing back up the tree is a file, not a document. It must be +/// read, not hung on. +#[test] +fn a_cycle_in_the_page_tree_terminates() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R 2 0 R] /Count 2 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ]); + assert_eq!(pdf.pages().len(), 1); +} + +/// So is a reference that points at itself. +#[test] +fn a_cycle_in_the_references_resolves_to_null() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R /Rotate 4 0 R >>"), + (4, b"5 0 R"), + (5, b"4 0 R"), + ]); + assert_eq!(pdf.get(&pdf.pages()[0].dict, "Rotate"), None); +} + +/// §7.3.9: a reference to an object the file does not have is null, and null +/// is indistinguishable from absent. +#[test] +fn a_dangling_reference_is_absent() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Rotate 99 0 R /CropBox null >>"), + ]); + let page = &pdf.pages()[0].dict; + assert_eq!(pdf.get(page, "Rotate"), None); + assert_eq!(pdf.get(page, "CropBox"), None); +} + +/// A trailer with no `/Root`, which a file assembled by hand often has. The +/// object that calls itself the catalog is taken instead. +#[test] +fn a_missing_root_is_found_by_type() { + let data = build( + &[ + (7, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + // /Root points at object 1, which does not exist. + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + assert!(pdf.catalog().contains_key("Pages")); + assert_eq!(pdf.pages().len(), 1); +} + +/// A document with a catalog nobody can find is unreadable, and says so. +#[test] +fn no_catalog_at_all_is_a_refusal() { + let data = build(&[(1, b"<< /Type /Page >>")], ""); + let error = Pdf::from_bytes(&data, "test").unwrap_err().to_string(); + assert!(error.contains("no document catalog"), "{error}"); +} + +/// An object at an offset that is not there is recovered from the body if it +/// is there at all — and only reported when it genuinely is not. +#[test] +fn an_object_the_table_misplaces_is_found_anyway() { + let mut data = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + ], + "", + ); + // Break the entry for object 3 without touching the object itself. + let table = table_at(&data); + let entry = table + b"xref\n0 4\n".len() + 20 * 3; + data[entry..entry + 10].copy_from_slice(b"0000009999"); + + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + assert_eq!(pdf.pages().len(), 1); + assert_eq!(pdf.reconstructed(), [Reconstruction::CrossReferenceTable]); + assert!(pdf.unread().is_empty(), "{:?}", pdf.unread()); +} + +/// And when it really is not there, it is named — with the offset the table +/// claimed, so a reviewer can go and look. +#[test] +fn an_object_that_is_not_in_the_file_is_named() { + let mut data = build( + &[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Parent 2 0 R >>"), + (4, b"<< /Type /Nothing >>"), + ], + "", + ); + // Delete object 4's body, leaving the table naming it. + let at = rfind(&data, b"4 0 obj").expect("object 4"); + let end = table_at(&data); + data.splice(at..end, std::iter::repeat_n(b' ', end - at)); + + let pdf = Pdf::from_bytes(&data, "test").expect("open"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 1, "{unread:?}"); + assert!(unread[0].starts_with("4 0 R —"), "{unread:?}"); + assert!(unread[0].contains("is not there"), "{unread:?}"); +} + +/// A dictionary key repeated in the file resolves to one entry, and the +/// document reads through it the same way. +#[test] +fn resolution_follows_a_chain_of_references() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page /Rotate 4 0 R >>"), + (4, b"5 0 R"), + (5, b"180"), + ]); + assert_eq!( + pdf.get(&pdf.pages()[0].dict, "Rotate") + .and_then(Object::as_i64), + Some(180) + ); +} + +#[test] +fn objects_are_matched_on_the_number_alone() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page >>"), + ]); + // A generation the table disagrees with is something writers produce and + // every reader ignores; refusing to follow it would lose the object. + assert!(pdf.object(ObjectId::new(3, 7)).is_some()); +} + +#[test] +fn a_document_reports_what_it_reconstructed_once() { + let pdf = Pdf::open( + &std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs/damaged.pdf"), + ) + .expect("open"); + assert_eq!(pdf.reconstructed(), [Reconstruction::CrossReferenceTable]); + assert_eq!( + Reconstruction::CrossReferenceTable.to_string(), + "the cross-reference table" + ); +} + +/// The text layer above this one has sources it cannot read too — a font with +/// no `ToUnicode` map is one — and they have to reach the same list. +#[test] +fn the_unread_list_is_open_to_the_layer_above() { + let pdf = open(&[ + (1, CATALOG), + (2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, b"<< /Type /Page >>"), + ]); + pdf.note_unread( + Some(ObjectId::new(9, 0)), + "no /ToUnicode and no standard encoding", + ); + pdf.note_unread(None, "page 1 draws its Arabic as an image"); + let unread = pdf.unread(); + assert_eq!(unread.len(), 2); + assert!( + unread + .iter() + .any(|u| u == "9 0 R — no /ToUnicode and no standard encoding") + ); + assert!( + unread + .iter() + .any(|u| u == "page 1 draws its Arabic as an image") + ); +} + +#[test] +fn a_name_answers_for_itself_and_nothing_else() { + let name = Name::from("FlateDecode"); + assert!(name.is("FlateDecode")); + assert!(!name.is("Flate")); + assert_eq!(name.as_str(), Some("FlateDecode")); +} + +/// mirsam is run on files people were sent. Every prefix and every +/// single-byte corruption of the fixture corpus has to come back as a document +/// or as an error, and never as a panic — the bounds in this crate are what +/// make that true, and this is what asserts they are all still there. +#[test] +fn no_prefix_or_corruption_of_the_corpus_panics() { + let dir = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs"); + let mut checked = 0usize; + for entry in std::fs::read_dir(&dir).expect("fixtures") { + let path = entry.expect("entry").path(); + if path.extension().is_none_or(|e| e != "pdf") { + continue; + } + let data = std::fs::read(&path).expect("read"); + checked += 1; + + for cut in (0..data.len()).step_by(7) { + let _ = Pdf::from_bytes(&data[..cut], "truncated"); + } + for at in (0..data.len()).step_by(13) { + let mut corrupted = data.clone(); + corrupted[at] = corrupted[at].wrapping_add(0x5b); + let _ = Pdf::from_bytes(&corrupted, "corrupted"); + } + } + assert!(checked >= 8, "the corpus lost fixtures: {checked}"); +} diff --git a/crates/mirsam-pdf/tests/encrypt.rs b/crates/mirsam-pdf/tests/encrypt.rs new file mode 100644 index 0000000..0845e2e --- /dev/null +++ b/crates/mirsam-pdf/tests/encrypt.rs @@ -0,0 +1,97 @@ +//! The refusal, and the sentence it gives. +//! +//! mirsam does not decrypt. What is being tested is that it says *what* it did +//! not decrypt: an encrypted document produces no text, and a reader that +//! stopped there would report a file with no defects. + +use mirsam_pdf::Pdf; +use mirsam_pdf::encrypt::describe; +use mirsam_pdf::lexer::Lexer; + +fn dict(source: &str) -> mirsam_pdf::Dictionary { + Lexer::new(source.as_bytes()) + .object() + .expect("parse") + .as_dict() + .expect("dictionary") + .clone() +} + +#[test] +fn the_algorithms_a_standard_handler_states() { + assert_eq!( + describe(&dict("<< /Filter /Standard /V 1 /R 2 >>")), + "RC4 40-bit (/Standard security handler, V1 R2)" + ); + assert_eq!( + describe(&dict("<< /Filter /Standard /V 2 /R 3 /Length 128 >>")), + "RC4 128-bit (/Standard security handler, V2 R3)" + ); + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 4 /R 4 /CF << /StdCF << /CFM /AESV2 >> >> /StmF /StdCF >>" + )), + "AES-128 (/Standard security handler, V4 R4)" + ); + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 5 /R 6 /CF << /StdCF << /CFM /AESV3 >> >> /StmF /StdCF >>" + )), + "AES-256 (/Standard security handler, V5 R6)" + ); +} + +/// `/StmF` names the filter streams actually use. A file whose stream filter +/// is not the one conventionally called `StdCF` would otherwise be described +/// by a dictionary entry nothing in it uses. +#[test] +fn the_filter_streams_use_is_the_one_reported() { + assert_eq!( + describe(&dict( + "<< /Filter /Standard /V 4 /R 4 /StmF /Custom \ + /CF << /StdCF << /CFM /AESV2 >> /Custom << /CFM /V2 /Length 16 >> >> >>" + )), + "RC4 128-bit (/Standard security handler, V4 R4)" + ); +} + +/// A third-party handler is named as itself. Claiming an algorithm it never +/// stated would be exactly the inference standing rule 4 forbids. +#[test] +fn a_handler_this_tool_does_not_know_is_named_rather_than_guessed_at() { + let answer = describe(&dict("<< /Filter /FOPN_foweb /V 4 /R 4 >>")); + assert!(answer.contains("/FOPN_foweb security handler"), "{answer}"); +} + +#[test] +fn an_encryption_that_states_nothing_says_so() { + let answer = describe(&dict("<< >>")); + assert_eq!( + answer, + "an undeclared algorithm (/Standard security handler)" + ); +} + +/// The `/Encrypt` dictionary is refused whether the trailer holds it inline or +/// points at it — §7.6.1 keeps it unencrypted precisely so a reader can find +/// out what it is up against. +#[test] +fn an_inline_encrypt_dictionary_is_refused_too() { + let mut out = b"%PDF-1.4\n".to_vec(); + let object = out.len(); + out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let table = out.len(); + out.extend_from_slice(b"xref\n0 2\n0000000000 65535 f \n"); + out.extend_from_slice(format!("{object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!( + "trailer\n<< /Size 2 /Root 1 0 R /Encrypt << /Filter /Standard /V 2 /R 3 \ + /Length 128 >> >>\nstartxref\n{table}\n%%EOF\n" + ) + .as_bytes(), + ); + + let error = Pdf::from_bytes(&out, "test").unwrap_err().to_string(); + assert!(error.contains("RC4 128-bit"), "{error}"); + assert!(error.contains("mirsam does not decrypt"), "{error}"); +} diff --git a/crates/mirsam-pdf/tests/filter.rs b/crates/mirsam-pdf/tests/filter.rs new file mode 100644 index 0000000..11eafda --- /dev/null +++ b/crates/mirsam-pdf/tests/filter.rs @@ -0,0 +1,189 @@ +//! Stream filters: §7.4, and the line between a filter that was not applied +//! and one this tool will never apply. + +use mirsam_pdf::filter::{Decoded, decode}; +use mirsam_pdf::object::{Dictionary, Name, Object}; + +fn bytes(data: &[u8], filters: &[&str], parms: &[Dictionary]) -> Vec { + let filters: Vec = filters.iter().copied().map(Name::from).collect(); + match decode(data, &filters, parms).expect("decode") { + Decoded::Bytes(bytes) => bytes, + Decoded::Stopped { filter } => panic!("stopped at {filter}"), + } +} + +fn stopped(data: &[u8], filters: &[&str]) -> Name { + let filters: Vec = filters.iter().copied().map(Name::from).collect(); + match decode(data, &filters, &[]).expect("decode") { + Decoded::Bytes(_) => panic!("expected to stop"), + Decoded::Stopped { filter } => filter, + } +} + +fn parms(entries: &[(&str, i64)]) -> Dictionary { + entries + .iter() + .map(|(key, value)| (Name::from(*key), Object::Integer(*value))) + .collect() +} + +fn deflate(data: &[u8]) -> Vec { + use flate2::Compression; + use flate2::write::ZlibEncoder; + use std::io::Write; + let mut encoder = ZlibEncoder::new(Vec::new(), Compression::best()); + encoder.write_all(data).unwrap(); + encoder.finish().unwrap() +} + +#[test] +fn no_filter_is_the_bytes_themselves() { + assert_eq!(bytes(b"plain", &[], &[]), b"plain"); +} + +#[test] +fn flate() { + let source = b"the annual report, at some length, so it actually compresses"; + assert_eq!(bytes(&deflate(source), &["FlateDecode"], &[]), source); + // Abbreviated names are §7.4's own, for inline images. + assert_eq!(bytes(&deflate(source), &["Fl"], &[]), source); +} + +/// A truncated stream keeps what it had. Discarding it would turn a document +/// with a damaged tail into a document with no text, which is the failure this +/// project treats as worse than a miss. +#[test] +fn a_truncated_flate_stream_keeps_what_decoded() { + let source = b"the annual report, at some length, so it actually compresses"; + let mut compressed = deflate(source); + compressed.truncate(compressed.len() - 4); + let out = bytes(&compressed, &["FlateDecode"], &[]); + assert!(!out.is_empty(), "a truncated stream decoded to nothing"); + assert!(source.starts_with(&out[..])); +} + +#[test] +fn ascii_hex() { + assert_eq!(bytes(b"48656C6C6F>", &["ASCIIHexDecode"], &[]), b"Hello"); + // White space between digits is ignored, and a missing final digit is a + // zero. + assert_eq!(bytes(b"48 65\n6C 6C 6F 9>", &["AHx"], &[]), b"Hello\x90"); +} + +#[test] +fn ascii85() { + assert_eq!( + bytes(b"87cURD]i,\"Ebo80~>", &["ASCII85Decode"], &[]), + b"Hello World!" + ); + // `z` is four zero bytes, and only at a group boundary. + assert_eq!(bytes(b"z~>", &["A85"], &[]), [0, 0, 0, 0]); + // The `<~` some writers open with. + assert_eq!(bytes(b"<~87cURDZ~>", &["ASCII85Decode"], &[]), b"Hello"); +} + +#[test] +fn run_length() { + // A length byte under 128 introduces that many literal bytes plus one; + // over 128 repeats the next byte 257 - length times; 128 ends the data. + let encoded = [4u8, b'H', b'e', b'l', b'l', b'o', 253, b'!', 128, b'?']; + assert_eq!(bytes(&encoded, &["RunLengthDecode"], &[]), b"Hello!!!!"); +} + +#[test] +fn lzw() { + // §7.4.4.2's own example: the codes for -----A---B, packed nine bits wide. + let encoded = [0x80u8, 0x0B, 0x60, 0x50, 0x22, 0x0C, 0x0C, 0x85, 0x01]; + assert_eq!(bytes(&encoded, &["LZWDecode"], &[]), b"-----A---B"); +} + +/// The PNG predictors, which a cross-reference stream always carries and +/// without which the table decodes to noise that looks like data. +#[test] +fn png_predictors() { + // Three rows of four bytes, each prefixed with its filter type: none, sub + // (each byte is a delta from its left neighbour) and up (from the row + // above). + let encoded = [ + 0u8, 1, 2, 3, 4, // 1 2 3 4 + 1, 1, 1, 1, 1, // 1 2 3 4 again, as deltas + 2, 1, 1, 1, 1, // 2 3 4 5 + ]; + let out = mirsam_pdf::filter::decode( + &deflate(&encoded), + &[Name::from("FlateDecode")], + &[parms(&[("Predictor", 12), ("Columns", 4)])], + ) + .expect("decode"); + let Decoded::Bytes(out) = out else { + panic!("stopped") + }; + assert_eq!(out, [1, 2, 3, 4, 1, 2, 3, 4, 2, 3, 4, 5]); +} + +#[test] +fn the_tiff_predictor() { + let encoded = deflate(&[1u8, 1, 1, 1, 5, 1, 1, 1]); + let out = mirsam_pdf::filter::decode( + &encoded, + &[Name::from("FlateDecode")], + &[parms(&[ + ("Predictor", 2), + ("Columns", 4), + ("BitsPerComponent", 8), + ])], + ) + .expect("decode"); + let Decoded::Bytes(out) = out else { + panic!("stopped") + }; + assert_eq!(out, [1, 2, 3, 4, 5, 6, 7, 8]); +} + +#[test] +fn a_chain_unwinds_in_order() { + let source = b"the annual report"; + let mut ascii85 = Vec::new(); + for chunk in deflate(source).chunks(4) { + let mut value = 0u32; + for i in 0..4 { + value = value * 256 + u32::from(chunk.get(i).copied().unwrap_or(0)); + } + let mut digits = [0u8; 5]; + for slot in digits.iter_mut().rev() { + *slot = 33 + (value % 85) as u8; + value /= 85; + } + ascii85.extend_from_slice(&digits[..chunk.len() + 1]); + } + ascii85.extend_from_slice(b"~>"); + assert_eq!( + bytes(&ascii85, &["ASCII85Decode", "FlateDecode"], &[]), + source + ); +} + +/// An image codec is named rather than decoded. The bytes behind one are not +/// text this crate failed to read; they are text that is not in the file, and +/// ADR 0009 says the difference has to reach the report. +#[test] +fn an_image_codec_stops_the_chain_by_name() { + for codec in ["DCTDecode", "JPXDecode", "CCITTFaxDecode", "JBIG2Decode"] { + assert!(stopped(b"\xff\xd8\xff", &[codec]).is(codec)); + } +} + +/// So does a filter this crate has simply never met, rather than an error that +/// would lose the rest of the document. +#[test] +fn an_unknown_filter_stops_the_chain_too() { + assert!(stopped(b"x", &["SomeVendorDecode"]).is("SomeVendorDecode")); +} + +/// `/Crypt` reaching this module means an encrypted document was not refused +/// at open, which is a bug in this crate rather than a fact about the file — +/// so it stops, and the name says which. +#[test] +fn crypt_stops_the_chain() { + assert!(stopped(b"x", &["Crypt"]).is("Crypt")); +} diff --git a/crates/mirsam-pdf/tests/lexer.rs b/crates/mirsam-pdf/tests/lexer.rs new file mode 100644 index 0000000..4dce385 --- /dev/null +++ b/crates/mirsam-pdf/tests/lexer.rs @@ -0,0 +1,202 @@ +//! Object syntax: §7.2 and §7.3, and the malformations every reader tolerates. + +use mirsam_pdf::lexer::Lexer; +use mirsam_pdf::object::{Name, Object, ObjectId}; + +fn parse(source: &str) -> Object { + Lexer::new(source.as_bytes()) + .object() + .unwrap_or_else(|e| panic!("{source:?}: {e}")) +} + +fn parse_bytes(source: &[u8]) -> Object { + Lexer::new(source).object().expect("parse") +} + +#[test] +fn the_atoms() { + assert_eq!(parse("null"), Object::Null); + assert_eq!(parse("true"), Object::Boolean(true)); + assert_eq!(parse("false"), Object::Boolean(false)); + assert_eq!(parse("42"), Object::Integer(42)); + assert_eq!(parse("-17"), Object::Integer(-17)); + assert_eq!(parse("+3"), Object::Integer(3)); + assert_eq!(parse("2.75"), Object::Real(2.75)); + assert_eq!(parse("-.002"), Object::Real(-0.002)); + // §7.3.3 spells this one out: a trailing point is legal. + assert_eq!(parse("4."), Object::Real(4.0)); +} + +/// Numbers a writer should not have produced and every reader accepts. A file +/// refused over its punctuation is a file this tool reported unreadable that +/// a viewer opens. +#[test] +fn numbers_no_writer_should_have_written() { + assert_eq!(parse("--5"), Object::Integer(0)); + assert_eq!(parse("6.-2"), Object::Real(6.0)); + assert_eq!(parse("."), Object::Integer(0)); +} + +#[test] +fn names_carry_their_escapes() { + assert_eq!(parse("/Type"), Object::Name(Name::from("Type"))); + // §7.3.5's own example: /A#42 is /AB, and a dictionary lookup depends on + // the two being one name. + assert_eq!(parse("/A#42"), Object::Name(Name::from("AB"))); + assert_eq!( + parse("/Lime#20Green"), + Object::Name(Name::from("Lime Green")) + ); + assert_eq!(parse("/"), Object::Name(Name::from(""))); + // A `#` a writer forgot to escape stays a `#` rather than failing the file. + assert_eq!(parse("/a#zz"), Object::Name(Name::from("a#zz"))); +} + +#[test] +fn a_name_prints_the_way_the_file_writes_it() { + assert_eq!(Name::from("Type").to_string(), "/Type"); + assert_eq!(Name::from("Lime Green").to_string(), "/Lime#20Green"); + assert_eq!(Name::from("A#B").to_string(), "/A#23B"); +} + +#[test] +fn literal_strings() { + assert_eq!(parse("(hello)").as_string(), Some(&b"hello"[..])); + // Balanced inner parentheses need no escaping, per §7.3.4.2. + assert_eq!(parse("(a (b) c)").as_string(), Some(&b"a (b) c"[..])); + assert_eq!(parse(r"(a\)b)").as_string(), Some(&b"a)b"[..])); + assert_eq!(parse(r"(a\nb)").as_string(), Some(&b"a\nb"[..])); + assert_eq!(parse(r"(\101)").as_string(), Some(&b"A"[..])); + // Octal beyond a byte drops the high bits, which the specification says + // in as many words. + assert_eq!(parse(r"(\400)").as_string(), Some(&b"\x00"[..])); + // A backslash before an end-of-line joins the lines. + assert_eq!(parse("(a\\\nb)").as_string(), Some(&b"ab"[..])); + // "\q" is q. + assert_eq!(parse(r"(\q)").as_string(), Some(&b"q"[..])); +} + +#[test] +fn hex_strings_pad_an_odd_final_digit() { + assert_eq!(parse("<48656C6C6F>").as_string(), Some(&b"Hello"[..])); + assert_eq!(parse("<48 65 6c>").as_string(), Some(&b"Hel"[..])); + // §7.3.4.3: a missing final digit is a zero. + assert_eq!(parse("<9>").as_string(), Some(&[0x90][..])); + assert_eq!(parse("<>").as_string(), Some(&[][..])); +} + +#[test] +fn arrays_and_dictionaries() { + let array = parse("[1 (two) /three [4]]"); + let items = array.as_array().expect("array"); + assert_eq!(items.len(), 4); + assert_eq!(items[0], Object::Integer(1)); + + let dict = parse("<< /Type /Page /Count 3 /Kids [1 0 R] >>"); + let dict = dict.as_dict().expect("dictionary"); + assert!(dict.get("Type").unwrap().as_name().unwrap().is("Page")); + assert_eq!(dict.get("Count").unwrap().as_i64(), Some(3)); + assert_eq!( + dict.get("Kids").unwrap().as_array().unwrap()[0], + Object::Reference(ObjectId::new(1, 0)) + ); +} + +/// A repeated key keeps the last value: the only way one arises in practice is +/// a writer appending an entry it meant to supersede an earlier one. +#[test] +fn a_repeated_key_keeps_the_last_value() { + let dict = parse("<< /N 1 /N 2 >>"); + let dict = dict.as_dict().expect("dictionary"); + assert_eq!(dict.len(), 1); + assert_eq!(dict.get("N").unwrap().as_i64(), Some(2)); +} + +#[test] +fn a_reference_is_two_integers_and_an_r() { + assert_eq!(parse("12 0 R"), Object::Reference(ObjectId::new(12, 0))); + // Without the R they are two numbers, and the lookahead has to rewind far + // enough to give both of them back. + assert_eq!(parse("[12 0]").as_array().unwrap().len(), 2); + assert_eq!(parse("[12 0 (R)]").as_array().unwrap().len(), 3); + // And a reference in the middle of an array is one item, not three. + let array = parse("[1 2 R 3]"); + let items = array.as_array().unwrap(); + assert_eq!(items.len(), 2); + assert_eq!(items[0], Object::Reference(ObjectId::new(1, 2))); + assert_eq!(items[1], Object::Integer(3)); +} + +#[test] +fn comments_are_white_space() { + assert_eq!(parse("% a comment\n42"), Object::Integer(42)); + let dict = parse("<< /A 1 % why\n/B 2 >>"); + assert_eq!(dict.as_dict().unwrap().len(), 2); +} + +#[test] +fn a_stream_takes_the_length_it_states() { + let source = b"<< /Length 5 >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// `/Length` may be an indirect reference, which the lexer cannot resolve: the +/// cross-reference table is itself parsed out of streams. The data runs to +/// `endstream` instead, and no caller has to hand over a resolver it has not +/// built yet. +#[test] +fn an_indirect_length_falls_back_to_endstream() { + let source = b"<< /Length 9 0 R >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// And a direct length that is simply wrong, which real writers produce. +#[test] +fn a_wrong_length_is_not_believed() { + let source = b"<< /Length 2 >>\nstream\nHELLO\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO"); +} + +/// A stream may end in a newline of its own, and only the one belonging to the +/// syntax is dropped. +#[test] +fn only_the_syntactic_newline_is_taken_off() { + let source = b"<< /Length 6 >>\nstream\nHELLO\n\nendstream"; + let stream = parse_bytes(source); + assert_eq!(stream.as_stream().expect("stream").data, b"HELLO\n"); +} + +#[test] +fn an_indirect_object() { + let mut lexer = Lexer::new(b"7 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let (id, object) = lexer.indirect().expect("indirect object"); + assert_eq!(id, ObjectId::new(7, 0)); + assert!(object.is_type("Catalog")); +} + +/// `endobj` is optional in practice: a writer that omits it produces a file +/// every reader opens, and the object is complete without it. +#[test] +fn endobj_is_not_required() { + let mut lexer = Lexer::new(b"7 0 obj\n<< /Type /Catalog >>\n8 0 obj\nnull\nendobj\n"); + assert_eq!(lexer.indirect().expect("first").0, ObjectId::new(7, 0)); + assert_eq!(lexer.indirect().expect("second").0, ObjectId::new(8, 0)); +} + +#[test] +fn a_parse_failure_says_where() { + let error = Lexer::new(b"<< /A 1").object().unwrap_err().to_string(); + assert!(error.contains("at byte 0"), "{error}"); + assert!(error.contains("no closing `>>`"), "{error}"); +} + +/// Depth is bounded: a hand-written file of ten thousand open brackets is not +/// a document, and must not be a stack overflow either. +#[test] +fn nesting_is_bounded() { + let source = "[".repeat(10_000); + assert!(Lexer::new(source.as_bytes()).object().is_err()); +} diff --git a/crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf b/crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf new file mode 100644 index 0000000..8bc1931 Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/ascii-filters.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/classic.pdf b/crates/mirsam-pdf/tests/pdfs/classic.pdf new file mode 100644 index 0000000..bc015ac Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/classic.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/compressed.pdf b/crates/mirsam-pdf/tests/pdfs/compressed.pdf new file mode 100644 index 0000000..819b65c Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/compressed.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/damaged.pdf b/crates/mirsam-pdf/tests/pdfs/damaged.pdf new file mode 100644 index 0000000..bc61eed Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/damaged.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/encrypted.pdf b/crates/mirsam-pdf/tests/pdfs/encrypted.pdf new file mode 100644 index 0000000..c05c6cb Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/encrypted.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/hybrid.pdf b/crates/mirsam-pdf/tests/pdfs/hybrid.pdf new file mode 100644 index 0000000..3c4154a Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/hybrid.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/image-only.pdf b/crates/mirsam-pdf/tests/pdfs/image-only.pdf new file mode 100644 index 0000000..63451db Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/image-only.pdf differ diff --git a/crates/mirsam-pdf/tests/pdfs/incremental.pdf b/crates/mirsam-pdf/tests/pdfs/incremental.pdf new file mode 100644 index 0000000..7b0228b Binary files /dev/null and b/crates/mirsam-pdf/tests/pdfs/incremental.pdf differ diff --git a/crates/mirsam-pdf/tests/xref.rs b/crates/mirsam-pdf/tests/xref.rs new file mode 100644 index 0000000..13916ac --- /dev/null +++ b/crates/mirsam-pdf/tests/xref.rs @@ -0,0 +1,131 @@ +//! The cross-reference chain: newest wins, and a malformed chain terminates. + +use mirsam_pdf::xref::{Slot, read, recover}; + +/// A document with two sections, the newer one redefining object 3. +fn incremental(previous_marks_free: bool) -> Vec { + let mut out = b"%PDF-1.4\n".to_vec(); + let first_object = out.len(); + out.extend_from_slice(b"3 0 obj\n(old)\nendobj\n"); + let first_table = out.len(); + out.extend_from_slice(b"xref\n0 4\n0000000000 65535 f \n0000000000 65535 f \n"); + out.extend_from_slice(b"0000000000 65535 f \n"); + out.extend_from_slice(format!("{first_object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!("trailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{first_table}\n%%EOF\n").as_bytes(), + ); + + let second_object = out.len(); + out.extend_from_slice(b"3 0 obj\n(new)\nendobj\n"); + let second_table = out.len(); + let entry = if previous_marks_free { + "0000000000 65535 f \n".to_string() + } else { + format!("{second_object:010} 00000 n \n") + }; + out.extend_from_slice(format!("xref\n3 1\n{entry}").as_bytes()); + out.extend_from_slice( + format!( + "trailer\n<< /Size 4 /Root 1 0 R /Prev {first_table} >>\nstartxref\n{second_table}\n%%EOF\n" + ) + .as_bytes(), + ); + out +} + +#[test] +fn the_newer_section_wins() { + let data = incremental(false); + let xref = read(&data, 0).expect("read"); + let Some(Slot::InFile { offset, .. }) = xref.slots.get(&3) else { + panic!("object 3 is not in the table") + }; + assert_eq!( + &data[*offset as usize..*offset as usize + 13], + b"3 0 obj\n(new)" + ); +} + +/// A free entry in a newer section is what *hides* an object an older one +/// still names. Reading it as "no entry here, look further back" would +/// resurrect a deleted object. +#[test] +fn a_free_entry_in_a_newer_section_hides_the_older_one() { + let data = incremental(true); + let xref = read(&data, 0).expect("read"); + assert_eq!(xref.slots.get(&3), Some(&Slot::Free)); +} + +/// A `/Prev` pointing at the section that carried it. Two writers producing +/// the same offset is a loop, not a longer history. +#[test] +fn a_prev_loop_terminates() { + let mut out = b"%PDF-1.4\n".to_vec(); + let object = out.len(); + out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog >>\nendobj\n"); + let table = out.len(); + out.extend_from_slice(b"xref\n0 2\n0000000000 65535 f \n"); + out.extend_from_slice(format!("{object:010} 00000 n \n").as_bytes()); + out.extend_from_slice( + format!("trailer\n<< /Size 2 /Root 1 0 R /Prev {table} >>\nstartxref\n{table}\n%%EOF\n") + .as_bytes(), + ); + let xref = read(&out, 0).expect("read"); + assert_eq!(xref.slots.len(), 2); +} + +/// A file with no `startxref` at all is not readable through the chain, and +/// says so rather than answering with an empty table. +#[test] +fn no_startxref_is_an_error() { + let error = read(b"%PDF-1.4\n1 0 obj\nnull\nendobj\n", 0) + .unwrap_err() + .to_string(); + assert!(error.contains("no `startxref`"), "{error}"); +} + +/// The scan takes the *last* definition of an object number, which is the same +/// rule the chain walk enforces from the other end: an update appends. +#[test] +fn recovery_takes_the_last_definition() { + let data = incremental(false); + let xref = recover(&data); + let Some(Slot::InFile { offset, .. }) = xref.slots.get(&3) else { + panic!("object 3 was not found") + }; + assert_eq!( + &data[*offset as usize..*offset as usize + 13], + b"3 0 obj\n(new)" + ); + // And the newest trailer, so a /Root moved by an update is the one used. + assert!(xref.trailer.contains_key("Root")); +} + +/// `objstm` is not `obj`. A scan that matched inside a longer token would +/// invent objects out of the middle of dictionaries. +#[test] +fn recovery_does_not_match_inside_a_longer_token() { + let data = b"%PDF-1.4\n1 0 objstm\n2 0 obj\nnull\nendobj\n"; + let xref = recover(data); + assert_eq!(xref.slots.keys().copied().collect::>(), [2]); +} + +/// The cross-reference streams in the fixture corpus, read through the same +/// entry point: a compressed object comes back as one, and the type-2 entry +/// names the stream that holds it. +#[test] +fn a_cross_reference_stream_names_object_streams() { + let path = + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/pdfs/compressed.pdf"); + let data = std::fs::read(path).expect("fixture"); + let xref = read(&data, 0).expect("read"); + assert_eq!( + xref.slots.get(&1), + Some(&Slot::InStream { + stream: 8, + index: 0 + }) + ); + assert!(matches!(xref.slots.get(&4), Some(Slot::InFile { .. }))); + assert!(xref.trailer.contains_key("Root")); +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 0c241f8..542daa6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -240,6 +240,43 @@ crates/ origin a browser gives it, and dir="auto" is Unset because "the renderer picks" is what auto asks for. DocumentReader only + mirsam-pdf/ adapter — the object layer of a format that is not a + package, and the first this project reads without being + able to write it. No DocumentWriter here or planned: a + broken Arabic PDF is rebuilt from its source. See + docs/adr/0010 for why the parser is written rather than + taken, and what is taken instead + object.rs PDF's eight types and two composites. A name is bytes, + because it is the one place a file stores a key and a + lossy decode would merge two entries the document kept + apart + lexer.rs bytes to objects, and the malformations every reader + tolerates: a wrong /Length, an unescaped `#`, `--5`. A + stream's length is verified against `endstream` rather + than believed, which covers indirect, absent and simply + wrong with one rule + filter.rs the general-purpose filters — Flate, LZW, the two ASCII + encodings, run-length, and the PNG and TIFF predictors a + cross-reference stream needs. The four image codecs are + *named* and never decoded: a stream behind DCTDecode is + not text this crate failed to read, it is text that is + not in the file + xref.rs both spellings of the cross-reference, the /Prev chain + newest-first, and the hybrid file where the stream + outranks the classic table beside it. Plus the scan that + rebuilds a table whose offsets have gone stale, because a + document reported unreadable while a viewer opens it is + the wrong half of standing rule 4 + encrypt.rs what the /Encrypt dictionary says, said precisely, so the + refusal names the algorithm. Nothing decrypts + document.rs the graph: objects loaded once, object streams expanded, + references resolved with a bound, and the list of what + could not be read — which is ADR 0009's port, reaching the + format that needs it most + page.rs the page tree and the four attributes §7.7.3.4 inherits + down it. /Resources is the one that matters: a text layer + reading it from the leaf alone finds no fonts on the very + common document that states them once at the root mirsam-conformance/ no library code at all: the crate exists so one suite can depend on every adapter at once. Living inside mirsam-ooxml, it would have made that crate depend on its diff --git a/docs/PLAN.md b/docs/PLAN.md index 4d56d4e..9cb230e 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -1621,7 +1621,7 @@ express — listed in `repairs.skipped`, never claimed. --- -## M6 — PDF `[ ]` +## M6 — PDF `[~]` PDF implements `DocumentReader` **only**. @@ -1637,27 +1637,104 @@ they should fire here without a line of `mirsam-core` changing. If either needs one, the abstraction was wrong and §3.5's instruction applies: fix the abstraction, not the rule. -### 6.1 The object layer `[ ]` +### 6.1 The object layer `[x]` `mirsam-pdf`: cross-reference table *and* cross-reference stream, object streams, `FlateDecode`, the page tree, the trailer. The package layer of a format that is not a package. -- [ ] **Decide the parser: own it or take one. This wants an ADR, not a +- [x] **Decide the parser: own it or take one. This wants an ADR, not a `Cargo.toml` line.** The constraints are pure Rust, no C, no network, a permissive licence, and no rasteriser — mirsam needs the object graph and the content streams, and most of what a PDF library carries is the rendering half this project has said it will never do. The precedent cuts both ways: `dom.rs` and `css.rs` were written here rather than taken, but they are hundreds of lines against a specification measured in thousands. -- [ ] Refuse an encrypted document with a message that names the encryption, +- [x] Refuse an encrypted document with a message that names the encryption, rather than extracting nothing and reporting a clean file. -- [ ] Fixture corpus: one PDF per generator family, since the defects this +- [x] Fixture corpus: one PDF per generator family, since the defects this milestone hunts are generator signatures rather than author mistakes. *A file the adapter could not open is not a file with no defects.* Whatever 6.1 cannot read has to reach the report as itself. +**The decision was to own the object layer and take the decompressor**, and +[ADR 0010](adr/0010-write-the-pdf-object-layer-take-the-decompressor.md) is the +record. What settled it was not the size of the specification but which half of +it mirsam asks for: the object graph and the content streams are §7.2 to §7.8, +and everything from §8 on is the rendering this project has said it will never +do. The second cost decided it — a library's object model becomes the adapter's +vocabulary, and with it every answer to *is a dangling reference null, is a +wrong `/Length` readable, is a stale `startxref` fatal*. Those are precisely the +honesty decisions §6.2 turns on, and inheriting them unexamined would have been +inheriting the wrong ones silently. `flate2` is taken for `FlateDecode`, held to +the `zlib-rs` backend `zip` already resolves, so no dependency entered the tree. + +**Every read of the file is checked against the file rather than believed.** A +`/Length` may be an indirect reference — the value is in an object the +cross-reference table has not been parsed yet to find — and plenty of writers +get the direct form wrong by a byte or two. So it is a hint, verified: taken +when `endstream` is where it says, and the data runs to the next `endstream` +otherwise. One rule covers indirect, absent and simply wrong, and no caller had +to be handed a resolver that cannot exist yet. + +**Newest wins, and it is the walk that enforces it.** An incremental update +appends a section and chains `/Prev` at the old one, so every insertion is +first-wins and an object an update redefined is found in the section that +redefined it. A free entry in a newer section therefore *hides* an object an +older one still names, which is what a deletion is. The trailers merge the same +way, which is why a `/Root` a signature moved is the one used. A hybrid file +(§7.5.8.4) carries a classic table *and* an `/XRefStm`, and the stream is read +first because the classic table beside it deliberately understates what the file +holds — a reader that took the classic table as the whole truth would draw a +blank page and report nothing wrong with it. + +**A stale `startxref` is recovered from, not refused.** Offsets go stale: a file +concatenated by a script, truncated by a transfer or edited by hand has one +pointing at nothing in particular, and every viewer opens it anyway by scanning +for `obj`. So does this. *A document reported unreadable while a reader opens it +is the wrong half of standing rule 4*, and `Reconstruction` is how the fact is +said out loud rather than hidden: the cross-reference table and the page tree +each report when they had to be rebuilt. + +**The refusals are the design.** An encrypted document is refused at +`Pdf::open` with the algorithm read out of the `/Encrypt` dictionary, which +§7.6.1 keeps in the clear precisely so a reader can say what it is up against — +`AES-256 (/Standard security handler, V5 R6)`, and a handler this tool does not +know is named rather than guessed at. Nothing decrypts, including the very +common permissions-only wrapper with an empty user password: a tool that opened +those would be one nobody could run on a file they were given. And the four +image codecs are *named*, never decoded, because a stream behind `DCTDecode` is +not text this crate failed to read — it is text that is not in the file. + +**`unread` is native here rather than bolted on.** ADR 0009 gave the port a way +to say what it could not read, and this is the format that needs it: an object +at an offset that is not there, an object stream behind a filter this crate does +not implement, a page drawn as a scan. Each is recorded where it happens and +named as the document names it — `5 0 R — /DCTDecode, a filter this tool does +not decode` — and `Pdf::note_unread` is open so that §6.2's own unread sources, +a font with no `ToUnicode` map among them, reach one list rather than two. + +*Acceptance:* met. Seventy tests over eight fixtures, one per generator +family, written byte by byte by `scripts/make-pdf-fixture.py` — because a +fixture built by the library under test proves only that the library agrees with +itself. A classic table with uncompressed content; a cross-reference stream with +its catalog, page tree, page and font inside an object stream; a hybrid file +whose `ToUnicode` map only the `/XRefStm` names; an incremental update that +supersedes the page's content; a file whose `startxref` points past its own end +and which reads identically to the one it was damaged from; an AES-256 document +refused by name; a distiller's `ASCII85Decode`/`FlateDecode` chain beside +`ASCIIHexDecode`, `RunLengthDecode` and `LZWDecode`; and a scanned page whose +image comes back named in `unread` rather than as a page with nothing wrong with +it. And every prefix and every single-byte corruption of all eight comes back +as a document or as an error and never as a panic, which is what the bounds on +nesting depth, reference-chain length, `/Prev` chain length and page-tree depth +are there for: this tool is run on files people were sent. + +No `DocumentReader` yet and no entry in the CLI's readable list: text +extraction is 6.2, and a `.pdf` the binary accepted before it could read one +would be a claim this milestone has not earned. + ### 6.2 Text extraction and its order `[~]` `BT`/`ET`, `Tj`/`TJ`/`'`/`"`, `Tf`/`Tm`/`Td`/`TD`/`T*`, and the `ToUnicode` diff --git a/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md b/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md new file mode 100644 index 0000000..969c3aa --- /dev/null +++ b/docs/adr/0010-write-the-pdf-object-layer-take-the-decompressor.md @@ -0,0 +1,130 @@ +# 10. Write the PDF object layer; take the decompressor + +Date: 2026-09-07 + +## Status + +Accepted. + +## Context + +`docs/PLAN.md` §6.1 asks the question directly: *"Decide the parser: own it or +take one. This wants an ADR, not a `Cargo.toml` line."* The constraints it sets +are pure Rust, no C, no network, a permissive licence, and no rasteriser. + +The precedent in this repository cuts both ways and says so. `dom.rs` and +`css.rs` were written here rather than taken, and the reasoning was specific: +`markup5ever_rcdom` describes itself as unsupported, and the part of CSS that +decides direction is a few hundred lines. But `html5ever` *was* taken, because +tree construction is a specification with observable, tested behaviour that a +hand-rolled nesting stack gets wrong in ways that move a node's ancestors — and +direction is inherited along ancestors. + +So the question is not "does this project write parsers". It is which half of +this particular specification mirsam needs, and what a dependency would carry +in with it. + +### What mirsam needs from a PDF + +The object graph and the content streams. Concretely: the two spellings of the +cross-reference table, `/Prev` chains, hybrid-reference files, object streams, +the general-purpose stream filters, the page tree with its four inheritable +attributes, and the bytes of the content streams. Above that, §6.2 needs the +text operators and `ToUnicode` CMaps, §6.4 the embedded font programs — and +`mirsam-fonts::sfnt` already parses those. + +### What a PDF library carries + +Most of a PDF library is the half this project has said it will never do. +Graphics state, path construction, colour spaces, transparency groups, shading +patterns, image decoding, annotation appearance streams, form field rendering — +none of it is reachable from anything mirsam asks. The read-write libraries add +a serialiser, an incremental-update writer and an encryption implementation, all +for a format this project has committed to never writing. + +There is a second cost, and it is the one that decided this. A library's object +model becomes the vocabulary of the adapter that uses it, and every subsequent +question — is a missing `/Type` an error, does a dangling reference resolve to +null, is a stream with a wrong `/Length` readable — is answered by that +library's judgement rather than by this project's. Those judgements are exactly +where a document reader is honest or is not. §6.2's whole difficulty is deciding +what the adapter is *entitled to claim* about a file that does not state its own +direction, and inheriting those decisions from a dependency would mean +inheriting them unexamined. + +### And what only a library should supply + +Inflate. `FlateDecode` is a codec, not a document format: it has one correct +answer, that answer is `zlib`'s, and a hand-written implementation would be a +liability with no upside. The same is true of nothing else in this crate. + +## Decision + +**`mirsam-pdf` owns the object layer.** The lexer, the object model, the +cross-reference table and stream, object streams, the page tree, the ASCII and +run-length and LZW filters and the PNG and TIFF predictors are written here, +against ISO 32000-1, with the section numbers in the comments. + +**`flate2` is taken for `FlateDecode`**, held to the `zlib-rs` backend the +workspace already resolves through `zip`. That is the same rule `ttf-parser` +is pinned under: two inflate implementations in one binary would be two answers +to what a stream decompresses to, and no new dependency enters the tree. + +**Three things are refused rather than implemented**, and the refusals are the +design rather than a gap: + +- **Decryption.** An encrypted document is refused at `Pdf::open` with the + algorithm named from the `/Encrypt` dictionary, which §7.6.1 keeps in the + clear. A tool that opened protected documents by trying the empty password + would be one nobody could run on a file they were given. +- **Rasterising.** `DCTDecode`, `JPXDecode`, `CCITTFaxDecode` and `JBIG2Decode` + are named, never decoded. A stream behind one is not text this crate failed + to read; it is text that is not in the file. +- **Writing.** There is no `DocumentWriter` here and none planned. A broken + Arabic PDF is rebuilt from its source document. + +## Consequences + +**The honesty decisions are this project's, and they are visible.** A wrong +`/Length` is verified rather than believed; a dangling reference is null, +because §7.3.9 says so; a reference cycle terminates and answers null rather +than hanging; a stale `startxref` is recovered from by scanning, because a +document reported unreadable while every viewer opens it is the wrong half of +standing rule 4. Each of those is a line in this crate with a test beside it, +and each would otherwise have been a dependency's default. + +**The unread list is native rather than bolted on.** ADR 0009 gave +`DocumentReader` a way to say what it could not read, and PDF is the format +that needs it most — a scanned page, an object at an offset that is not there, +a filter this crate does not implement. Because the loader is ours, each of +those is recorded where it happens, named as the document names it. Through a +library those cases are an `Err` or an empty `Vec`, and the difference between +"no text here" and "no text this tool can see" would have had to be +reconstructed from the outside. + +**It is more code than a dependency line, and less than a PDF library.** Two +and a half thousand lines, comments included, against a specification of some +hundreds of pages — because the pages that matter here are §7.2 through §7.8, +and everything from §8 on is rendering. That ratio is the decision, and it +would not survive the day mirsam needed a graphics state. + +**The bound is stated so it can be tested.** Nesting depth, reference-chain +length, `/Prev` chain length and page-tree depth all have explicit limits, and +a file that exceeds one is malformed rather than a stack overflow. mirsam is +run on files people were sent, which is the threat model a document tool has +whether or not it says so. + +**The cost is real: this crate now owns every PDF bug it has.** A file that +opens in a viewer and not here is this repository's to fix, and the fixture +corpus is the answer to that — one file per generator family, written byte by +byte by `scripts/make-pdf-fixture.py` rather than by a library, so a fixture +can never prove only that the parser agrees with itself. + +## Related + +- Standing rule 4, `AGENTS.md` — "Report only what was verified." +- [ADR 0002](0002-rust-and-token-preserving-xml.md) — the same shape one format + earlier: parse what is needed, keep the bytes, do not adopt a model. +- [ADR 0009](0009-a-source-the-adapter-could-not-read-is-part-of-the-report.md) + — the port this crate's `unread` list feeds. +- `docs/PLAN.md` §6.1, which asked for this record. diff --git a/scripts/make-pdf-fixture.py b/scripts/make-pdf-fixture.py new file mode 100644 index 0000000..99478ce --- /dev/null +++ b/scripts/make-pdf-fixture.py @@ -0,0 +1,546 @@ +#!/usr/bin/env python3 +"""Generate the PDF fixture corpus the object layer is proved against. + +Writes into crates/mirsam-pdf/tests/pdfs/. One file per *generator family*, +because the things PLAN §6 hunts are generator signatures rather than author +mistakes: how a producer assembles a file decides what a reader has to be able +to do before it can see a single character. + + classic.pdf + A cross-reference *table*, uncompressed content, no object streams — + PDF 1.4 as anything predating 2003 writes it, and as a great many + report generators still do. The baseline: if this does not read, + nothing else will. + + compressed.pdf + A cross-reference *stream* and an object stream, PDF 1.7. This is what + LibreOffice, modern Word and Chrome's print-to-PDF emit, and every + object but the stream containers themselves is inside a FlateDecode + stream. A reader with no inflate sees an empty document. + + hybrid.pdf + A classic table *and* an /XRefStm beside it (ISO 32000-1 §7.5.8.4) — + Acrobat's compatibility output, where a 1.4 reader is shown a subset of + what a 1.5 reader is shown. The page's real content object is reachable + only through the stream, so a reader that took the classic table as the + whole truth draws a blank page and reports no defects. + + incremental.pdf + classic.pdf plus an appended update chained by /Prev, replacing the + page's content. Every form filler, signer and annotation tool writes + this shape. Newest-wins is the whole test: read the first definition + and the document says the wrong thing. + + damaged.pdf + Correct objects, a startxref pointing into the middle of nothing. What + a truncated download, a naive concatenation or a hand edit leaves + behind. Every viewer opens it by scanning for `obj`, so mirsam must + too: a document reported unreadable while a reader opens it is the + wrong half of standing rule 4. + + encrypted.pdf + /Encrypt with the AES-256 crypt filter, V5 R6. Must be refused *by + name*. The streams inside are not real ciphertext and this file is not + a cryptographic fixture — nothing here decrypts, and what is being + proved is that the refusal reads the /Encrypt dictionary, which + §7.6.1 keeps in the clear precisely so a reader can say what it is up + against. + + ascii-filters.pdf + ASCIIHexDecode, ASCII85Decode, RunLengthDecode and LZWDecode, one per + stream, plus an ASCII85+Flate chain. The distiller family: dvips and + pdfTeX emitted ASCII-safe streams for decades so that a PDF survived a + mail gateway, and the files are still in circulation. + + image-only.pdf + A page whose entire content is a DCTDecode image. The scanner family, + and the one this crate must be *unable* to read: the text is not in the + file. It has to come back named in `unread` rather than as a page with + nothing wrong with it. + +Every one of them is written byte by byte here, with no third-party module, +for the reason the OOXML generators avoid python-pptx: a fixture built by the +library under test proves the library agrees with itself. + +Regenerate with `make pdfs`. +""" + +from __future__ import annotations + +import zlib +from pathlib import Path + +OUT = Path(__file__).resolve().parent.parent / "crates" / "mirsam-pdf" / "tests" / "pdfs" + +# The Arabic every fixture carries, so §6.2 has something to grow into and so a +# reader that silently drops non-ASCII fails here rather than on a user's file. +ARABIC = "التقرير السنوي" + + +def utf16be(text: str) -> bytes: + """A PDF text string in UTF-16BE with the byte-order mark §7.9.2.2 wants.""" + return b"\xfe\xff" + text.encode("utf-16-be") + + +def hex_string(data: bytes) -> bytes: + return b"<" + data.hex().upper().encode("ascii") + b">" + + +def tounicode(codes: dict[int, str]) -> bytes: + """A ToUnicode CMap mapping single-byte codes to the characters they draw. + + This is how a generator says what its glyph codes mean, and the only thing + standing between an extractor and a page of numbers. + """ + pairs = b"".join( + b"<%02X> <%s>\n" % (code, char.encode("utf-16-be").hex().upper().encode("ascii")) + for code, char in sorted(codes.items()) + ) + return ( + b"/CIDInit /ProcSet findresource begin\n" + b"12 dict begin\nbegincmap\n" + b"/CMapName /Mirsam-Fixture def\n/CMapType 2 def\n" + b"1 begincodespacerange\n<00> \nendcodespacerange\n" + + b"%d beginbfchar\n" % len(codes) + + pairs + + b"endbfchar\n" + b"endcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n" + ) + + +def arabic_codes(text: str, first: int = 0x41) -> tuple[bytes, dict[int, str]]: + """One byte per character, plus the map back. The shape a generator that + subsets a font produces, and the shape §6.3 hunts a reversal in.""" + codes: dict[int, str] = {} + out = bytearray() + for index, char in enumerate(text): + code = first + index + codes[code] = char + out.append(code) + return bytes(out), codes + + +class Builder: + """Objects in, PDF out. Holds the offsets a cross-reference table needs.""" + + def __init__(self, version: str = "1.4") -> None: + self.version = version + self.objects: dict[int, bytes] = {} + self.next = 1 + + def add(self, body: bytes, number: int | None = None) -> int: + if number is None: + number = self.next + self.next = max(self.next, number + 1) + self.objects[number] = body + return number + + def stream(self, dict_entries: bytes, data: bytes, number: int | None = None) -> int: + body = b"<< " + dict_entries + b" /Length %d >>\nstream\n" % len(data) + data + b"\nendstream" + return self.add(body, number) + + def serialise(self, root: int, info: int | None = None, extra_trailer: bytes = b"") -> bytes: + out = bytearray(b"%%PDF-%s\n%%\xe2\xe3\xcf\xd3\n" % self.version.encode("ascii")) + offsets: dict[int, int] = {} + for number in sorted(self.objects): + offsets[number] = len(out) + out += b"%d 0 obj\n" % number + self.objects[number] + b"\nendobj\n" + + size = max(self.objects) + 1 + startxref = len(out) + out += b"xref\n0 %d\n" % size + out += b"0000000000 65535 f \n" + for number in range(1, size): + if number in offsets: + out += b"%010d 00000 n \n" % offsets[number] + else: + out += b"0000000000 65535 f \n" + trailer = b"<< /Size %d /Root %d 0 R" % (size, root) + if info is not None: + trailer += b" /Info %d 0 R" % info + trailer += extra_trailer + b" >>" + out += b"trailer\n" + trailer + b"\nstartxref\n%d\n%%%%EOF\n" % startxref + return bytes(out) + + +def page_document(builder: Builder, content: bytes, *, font_extra: bytes = b"") -> tuple[int, int]: + """The four objects every one-page fixture shares. Returns (catalog, content).""" + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + 3, + ) + stream = builder.stream(b"", content, 4) + builder.add( + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica" + font_extra + b" >>", + 5, + ) + return catalog, stream + + +def content_for(text: str, *, first: int = 0x41) -> tuple[bytes, dict[int, str]]: + codes, mapping = arabic_codes(text, first) + stream = ( + b"BT\n/F1 18 Tf\n72 760 Td\n(Annual report) Tj\n" + b"0 -28 Td\n" + hex_string(codes) + b" Tj\nET\n" + ) + return stream, mapping + + +def write(name: str, data: bytes) -> None: + OUT.mkdir(parents=True, exist_ok=True) + (OUT / name).write_bytes(data) + print(f" {name:22} {len(data):7} bytes") + + +# -------------------------------------------------------------------------- +# classic.pdf +# -------------------------------------------------------------------------- +def classic() -> bytes: + builder = Builder("1.4") + content, mapping = content_for(ARABIC) + catalog, _ = page_document(builder, content, font_extra=b" /ToUnicode 6 0 R") + builder.stream(b"", tounicode(mapping), 6) + info = builder.add( + b"<< /Title " + hex_string(utf16be(ARABIC)) + b" /Producer (mirsam fixtures) >>", 7 + ) + return builder.serialise(catalog, info) + + +# -------------------------------------------------------------------------- +# compressed.pdf — cross-reference stream and object stream +# -------------------------------------------------------------------------- +def compressed() -> bytes: + content, mapping = content_for(ARABIC) + content_z = zlib.compress(content, 9) + cmap_z = zlib.compress(tounicode(mapping), 9) + + # 1..3 and 7 go inside the object stream; 4, 5 and 6 are streams, which + # §7.5.7 forbids an object stream from holding. + inside = { + 1: b"<< /Type /Catalog /Pages 2 0 R >>", + 2: b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", + 3: ( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>" + ), + 5: b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>", + 7: b"<< /Title " + hex_string(utf16be(ARABIC)) + b" /Producer (mirsam fixtures) >>", + } + pairs = bytearray() + bodies = bytearray() + for number, body in sorted(inside.items()): + pairs += b"%d %d " % (number, len(bodies)) + bodies += body + b" " + objstm_plain = bytes(pairs) + bytes(bodies) + first = len(pairs) + objstm_z = zlib.compress(objstm_plain, 9) + + out = bytearray(b"%PDF-1.7\n%\xe2\xe3\xcf\xd3\n") + offsets: dict[int, int] = {} + + def emit(number: int, body: bytes) -> None: + offsets[number] = len(out) + out.extend(b"%d 0 obj\n" % number + body + b"\nendobj\n") + + emit( + 4, + b"<< /Filter /FlateDecode /Length %d >>\nstream\n" % len(content_z) + + content_z + + b"\nendstream", + ) + emit( + 6, + b"<< /Filter /FlateDecode /Length %d >>\nstream\n" % len(cmap_z) + cmap_z + b"\nendstream", + ) + emit( + 8, + b"<< /Type /ObjStm /N %d /First %d /Filter /FlateDecode /Length %d >>\nstream\n" + % (len(inside), first, len(objstm_z)) + + objstm_z + + b"\nendstream", + ) + + # 9 is the cross-reference stream itself: three fields, 1/4/2 bytes wide. + size = 10 + rows = bytearray() + startxref = len(out) + for number in range(size): + if number == 0: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + elif number in offsets: + rows += bytes([1]) + offsets[number].to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number == 9: + rows += bytes([1]) + startxref.to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number in inside: + index = sorted(inside).index(number) + rows += bytes([2]) + (8).to_bytes(4, "big") + index.to_bytes(2, "big") + else: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + rows_z = zlib.compress(bytes(rows), 9) + out.extend( + b"9 0 obj\n<< /Type /XRef /Size %d /W [1 4 2] /Root 1 0 R /Info 7 0 R" + b" /Filter /FlateDecode /Length %d >>\nstream\n" % (size, len(rows_z)) + + rows_z + + b"\nendstream\nendobj\n" + ) + out.extend(b"startxref\n%d\n%%%%EOF\n" % startxref) + return bytes(out) + + +# -------------------------------------------------------------------------- +# hybrid.pdf — a classic table with /XRefStm beside it +# -------------------------------------------------------------------------- +def hybrid() -> bytes: + content, mapping = content_for(ARABIC) + cmap = tounicode(mapping) + + out = bytearray(b"%PDF-1.5\n%\xe2\xe3\xcf\xd3\n") + offsets: dict[int, int] = {} + + def emit(number: int, body: bytes) -> None: + offsets[number] = len(out) + out.extend(b"%d 0 obj\n" % number + body + b"\nendobj\n") + + emit(1, b"<< /Type /Catalog /Pages 2 0 R >>") + emit(2, b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>") + emit( + 3, + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + ) + emit(4, b"<< /Length %d >>\nstream\n" % len(content) + content + b"\nendstream") + emit(5, b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>") + # 6 is the object the classic table deliberately does not name: only the + # /XRefStm knows where it is, which is the whole point of a hybrid file. + emit(6, b"<< /Length %d >>\nstream\n" % len(cmap) + cmap + b"\nendstream") + + size = 8 + rows = bytearray() + xrefstm_at = len(out) + for number in range(size): + if number in offsets: + rows += bytes([1]) + offsets[number].to_bytes(4, "big") + (0).to_bytes(2, "big") + elif number == 7: + rows += bytes([1]) + xrefstm_at.to_bytes(4, "big") + (0).to_bytes(2, "big") + else: + rows += bytes([0]) + (0).to_bytes(4, "big") + (0xFFFF).to_bytes(2, "big") + rows_z = zlib.compress(bytes(rows), 9) + out.extend( + b"7 0 obj\n<< /Type /XRef /Size %d /W [1 4 2] /Root 1 0 R" + b" /Filter /FlateDecode /Length %d >>\nstream\n" % (size, len(rows_z)) + + rows_z + + b"\nendstream\nendobj\n" + ) + + startxref = len(out) + out.extend(b"xref\n0 6\n0000000000 65535 f \n") + for number in range(1, 6): + out.extend(b"%010d 00000 n \n" % offsets[number]) + out.extend( + b"trailer\n<< /Size %d /Root 1 0 R /XRefStm %d >>\nstartxref\n%d\n%%%%EOF\n" + % (size, xrefstm_at, startxref) + ) + return bytes(out) + + +# -------------------------------------------------------------------------- +# incremental.pdf — classic.pdf, then an update that supersedes the content +# -------------------------------------------------------------------------- +def incremental() -> bytes: + base = classic() + first_startxref = int(base.rsplit(b"startxref\n", 1)[1].split(b"\n", 1)[0]) + + revised, _ = content_for("التقرير السنوي المعدل") + out = bytearray(base) + offset = len(out) + out.extend( + b"4 0 obj\n<< /Length %d >>\nstream\n" % len(revised) + revised + b"\nendstream\nendobj\n" + ) + startxref = len(out) + out.extend( + b"xref\n0 1\n0000000000 65535 f \n4 1\n%010d 00000 n \n" % offset + + b"trailer\n<< /Size 8 /Root 1 0 R /Info 7 0 R /Prev %d >>\nstartxref\n%d\n%%%%EOF\n" + % (first_startxref, startxref) + ) + return bytes(out) + + +# -------------------------------------------------------------------------- +# damaged.pdf — good objects, a startxref pointing nowhere useful +# -------------------------------------------------------------------------- +def damaged() -> bytes: + base = classic() + head, _ = base.rsplit(b"startxref\n", 1) + # Past the end of the file, which is what a truncated transfer leaves and + # what a reader must survive by scanning the body instead. + return head + b"startxref\n%d\n%%%%EOF\n" % (len(base) + 4096) + + +# -------------------------------------------------------------------------- +# encrypted.pdf — refused by name, never decrypted +# -------------------------------------------------------------------------- +def encrypted() -> bytes: + builder = Builder("1.7") + content, mapping = content_for(ARABIC) + catalog, _ = page_document(builder, content, font_extra=b" /ToUnicode 6 0 R") + builder.stream(b"", tounicode(mapping), 6) + encrypt = builder.add( + b"<< /Filter /Standard /V 5 /R 6 /Length 256" + b" /CF << /StdCF << /CFM /AESV3 /AuthEvent /DocOpen /Length 32 >> >>" + b" /StmF /StdCF /StrF /StdCF" + b" /O <" + b"00" * 48 + b"> /U <" + b"00" * 48 + b">" + b" /OE <" + b"00" * 32 + b"> /UE <" + b"00" * 32 + b">" + b" /Perms <" + b"00" * 16 + b"> /P -1340 >>", + 7, + ) + return builder.serialise(catalog, extra_trailer=b" /Encrypt %d 0 R" % encrypt) + + +# -------------------------------------------------------------------------- +# ascii-filters.pdf — the distiller family +# -------------------------------------------------------------------------- +def ascii_hex(data: bytes) -> bytes: + return data.hex().upper().encode("ascii") + b">" + + +def ascii85(data: bytes) -> bytes: + out = bytearray() + for start in range(0, len(data), 4): + group = data[start : start + 4] + pad = 4 - len(group) + value = int.from_bytes(group + b"\0" * pad, "big") + if value == 0 and pad == 0: + out += b"z" + continue + digits = bytearray() + for _ in range(5): + digits.insert(0, 33 + value % 85) + value //= 85 + out += bytes(digits)[: 5 - pad] + return bytes(out) + b"~>" + + +def run_length(data: bytes) -> bytes: + """Literal runs only. A decoder that mishandles the length byte produces + plausible rubbish, which is what this is here to catch.""" + out = bytearray() + for start in range(0, len(data), 128): + chunk = data[start : start + 128] + out.append(len(chunk) - 1) + out += chunk + return bytes(out) + b"\x80" + + +def lzw(data: bytes) -> bytes: + """Variable-code LZW with early change, as §7.4.4.2 defines it.""" + table = {bytes([b]): b for b in range(256)} + nxt = 258 + width = 9 + bits = 0 + held = 0 + out = bytearray() + + def push(code: int) -> None: + nonlocal bits, held + bits = (bits << width) | code + held += width + while held >= 8: + out.append((bits >> (held - 8)) & 0xFF) + held -= 8 + + push(256) + current = b"" + for byte in data: + candidate = current + bytes([byte]) + if candidate in table: + current = candidate + continue + push(table[current]) + table[candidate] = nxt + nxt += 1 + # Early change: the width grows one code before the table is full. + if nxt + 1 > (1 << width) and width < 12: + width += 1 + current = bytes([byte]) + if current: + push(table[current]) + push(257) + if held: + out.append((bits << (8 - held)) & 0xFF) + return bytes(out) + + +def ascii_filters() -> bytes: + builder = Builder("1.4") + content, mapping = content_for(ARABIC) + cmap = tounicode(mapping) + + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents [4 0 R 8 0 R 9 0 R]" + b" /Resources << /Font << /F1 5 0 R >> >> >>", + 3, + ) + # The page's own content, through ASCII85 and Flate in that order — the + # chain a distiller writes, and the one a reader must unwind in reverse. + chained = ascii85(zlib.compress(content, 9)) + builder.stream(b"/Filter [/ASCII85Decode /FlateDecode]", chained, 4) + builder.add(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /ToUnicode 6 0 R >>", 5) + builder.stream(b"/Filter /ASCIIHexDecode", ascii_hex(cmap), 6) + builder.add(b"<< /Producer (mirsam fixtures) >>", 7) + builder.stream(b"/Filter /RunLengthDecode", run_length(b"% run length\n"), 8) + builder.stream(b"/Filter /LZWDecode", lzw(b"% lzw\n"), 9) + return builder.serialise(catalog, 7) + + +# -------------------------------------------------------------------------- +# image-only.pdf — the scanner family, which has no text to find +# -------------------------------------------------------------------------- +# The smallest thing a JPEG decoder would accept, and this one never runs: what +# matters is that the filter is named and the bytes are left alone. +JPEG = bytes.fromhex( + "ffd8ffe000104a46494600010100000100010000ffdb004300" + + "08" * 64 + + "ffc0000b080001000101011100ffc40014000100000000000000000000000000000000" + "03ffda0008010100003f00d2cf20ffd9" +) + + +def image_only() -> bytes: + builder = Builder("1.4") + content = b"q\n595 0 0 842 0 0 cm\n/Im1 Do\nQ\n" + catalog = builder.add(b"<< /Type /Catalog /Pages 2 0 R >>", 1) + builder.add(b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 595 842] >>", 2) + builder.add( + b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R" + b" /Resources << /XObject << /Im1 5 0 R >> >> >>", + 3, + ) + builder.stream(b"", content, 4) + builder.stream( + b"/Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray" + b" /BitsPerComponent 8 /Filter /DCTDecode", + JPEG, + 5, + ) + return builder.serialise(catalog) + + +def main() -> None: + print(f"writing PDF fixtures into {OUT}") + write("classic.pdf", classic()) + write("compressed.pdf", compressed()) + write("hybrid.pdf", hybrid()) + write("incremental.pdf", incremental()) + write("damaged.pdf", damaged()) + write("encrypted.pdf", encrypted()) + write("ascii-filters.pdf", ascii_filters()) + write("image-only.pdf", image_only()) + + +if __name__ == "__main__": + main()