[Kikobeats]

Three separately measured commits.

1. Probe the default pandoc lazily and async

pandoc = defaultPandoc() was a default parameter, so the first getHTML call of every process ran execFileSync('pandoc', ['--list-input-formats']), even for plain HTML. The probe now runs only when a response is an office document, async through tinyspawn, once per process (memoized promise).

The benchmark harness has no first-call case, so this is a one-off measurement: a fresh process serving a local plain HTML page, timing the synchronous part of the first getHTML() call, 5 processes each.

	base	change
sync block on first call	33.3-46.8ms	7.0-10.3ms

The remaining ~4-5ms of that is defaultMutool() running which mutool the same way (not touched here).

Unchanged: explicit pandoc values (function or false), the exported sync defaultPandoc(), and the branching (an office file mislabeled application/pdf with no pandoc installed still goes to mutool, because pandoc availability is resolved before that branch).

2. Bound the content-type cache

src/util.js cached parsed content-types keyed by the raw remote header, unbounded: 200k distinct headers retained 64.6MB. It now resets after 1000 entries. Hit path microbench (4 hot headers, getContentType + getCharset): 16.6ns base vs 15.7ns change, so no regression. 200k distinct headers now retain ~0MB.

3. Rewrite url attributes in one tree walk

rewriteHtmlUrls ran one full-document selector query per url attribute in html-urls TAGS (10 walks; domutils find was the top self-time frame). It now selects every url-bearing tag once and checks that tag's attributes in the same order.

Output is identical. An element removed for a local address ends up removed either way, and surviving elements get the same rewrites. Checked by rendering old and new on every HTML fixture under test/ plus crafted cases (nested local-address elements, ping/href, formaction/src on the same element, uppercase tags, invalid and non-http urls), 3 base urls, rewriteUrls on/off: 48 renders, byte-identical.

A/B

benchmark/index.js unchanged. 6 alternating runs per side, median of per-run medians. node v24.16.0, arm64 (12 cores), load1 8.8-9.4.

case	base	change	delta
html rewriteUrls off	71.3	73.2	+2.7% (noise, ranges overlap: 67.8-79.2 vs 63.8-88.8)
html rewriteUrls on	143.0	111.2	-22.2%
pdf extract	62.3	64.1	+2.9% (noise, untouched code)
cold require	156.9	173.5	noise: rerun as 25 alternating spawns per side, 174.0 vs 174.9 median

Behavior changes

None. The test suite passes. Two tests added. One checks that a plain HTML call never runs pandoc and the first office document probes it exactly once (it fails on master). The other checks that the content-type cache keeps resolving after eviction.

How tested

npm test (standard + c8 ava): 143 passed, 54 skipped (pre-existing skips). The one live-network test (from image URL that returns HTML markup, timing < 3000) flaked once and passed on rerun.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches core getHTML fetch branching (office, PDF, pandoc/mutool) and HTML URL sanitization; changes are tested but alter when external binaries are spawned and how document types are classified.

Overview
This PR bundles three performance-oriented changes to the HTML fetch pipeline, with expanded tests around each area.

Lazy async default pandoc: getHTML no longer runs a synchronous pandoc --list-input-formats probe on the first call via a default parameter. The default is a sentinel resolved with resolvePandoc only when an office format is detected, using a memoized async loadDefaultPandoc. Plain HTML requests avoid the spawn cost; concurrent office fetches share one probe.

Bounded content-type cache: Parsed Content-Type headers in util.js are capped at 1000 distinct keys, with a full cache reset on overflow, replacing an unbounded in-memory map.

Single-pass URL rewriting: rewriteHtmlUrls builds a tag→attributes map from html-urls and walks URL-bearing elements once, rewriting every applicable attribute per element (including continuing after skipped protocols and removing the node on the first local-address hit).

Office-vs-PDF routing now always runs getOfficeFormat and only attempts pandoc when a converter is available (convertOffice), which keeps mislabeled office files on the pandoc path when enabled and allows PDF-labeled office bytes to fall through to mutool when pandoc is off—behavior covered by new integration tests.

^{Reviewed by Cursor Bugbot for commit 657c7d0. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

• Bug Fixes

  • Office documents are recognized and converted even when responses are labeled as PDFs, without sending correctly identified office files to PDF extraction.
  • Processing of supported links continues when another URL attribute is invalid or uses an unsupported protocol. Elements with local-address links are still removed.
  • Content-type lookups remain accurate after many distinct values have been checked.

• Tests

  • Added coverage for office-document handling, link rewriting, and content-type lookups.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: ab14e451-c39d-47a5-aebb-a91c92858d9d

Commits

Reviewing files that changed from the base of the PR and between 5d5c2c2 and 657c7d0.

Files selected for processing: 3

• test/html/rewrite-urls.js
• test/office.js
• test/util/get-content-type.js

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

Walkthrough

The change updates HTML URL rewriting, defers default Pandoc probing until office conversion is requested, and bounds the content-type cache at 1,000 entries. Tests cover URL handling, office conversion behavior, and cache overflow.

Changes: HTML URL rewriting

Layer / File(s)	Summary
Element URL processing src/html.js, test/html/rewrite-urls.js	The combined selector finds URL-bearing elements once. Non-string values and non-HTTP protocols skip only the current attribute. HTTP(S) URLs are made absolute, and an element is removed when an attribute resolves to a local address. Tests cover these cases and early termination after a local address is found.

Office conversion

Layer / File(s)	Summary
Lazy converter resolution src/index.js, test/office.js	The default Pandoc runner is resolved lazily. Its supported-format probe and converter are cached. Tests cover deferred probing, custom and disabled runners, and probe failures.
Office detection and conversion routing src/index.js, test/office.js	Office-format detection no longer depends on a configured runner. Conversion runs only when a converter is resolved. Tests cover fallback behavior for PDF-labeled office files when Pandoc is disabled.

Content-type cache

Layer / File(s)	Summary
Bounded cache and lookup checks src/util.js, test/util/get-content-type.js	The cache clears before an insertion would exceed 1,000 entries. Cache hits are checked against undefined. Tests cover overflow, reparsing, and subsequent caching.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant Request as Office conversion path
  participant Resolver as resolvePandoc
  participant Pandoc as Pandoc executable
  participant Converter as Office converter
  Request->>Resolver: Resolve converter for detected office format
  Resolver->>Pandoc: Probe supported input formats on first use
  Pandoc-->>Resolver: Return supported input formats
  Resolver-->>Request: Return converter
  Request->>Converter: Convert detected format and temporary file

Loading

Possibly related PRs

• microlinkhq/html-get#302: Adds benchmarks for URL rewriting and cold module loading, which measure behaviors affected by this change.

Merge Risk: Minimal · up to 657c7

No material merge risks were identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the three main performance changes: lazy Pandoc probing, a bounded content-type cache, and a single-walk URL rewrite.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 6 files.

Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

html-get

Details

Benchmark suite	Current: 657c7d0	Previous: 4fec2ee	Ratio
html rewriteUrls off	147 ms (133.2-189.0)	163 ms (146.3-226.4)	0.90
html rewriteUrls on	223.8 ms (215.5-291.6)	326.3 ms (308.8-390.2)	0.69
pdf extract	98.3 ms (92.2-129.8)	150.8 ms (145.4-213.8)	0.65
cold require	236.1 ms (220.2-244.1)	317.9 ms (302.5-328.1)	0.74

This comment was automatically generated by workflow using github-action-benchmark.