Rewrite as Nokogiri-compatible FFI binding for libtaurus v0.5.14 (v0.1.0) - #1
Merged
Conversation
…1.0)
Complete rewrite. The C DOM is now the single source of truth; Ruby
objects are thin FFI wrappers (one Ruby method = one FFI call).
Added (lib/taurus/xml/):
- Document, Element, Node, Text, Comment, CDATA, ProcessingInstruction,
Attr, Namespace, NodeSet, Searchable, ParseOptions
- SAX::Parser + SAX::Document handler base class
- Minimal CSS-to-XPath translator covering the common Nokogiri subset
- C14N with all modes (canonical 1.0/1.1, exclusive, with-comments,
inclusive namespaces) via taurus_c14n_canonicalize_ex / _subtree_ex
- Document/Element#to_xml, Document#save, Document/Element#canonicalize
Removed:
- Pure-Ruby XML tree model (lib/taurus/{document,element,node,node_set}.rb)
- Pure-Ruby XPath engine (lib/taurus/xpath/)
- Stale bundled C source at ext/taurus/lib/
- taurus CLI (lib/taurus/cli.rb, lib/taurus/commands/)
- Pure-Ruby adapter framework (lib/taurus/adapter*)
- Thor runtime dependency
- spec/taurus/* (old specs that crashed on the new dylib)
- spec/spec_helper.rb's dependency on the broken old entry point
Build / CI:
- Gemfile trimmed to ffi + rake + rspec + rubocop
- taurus.gemspec: spec.executables = [], spec.extensions = [],
description updated, thor dropped
- .github/workflows/test.yml replaced with canon's rake.yml + release.yml
(use metanorma/ci and relaton/support reusable workflows)
- lib/libtaurus.dylib removed from index (was tracked by mistake);
users install libtaurus v0.5.14+ themselves
Specs: 156 passing, 0 pending. Run with 'bundle exec rake'.
…ent parsing, DOCTYPE access libtaurus v0.6.0/v0.6.1 shipped 4 Nokogiri-compat C-API gaps: - taurus_element_copy(src, dest_doc) — Element/Node#dup, #clone - taurus_document_copy(src) — Document#dup, #clone - taurus_node_get_xpath(node) — Node#path, #css_path - taurus_parse_fragment(xml, len, dest_doc, status) — Document#fragment, Element#add_child with String markup - taurus_document_internal_subset + taurus_doctype_get_* — DocType class Ruby binding changes: - New classes: DocType (name/root_name/public_id/system_id/internal_subset/ external_id/to_s), DocumentFragment (children, name) - Node#path, #css_path, #dup, #clone - Element#dup, #clone, add_child(String) via fragment parsing - Document#fragment, #dup, #clone, #doctype, #internal_subset 1 new upstream bug filed: #253 — DOCTYPE PUBLIC/SYSTEM/internal_subset not exposed (only the name comes through). 4 specs marked pending pending upstream fix. Specs: 176 passing, 4 pending (all blocked on #253).
Run with: bundle exec ruby -Ilib benchmark/taurus_vs_nokogiri.rb Findings (libtaurus v0.11.0 vs Nokogiri 1.19.4): - Parse small (431 B): Taurus 1.99x faster - Parse medium (12 KB): Taurus 6.51x faster - XPath count() / boolean() (scalar returns): Taurus 5.45x faster - XPath predicate (single match): Taurus 12.39x faster - Serialize: Taurus 2.90x faster - XPath nodeset-returning (100 nodes): Nokogiri 7x faster - XPath union (200 nodes): Nokogiri 6x faster - Tree traversal: Nokogiri 2x faster Losses attributable to Ruby-side Node/NodeSet wrapper allocation: - NodeSet.from_result eagerly materializes every match - Node.wrap creates a new wrapper per c_ptr per call (no cache) - traverse path allocates a wrapper per visited node These are addressable without libtaurus work — lazy NodeSet, wrapper cache, specialized traverse. 1 new upstream bug filed: #256 — taurus_parse_string segfaults under tight parse loops on ~38 KB docs (memory pool reuse issue). benchmark/README.md has the full analysis.
v0.11.1 perf is within run-to-run variance of v0.11.0 (no algorithmic change). The v0.11.1 fix for #256 covers the GC-pressure path but not the parse+explicit-free path; both still segfault on ~38 KB docs. Added link to the follow-up comment on #256.
Two independent fixes combined to resolve most of the v0.11.0-era crashes and gaps: libtaurus v0.11.2 fixes: - #253 (DOCTYPE): PUBLIC/SYSTEM identifiers now exposed correctly. 3 of 4 DOCTYPE specs pass; internal_subset still nil (partial fix). - #256 (segfault): taurus_node_freeze converted from recursive to iterative with fixed 256-deep explicit stack. Partial fix — addresses the unbounded-recursion vector but residual issue remains under benchmark-ips on 38 KB docs (see #256 follow-up comment). Ruby binding lifetime refactor (lib/taurus/xml/document.rb): - Replaced FFI::AutoPointer with plain FFI::Pointer + ObjectSpace finalizer sharing a Freed flag container between instance and finalizer. AutoPointer double-freed when Document#free was called explicitly and then GC ran later — that was the actual root cause of the parse-loop crashes attributed to libtaurus #256 in earlier runs, not just the C-level stale-thread-local. - After the refactor, 5000 parse+free iterations on 38 KB docs work cleanly. 1000 parses with GC pressure work cleanly. Only benchmark-ips (which accumulates return values differently) still crashes inside taurus_node_freeze+0x150. Specs: 176 passing, 1 pending (DOCTYPE internal_subset, #253 partial). Slight numeric regression vs v0.11.0 (parse small 5.49 -> 9.02 us, serialize 27.40 -> 33.73 us) is the cost of the explicit finalizer pattern — ObjectSpace.define_finalizer per-Document is heavier than FFI::AutoPointer's built-in GC hook. Worth it for correctness. Benchmark wins preserved: - Parse medium (12 KB): 7.09x faster than Nokogiri - XPath count() (scalar): 5.63x faster - XPath predicate (1 match): 14.03x faster - Serialize: 2.94x faster - Parse small (431 B): 1.68x faster
…on 8/9 ops Two Ruby-side optimizations to close the nodeset-XPath gap without waiting on libtaurus #262: 1. Lazy NodeSet (lib/taurus/xml/node_set.rb) NodeSet.from_result used to walk the entire TaurusXPathResult and call Node.wrap for each entry on construction. For a 100-node query that was 100x FFI calls + 100x Ruby allocations just to build the container. The new code keeps the TaurusXPathResult pointer alive via FFI::AutoPointer and materializes self[i] on demand via taurus_xpath_result_get. 2. Per-Document Node wrapper cache (lib/taurus/xml/document.rb, lib/taurus/xml/node.rb) Each Document now owns an ObjectSpace::WeakMap keyed on c_ptr address. Node.wrap checks the cache before allocating. Dies with the document so no stale entries pointing at freed memory. Benchmark deltas (libtaurus v0.11.2, M1, Nokogiri 1.19.4): | Operation | Before | After | Taurus/Nokogiri | |----------------------------|-----------|----------|-----------------| | XPath //book (100 nodes) | 87.77 us | 2.70 us | 0.15x -> 3.43x | | XPath union (200 nodes) | 188.40 us | 10.61 us | 0.14x -> 2.13x | | XPath complex | 126.88 us | 64.54 us | 0.67x -> 1.13x | | Tree traversal | 1203 us | 777 us | 0.50x -> 0.61x | | Parse small | 9.02 us | 6.24 us | 1.68x -> 1.95x | Taurus now beats Nokogiri on 8 of 9 benchmarked operations. Only tree traversal still loses (by 1.6x) — closing that needs the C-side back-ref proposal in libtaurus #262. Specs: 176 passing, 1 pending (DOCTYPE internal_subset, libtaurus #253).
…sues closed libtaurus v0.12.0 shipped both #262 proposals + finished #253 and #261: - v0.11.4: taurus_xpath_result_get_nodes (batch accessor) - v0.12.0: taurus_node_get/set_binding_wrapper (per-node back-ref field) - v0.12.0: #261 fully resolved (benchmark-ips on 38 KB docs no longer segfaults) - v0.12.0: #253 internal_subset now exposed (last pending spec removed) Ruby binding changes: - FFI declarations added for the 3 new functions (172 total symbols) - NodeSet#each uses taurus_xpath_result_get_nodes for batch fetch (1 FFI call instead of N) when iterating a lazy XPath result - NodeSet#to_a memoizes after first materialization (subsequent iterations hit the cache) - binding_wrapper field left for non-Ruby FFI bindings — Ruby's ObjectSpace::WeakMap cache is faster (no FFI call per lookup). The field is opaque to libtaurus and not used by this binding. Specs: 176 passing, 0 pending. All upstream issues closed (#166-#262, 21 issues total). Benchmark deltas (median of 3 runs, libtaurus v0.12.0, Nokogiri 1.19.4): - Parse small: Taurus ~2x faster - Parse medium (12 KB): Taurus ~6x faster - XPath count() (scalar): Taurus ~6x faster - XPath //book (100 nodes): Taurus ~5x faster - XPath predicate (1 match): Taurus ~10x faster - XPath complex: Taurus ~1.3x faster - XPath union (200 nodes): Taurus ~3x faster - Serialize: Taurus ~3x faster - Tree traversal: Nokogiri ~1.8x faster (only place Taurus still loses) Taurus beats Nokogiri on 8 of 9 benchmarked operations. Tree traversal remains the only loss — Ruby-side wrap-on-visit cost is fundamental; fixing it needs a specialized read-only traverse path, not libtaurus work.
Previous traverse went through Element#children which built a NodeSet (Array + Node.wrap per child) per parent. For a tree of N nodes, that's ~N NodeSet allocations + ~N Array allocations on a full traversal. New traverse walks via raw FFI (taurus_node_first_child + taurus_node_next_sibling) and wraps each node directly via Node.wrap, skipping the NodeSet allocation. Same FFI call count, fewer allocations. Benchmark (median of multiple runs, libtaurus v0.12.0, Nokogiri 1.19.4): - Before: ~900 µs/iter (0.55x vs Nokogiri, Nokogiri 1.8x faster) - After: ~600 µs/iter (0.9x-1.1x vs Nokogiri, roughly tied) - With GC disabled (theoretical max): 459 µs vs Nokogiri 682 µs (1.49x faster) The realistic case is noisy because Nokogiri benefits from GC pressure (it allocates per-visit and the GC pressure happens to align well with its allocation rate). Taurus with the WeakMap cache allocates once per unique node, so GC pressure hurts less. To CONSISTENTLY beat Nokogiri (not just match), needs libtaurus #273 (C-side traverse with callback). Tracked upstream. 176 passing specs, 0 pending.
CI runs on macos-latest, ubuntu-latest, windows-latest × Ruby 3.3/3.4/4.0.
Gemfile.lock was generated locally on macOS arm64 only and rejected by
Bundler on other platforms ('Your bundle only supports arm64-darwin,
x86_64-darwin but your local platform is x86_64-linux'). Adding the
matrix platforms lets bundle install succeed on every CI runner.
The metanorma/ci reusable workflow supports a 'before-setup-ruby' command input. Using it to download + build libtaurus v0.12.0 on each runner (macos/ubuntu/windows), then export TAURUS_LIB_PATH so the FFI binding finds the shared library. Build flags: shared only, no CLI/tests/benchmarks/man-pages, no optional deps (utf8proc/iconv) — keeps the install fast and dependency-free.
…ink) Previous find -type f skipped libtaurus.so on Linux because it's a symlink to libtaurus.so.0.12.0. Use -type f -o -type l to catch both real files and symlinks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrite as Nokogiri-compatible FFI binding for libtaurus v0.5.14 (v0.1.0)
Complete rewrite. The C DOM is the single source of truth; Ruby objects are thin FFI wrappers (one Ruby method = one FFI call).
What changed
Added (
lib/taurus/xml/):Document,Element,Node,Text,Comment,CDATA,ProcessingInstruction,Attr,Namespace,NodeSet,Searchable,ParseOptionsSAX::Parser+SAX::Documenthandler base classtaurus_c14n_canonicalize_ex/_subtree_exDocument/Element#to_xml,Document#save,Document/Element#canonicalizeRemoved:
lib/taurus/{document,element,node,node_set}.rb)lib/taurus/xpath/)ext/taurus/lib/taurusCLIspec/taurus/that crashed on the new dylibBuild / CI:
Gemfiletrimmed toffi + rake + rspec + rubocoptaurus.gemspec:spec.executables = [],spec.extensions = [], description updated, thor dropped.github/workflows/test.ymlreplaced with canon'srake.yml+release.yml(metanorma/ci and relaton/support reusable workflows)lib/libtaurus.dylibremoved from git index (was tracked by mistake); users install libtaurus v0.5.14+ themselvesTest results
156 examples, 0 failures, 0 pending. Run with
bundle exec rake.Required external dependency
libtaurus v0.5.14 or later, installed separately. Get it from
https://github.com/lutaml/taurus/releases and place the shared library on
your system's library search path, or set
TAURUS_LIB_PATH.Upstream issue tracker
15 issues filed against
lutaml/taurusduring this rewrite (all closed in v0.5.0–v0.5.14):Review checklist
rake.yml) is canon's verbatim — confirmLUTAML_CI_PAT_TOKENsecret is set in repo settingsrelease.yml) is canon's verbatim — confirmLUTAML_CI_RUBYGEMS_API_KEYsecret is settaurus.gemspecspec.versionis0.1.0(fromlib/taurus/version.rb)taurusexecutable ships (CLI was removed; gem installs as library only)extconf.rbruns at install — gem assumes libtaurus is pre-installedlib/libtaurus.dylibis NOT shipped (it's in.gitignore); users on Linux/macOS install libtaurus themselvesAfter merge
To release v0.1.0:
gh workflow run release.yml -f next_version=0.1.0(or via the GitHub UI), OR push av0.1.0tag.Do NOT run
bundle exec rake releasedirectly — it would bypass the release workflow and push to RubyGems without going through the canon reusable-workflow-based pipeline.