Flutter IB JSON API generator - #802
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdded the Merge Risk: 🟡 Moderate · up to The generator can produce incorrect mobile-book content, publish an incomplete API, and—with an overlapping configuration—replace source documentation with generated files. These bounded but concrete correctness and publication risks require fixes or explicit owner acceptance before merge. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 45 functions across 11 files. (2 skipped: 2 unsupported.) ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a new Python-based utils/md2json package that converts the repository’s Jekyll markdown book (docs/) into the view-JSON format consumed by the app, including support for includes/widgets and BibTeX-backed citations.
Changes:
- Introduces a markdown-to-view-JSON parser (
blocks.py+inline.py) and book discovery/navigation builder (book.py). - Adds BibTeX parsing +
{% cite %}/{% bibliography %}handling to preserve citation markers and generate References content. - Adds a CLI entrypoint that regenerates an output tree in one run (
cli.py,__main__.py,output.py).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/md2json/init.py | Exposes public module API and documents the package purpose. |
| utils/md2json/main.py | Supports running the generator as a module or by executing the package directory. |
| utils/md2json/bibliography.py | Parses BibTeX and renders citations/references into plain text for widgets. |
| utils/md2json/blocks.py | Block-level markdown parser that builds the view-document list (widgets/text/TOC). |
| utils/md2json/book.py | Discovers chapters/sections from docs/ and assembles navbar + pages. |
| utils/md2json/cli.py | End-to-end regeneration loop writing navbar + per-page JSON outputs. |
| utils/md2json/config.py | Central configuration for paths, widget mappings, and dropped sections/headings. |
| utils/md2json/frontmatter.py | Extracts flat Jekyll front matter and returns the remaining markdown body. |
| utils/md2json/inline.py | Flattens inline markdown/HTML to plain text expected by widgets. |
| utils/md2json/model.py | Dataclasses for the parsed book/page structure. |
| utils/md2json/output.py | JSON serialization + write-to-disk helper. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: cd67a57c-d40f-4edc-989b-28f21f5ada8c
📒 Files selected for processing (11)
utils/md2json/__init__.pyutils/md2json/__main__.pyutils/md2json/bibliography.pyutils/md2json/blocks.pyutils/md2json/book.pyutils/md2json/cli.pyutils/md2json/config.pyutils/md2json/frontmatter.pyutils/md2json/inline.pyutils/md2json/model.pyutils/md2json/output.py
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.
The generator's widget vocabulary was split across two files: INCLUDE_WIDGETS lived in config.py while nine sub_type strings were literals in blocks.py, so "what does the app render?" could not be answered from one place and renaming a widget meant grepping the parser. - move the nine generator-named sub_types into config.py as WIDGET_* constants - name image.html / chapter_toc.html once in config; both were duplicated as literals in blocks.py, with chapter_toc.html already in STRUCTURAL_INCLUDES - move the "references" heading into config alongside the other heading rules - locate the repo by walking up to _config.yml instead of parents[2], which silently resolved to the wrong tree if the package were moved Also fixes a quiz in docs/logic-design/kmaps.md that mixed tab and 8-space indentation in one list. A tab expands to 4 columns and the spaces to 8, so three options parsed as shallower than their siblings and were lifted to the top level, producing a question with no correct answer plus a phantom question titled "3-variable", and dropping an option from an earlier question. Whitespace only, no wording changed; kramdown resolves indentation the same way, so this should correct the website rendering too. Generated output is byte-for-byte unchanged, verified against a snapshot.
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
utils/md2json/blocks.py (1)
184-193: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winStop paragraph collection before an iframe.
When
handle_paragraph()encounters an iframe line, it appends the line to the paragraph instead of returning control toparse(). The iframe then does not reachhandle_iframe(), so noWIDGET_IMAGEis emitted. Add the same iframe detection used byparse()to the paragraph terminators.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 967eddba-f0ef-4203-b317-d8ff92bd8da3
📒 Files selected for processing (5)
.gitignoredocs/logic-design/kmaps.mdutils/md2json/blocks.pyutils/md2json/cli.pyutils/md2json/config.py
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.
Recovers content the parser was dropping, and makes the output tree safe to
regenerate.
handle_paragraph() did not treat an iframe as a paragraph terminator, so an
iframe that followed prose without a blank line was absorbed into the paragraph
and then stripped as an HTML tag. Twelve embedded CircuitVerse simulators were
missing from the generated JSON as a result; they are now emitted.
collect_list() accepted ordered and unordered markers at the same root level,
so a bullet list followed by a numbered list merged into one widget typed from
the first item. Collection now stops when a root-level marker type changes.
Nested items still mix markers freely, which is how {:.quiz} encodes answers.
CitationRegistry numbered keys it could not resolve but rendered() omitted them,
so an inline [2] could point at the first entry in the list. Unresolved keys now
render a visible placeholder, keeping markers and list positions aligned.
The output tree is generator-owned: it is now built in a staging directory and
swapped in only after a successful run, so a removed or renumbered section
cannot leave stale JSON and a failure part-way through cannot leave a partial
tree.
Also:
- _find_repo_root() fell back to parents[2], one level above the repository,
since it is passed utils/md2json rather than the config file itself
- write_document() raised ValueError labelling any path outside OUTPUT_PATH
- dropped the "1. TOC" placeholder guard, which never fired on any page because
the dropped-section handling already removes it, and could only misfire on a
legitimate single-item list
- fixed a docstring opening with four quote characters
- documented the remaining undocumented functions (45/45)
Verified against a pre-change snapshot: the only content differences are the
twelve recovered iframe widgets.
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2bc97d75-c831-404f-b2b6-e6c3c54e2f02
📒 Files selected for processing (6)
utils/md2json/bibliography.pyutils/md2json/blocks.pyutils/md2json/book.pyutils/md2json/cli.pyutils/md2json/config.pyutils/md2json/output.py
🚧 Files skipped from review as they are similar to previous changes (2)
- utils/md2json/book.py
- utils/md2json/blocks.py
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.
The previous swap deleted the existing output before moving the staging tree into place, so a failure in that move left no output at all, and concurrent readers could observe the path as absent between the two operations. Move the existing tree to a sibling backup instead of deleting it, then swap the staging tree in, then discard the backup. If the swap fails the backup is restored. If both the swap and the restore fail the backup is kept rather than cleaned up, since it then holds the only copy, and its path is reported. Verified by fault injection: a failure during generation leaves the tree untouched, a failure during the swap restores all 57 files, and a failure of both leaves a named backup behind. No staging or backup directories are left in any case.
The generator wrote to a directory that never reached the deploy, so the JSON was only ever available locally. The existing page API is published by building the site into out/ and having utils/api_generator.py write out/_api before peaceiris/actions-gh-pages deploys out/ to GitHub Pages. Serve the book API the same way. - write to out/api/ instead of a top-level directory, so the deploy publishes it - run the generator in the deploy workflow, after the jekyll build, since jekyll clears its destination directory - refuse to generate into the repository root or any ancestor of docs/, as the output tree is now inside the build output and is replaced wholesale Endpoints become <site>/api/navbar.json, <site>/api/about.json and <site>/api/<chapter>/<n>.json, alongside the existing <site>/_api/pages/. Verified in deploy order: with out/ already populated by a build, the run adds out/api/ with all 57 documents, leaves the rest of the site untouched, and leaves no staging or backup directories that would be published.
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 500d9092-2153-4477-98c9-32398af1250d
📒 Files selected for processing (4)
.github/workflows/deploy.yml.gitignoreutils/md2json/cli.pyutils/md2json/config.py
💤 Files with no reviewable changes (1)
- .gitignore
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.
The site published two APIs. utils/api_generator.py crawled jekyll-admin's endpoint to mirror out/_api, serving each page as Jekyll-rendered HTML inside a JSON envelope. utils/md2json now publishes out/api, serving the same pages as a structured view model. The app consumes the structured form, so the markdown API is retired. Removing it also simplifies the deploy. Generating _api required booting a detached jekyll server, crawling it over HTTP with an unqualified `sudo python`, then pkill-ing the server: three steps, a background process and a second interpreter invocation. The generator reads docs/ directly, so the workflow drops to build, generate, deploy. jekyll-admin stays in the Gemfile: it is in the :jekyll_plugins group and still powers the local /admin editing UI for contributors. BREAKING CHANGE: https://learn.circuitverse.org/_api/ is no longer published. Consumers should move to https://learn.circuitverse.org/api/, which serves navbar.json, about.json, guidelines.json and <chapter>/<n>.json.
navbar.json identified each chapter by id and display name only, while its documents are generated into a slug directory. As a local bundle a client could carry its own ordered list of directories; as a remote API there was no way to build a URL from the navbar at all, and any change to a chapter's nav_order would silently repoint a hardcoded list at the wrong chapter. Each chapter now carries the directory it was generated into, so a client can request "<base>/<path>/0.json" for a chapter index and "<base>/<path>/<id>.json" for a section. The field is additive; existing consumers ignore it. Verified that every URL derivable from navbar.json resolves to valid JSON, and that no generated document is unreachable from it.
The guard rejected only paths strictly above docs/, so OUTPUT_PATH == DOCS_PATH passed it. The swap would then move docs/ aside, write the generated JSON in its place, and delete the backup once the swap succeeded, destroying the book source. A path inside docs/ was likewise allowed and would have replaced a chapter directory. Reject overlap in either direction, equal to a source tree, containing it, or inside it, and resolve the paths first so a relative path or symlink cannot slip past. _bibliography/ is protected alongside docs/, since it carries the same risk. The repository root is covered by the containment check. Verified: out/api generates normally, while docs/, docs/api, docs/logic-design, _bibliography/, the repository root, / and out/../docs are all refused.
The workflow file is stored with CRLF. Editing it rewrote the whole file with LF, so the diff showed all 34 lines as changed rather than just the three removed steps. No content change.
Summary
Replaces the markdown page API with a structured JSON API generated from the
Jekyll book, for the Interactive Book Flutter app.
The book is authored once in
docs/as kramdown markdown and served two ways:Jekyll renders the website, and
utils/md2jsonrenders the same source into aflat view model the mobile client renders natively.
No flags, no arguments. Reads
docs/, rewrites the whole output tree, printswarnings for anything it cannot translate. No third-party dependencies.
Important
Breaking:
https://learn.circuitverse.org/_api/is no longer published.Consumers move to
https://learn.circuitverse.org/api/. Nothing in thisrepository referenced
_api, and the Flutter app — its only known consumer —now uses the structured form. Worth a maintainer confirming no external
consumer remains.
Why replace rather than add
The two APIs answer the same question in different shapes:
_api/(removed)api/(added)viewsarraysub_typenaming a native widget[n]markers plus a reference list<ol>,<table>docs/directlyServing both would mean maintaining two representations of every page. The HTML
form suits a WebView; the app renders natively, so it needs the structured form.
Removing
_apialso simplifies the deploy. Generating it meant booting adetached Jekyll server, crawling it over HTTP with an unqualified
sudo python,then
pkill-ing the server. The workflow is now build, generate, deploy.jekyll-adminstays in the Gemfile: it is in the:jekyll_pluginsgroup andstill powers the local
/adminediting UI.The API
Published to GitHub Pages by the existing deploy job, alongside the site.
57 documents: 10 chapter indexes, 44 sections,
navbar.json,about.json,guidelines.json. Each page is{"name": str, "views": [...]}whereviewsisa flat list a single
ListView.buildercan walk. Markdown nesting iscollapsed deliberately: a tree would only have to be flattened again on device.
{"name": "Registers", "views": [ {"type": "widget", "sub_type": "toc", "items": ["Introduction", "..."]}, {"type": "text", "size": "H1", "content": "Introduction", "scrollToId": 0}, {"type": "text", "size": "H3", "content": "A flip-flop is a 1 bit memory cell..."}, {"type": "widget", "sub_type": "table", "content": {"heading": [], "rows": []}} ]}H1is a section heading,H2a sub-heading,H3body copy. Headings carry ascrollToIdmatching their index in thetocwidget, which drivestap-to-scroll.
navbar.jsoncarries each chapter's directory so a client canbuild URLs without knowing the chapter ordering in advance.
Widgets:
toc,chapter_contents,table,bullet_list,numbered_list,clipboard,image,pop-quiz, and ten interactive simulators mapped from_includes/.How it works
config.pyfrontmatter.pyinline.pyblocks.pybibliography.pymodel.pyPage/Section/Chapterbook.pyoutput.pycli.pyblocks.pyis a single-pass line scanner rather than an AST parser: the outputis flat, and the input is a closed corpus in a known house style. Chapter and
section ordering comes from the same
nav_orderfront matter Jekyll uses, sothe app's navigation matches the website's without a second source of truth.
The output tree is generator-owned. Each run builds into a staging directory and
swaps it in only on success, so a removed or renumbered section cannot leave
stale JSON and a failure part-way through cannot leave a partial tree. The
previous tree is moved aside rather than deleted and restored if the swap fails.
The generator refuses to write to any path overlapping
docs/or_bibliography/.Content recovered
Three classes of Jekyll construct were being dropped. The generator now runs
with zero warnings.
Embedded simulators. An
<iframe>following prose without a blank line wasabsorbed into the paragraph and then stripped as an HTML tag. Twelve embedded
CircuitVerse simulators were missing from the output; they are now emitted.
Citations.
{% cite %}/{% bibliography %}are rendered on the website byjekyll-scholar (
style: ieee-with-url). Unresolved, they left dangling prose —docs/binary-algebra/shannon.mdread as "...can be found in Section 1.9 inand in Section 3.2 in ." All 13 entries in
_bibliography/are now resolved andrendered IEEE style, reusing the existing
numbered_listwidget.Interactive includes.
binary2.html,flipflop2.htmlandapplication1.htmlwere silently dropped and are now mapped to widgets.Chapter contents.
logic-designfiles its{% include chapter_toc.html %}under
## Table of contentswhile the other nine use## Chapter contents, soits index page came out empty. Handled in the generator;
docs/is unchanged.The one
docs/changedocs/logic-design/kmaps.mdmixed tab and 8-space indentation inside one quizlist. A tab expands to 4 columns and the spaces to 8, so three options parsed as
shallower than their siblings and were lifted to the top level. The quiz
rendered as a question with no correct answer plus a phantom question titled
"3-variable", and lost an option from an earlier question.
Two lines, whitespace only, no wording changed. kramdown resolves indentation
the same way, so this should correct the website rendering too. Happy to split
it into its own PR if preferred.
Testing
Against the current
docs/: 57 documents, 10 chapters, 44 sections, 21 quizzes(70 questions), 159 images, 60 tables, 69 code blocks, 0 warnings.
navbar.jsonresolves to valid JSON, with noorphaned documents
tree untouched, a failure during the swap restores it, and a failure of both
leaves a named backup rather than deleting it
docs/,docs/api,_bibliography/, the repositoryroot,
/andout/../docsout/populated by a build, the run addsout/api/and leaves the rest of the site untouchedNotes for reviewers
about.jsonandguidelines.jsoncome fromabout.mdandCONTRIBUTING.mdat the repository root, which Jekyll already builds as ordinary pages.
referenceswidget carrying{text, url}would fix that if wanted.docs/logic-design/kmaps.mdmarks "Entries" as the answer to "___ are knownas diagonal mapping?", which looks wrong on the merits. Left alone: that is a
content decision, not a parsing one.