The module scan defers a manifest's C++ modules to their first require; -log-compile-time prints the startup timeline by borisbat · Pull Request #3971 · GaijinEntertainment/daScript

borisbat · 2026-09-08T14:44:45Z

Behavior change: a .shared_module C++ module in a tree with a warm .das_module.manifest loads at the first require that names it, not on start - a host that enumerates modules (the MCP server) passes -ignore-manifest; a vector<T> of a handled element now registers into the element's module, so a module cache and the JIT DLL cache from an earlier binary re-mint, and its described type name carries that module (ast::dasvector`ptr`Expression, no longer $::...) - code that compares such a name must compare the part after ::.

Why. A warm -jit hello world on an M5 spent 240 ms before user code, and the module scan alone 27 ms of it - every C++ module in the tree was dlopened and its constructor run on every start, whether or not the program required it. Nothing showed where the time went.

What changes.

-log-compile-time prints the process startup timeline (builtin module registration, module scan, initialize, compile, simulate, run, teardown, shutdown) and, per cached module, its cache read time with the macro-context simulate share.
The scan's manifest replay defers a dm row that carries a das-visible module name; the load happens at the first require naming it, through a loader getPrerequisits calls when Module::requireEx misses; a module that cannot initialize, or a row whose own dlopen fails, brings every deferred module in and retries. A parse or a read that no prerequisite walk precedes - the compile() of a string, a deserialized stream, Module::Initialize on a half-warm tree - asks the loader at its own module fetch. The rows and the loader are the scan's: cleared at scan start and at Module::Shutdown.
require ?X target and builtin_module_exists mean "the build has X": linked in, or waiting in a manifest row, which the guard then loads; rtti has_module answers the same question and loads nothing.
The lazy load runs under one gc root with the thread root parked, and every module collects from it afterwards.
vectorHomeModule: a vector<T> of a handled element registers into the element's module whichever module builds it, so a lazy load adds nothing to $ and every module-cache record keeps its builtin hash.
-ignore-manifest reads and writes no manifest; every descriptor compiles and every C++ module loads on start. Manifest read and write sit behind the same DAS_NO_FILEIO guard as the fio module.

Observable behavior.

warm hello on an M5: interpreter 44 ms -> 14 ms, -jit 240 ms -> 211 ms
DAS_TRACE_MODULE_LOAD=1 replay line replayed N row(s) -> replayed N row(s) in <sec> (shared module load <sec>, deferred K), plus [module] require <name>: loading the deferred <class> and [module] loading every deferred module (K)
-log-compile-time -> a startup: timeline and per-module cache read took lines
a stale JIT DLL or module cache from before this change -> re-minted (codegen version 0x77, serializer version 205)

Where to look. load_deferred_module_for_require in src/ast/dyn_modules.cpp (the gc-root dance and the failed-load fallback), the getPrerequisits hook and the collector guard in src/ast/ast_parse.cpp, vectorHomeModule in include/daScript/ast/ast_handle.h.

Validation, claims, ledger

Validation

One preflight --full on macOS: 21 gates green, four red - the module-cache tests (a helper module named _common collided with tests/linq/_common in a batched sweep worker), tests/language/reflection.das (its sample compiles a string that requires UnitTest - no prerequisite walk, so the parser's require now loads), the MCP ast_dump test (the printer compared the $:: vector name), and the AOT gate (the AOT host is a static build). Each fixed and re-validated with its targeted gate: interpreter and JIT suites green, utils-tests green, AOT suite green; the dasLLVM module suite green; das2rst and sphinx -W clean.
Negative-controlled: the deferral, -ignore-manifest, the gc-root parking, the failed-load fallback, the every-module collect, the getPrerequisits hook, the collector guard, builtin_module_exists, has_module, the vector home, the half-warm initialize and the deserializer's fetch each red their arm with the line reverted.
Two codex rounds: round 1 found the failed-dlopen path, round 2 the environment lifecycle of the deferred rows and the per-file timing accumulation; all three fixed with tests where a test can reach them.
Rebased twice over master's merges during the chain (the nightly-lint fix, the leak-dump ordering fix); the tree rebuilt and the module-cache folder and the timeline print re-smoked after each.
CI found three holes a warm developer tree cannot show, each fixed with a test that was red first: the no_fileio lane's unused helpers under the reader's guard; a half-warm tree (parallel AOT batches on a cold checkout warm the manifests concurrently, so one process saw a module loaded on start beside its dependency's waiting row) making Module::Initialize fatal; and the ser/deser sweep, where the deserializer fetched a lazily loaded module by name in a process that had not loaded it. tests/module_cache is 38/38 after them, and the whole-tree --ser/--deser sweep round-trips 1381 programs locally.
The dasImgui/dasGlfw fallback arms (a module whose C++ dependencies are deferred too; the half-warm tree) run only where those shared modules are built - this box and the release-modules CI cells; a tree without them skips them and says so.

Claims - stated, not tested

The DAS_NO_FILEIO arms of read_manifest, write_manifest, the deferred-module registry and its stubs compile: the build_eastl no_fileio lane is a Linux gcc lane, and this PR's CI run of it is the check.
A deferred module whose .shared_module links a sibling (DT_NEEDED) still loads: the failed first dlopen goes to the pending list, the fallback loads every deferred module and retries the pending ones; no tree in CI holds such a pair, so the arm the test exercises is the absent-artifact one.
The parser's own require and guard (ast_requireModule) load a deferred module only when no prerequisite walk preceded the parse - the compile() of a string, pinned by the nested arm and by tests/language/reflection.das; under compileDaScript the collector has loaded it already, so the parser's call there reaches nothing the collector did not.
The inner retry in the loader (a loaded module whose initDependencies fails) is covered only by the dasImgui arm above; without those modules it is unpinned.
-ignore-manifest on the -aot path and in the MCP launchers (the python supervisor, the .cmd): the flag's absence is not caught by a test. The LSP subtools spawn without it - neither validate nor nav enumerates the process's modules, and the flag cost a quarter second per request.
Several Initialize/Shutdown cycles in one host: the deferred rows and the loader clear at shutdown; no in-tree host runs two cycles with a module scan in each.
The handmade has_module page names .das_module.manifest, require, and typeinfo builtin_module_exists(mod) with a bare module name - checked against src/ast/dyn_modules.cpp and src/ast/ast_infer_type.cpp.
Windows: the test's trace_prefix uses the set X=1&& command form the sibling test already relies on.

Not done

require [group] (manifest-declared group membership) and cross-context macro extension are the next arc; the JIT-skip for a hello world waits on the serializer work after them.
An MCP test fixture with a custom -project_root modules/ tree, then the exe form of the MCP server and the LSP subtools again, and the watchdog supervising both - ledgered in utils/mcp/ROADMAP.md and utils/lsp/ROADMAP.md.
Four checklist splits the audit proposed (utils/REVIEW.md, doc/REVIEW.md, modules/dasLLVM/REVIEW.md, doc/source/stdlib/handmade/REVIEW.md) ride a docs-only PR at the end of this chain - plans/review_md_splits.md.

… registration / module scan / initialize / compile / simulate / run / teardown / shutdown) and, per cached module, its cache read time with the macro-context simulate share, plus a cache read total in the compile summary - the serializer's totMacroTime was accumulated and never read, so a warm -jit hello world (237 ms on the M5) had no way to show that 154 of its 178 ms compile reads 61 cached modules, 92 of that simulating macro-module contexts (llvm_macro alone 49: its context is the whole emitter, simulated to run a 1 ms DLL cache hit) and 12 reparsing llvm_func in place every run; DAS_TRACE_MODULE_LOAD replay lines carry their time and the shared-module load share (dasVulkan's dlopen is 14 of the scan's 27 ms); test_descriptor_manifest's probe_line strips the timing clause

…uire that names it, not on start - a replayed dm row carrying the das-visible name the recording start learned waits (defer_dynamic_module), and the prerequisite walk's miss asks the loader the scan installed (setDeferredModuleLoader), which dlopens, runs the factored initDependencies fixed point (Module::InitializeDependencies) and, when a module reports it needs another deferred one, brings the whole deferred set in and runs it again; the load runs under one gc root with the thread root's own nodes parked, since a constructor's builtin das module dumps its leftovers on the thread root and TypeDecl::gc_collect stops at a node owned elsewhere, and the collect walks every module, since a constructor registers into existing ones too; a `require ?mod` and builtin_module_exists mean "a require of this compile named mod" (a scan-loaded module is unrequired until a require resolves to it, so a cold start answers as a warm one), a plain-name guard rides the RequireRecord and is tested at its line, compileDaScript walks again while a skipped guard's module got loaded so order never matters, and the parser reads the walk's verdict (walkedGuardVerdict) instead of re-testing; the daslang host names the llvm witness under -jit and -exe, since no das file requires it unguarded and a static host runs the JIT without it; -ignore-manifest keeps the eager start for the MCP server and the LSP subtools, which enumerate; has_module answers loaded-or-deferred so the sweep gates keep their answer; a vector of a module's own handled type registers into that module (vectorHomeModule), since library.front() is always `$` and a load must not move the builtin module's cumulative hash under the module cache, and LLVM_JIT_CODEGEN_VERSION goes 0x76 because the externs a DLL binds by name moved; manifest read and write are guarded by DAS_NO_FILEIO like the fio builtins; hello world on the M5: interpreter 44 -> 14 ms, -jit 240 -> 211; tests/module_cache/test_deferred_modules.das covers cold, warm, every guard order, -ignore-manifest and the fallback on a copied dasUnitTest that shadows the tree's

…odule_exists ask whether the build has X (guardModuleAvailable: linked in, or a manifest row the loader brings in now), so `require ?das_metal metal/das_metal_boost` keeps meaning "on a build with Metal" and dasLLAMA's GPU tier, tests/metal and daslib/tune's llvm arms reach their modules as they did on an eager start, which the sweeps could not show because every such body sits inside a static_if; the "required earlier" reading needed an unguarded require somewhere for each witness, which one source that must also compile without the module cannot carry, and it leaves with the unrequired set, the RequireRecord guard field, the walk's fixed point, the parser's verdict map and the host naming llvm under -jit

…efore-the-walk ordering is a duty in src/ast/REVIEW.md, the load's single-root gc mechanism is a sentence of src/ast/ARCHITECTURE.md sec.2, the rest are one-line pointers to that section or gone

… in and retries the pending ones before its require fails, and a module set that grew runs the initDependencies fixed point and the collect whether or not the named module came in - the fallback skipped both, so the sweep freed nodes the constructors had just registered (SIGBUS on a manifest row naming an absent artifact); the serializer version is 205 because a vector of a handled element streams under the element's module; the src/ast gate scans the cache-read timing line outside the reader for its log_module_compile_time guard; test_descriptor_manifest's probe_line keeps the deferred count; checklist self-review fixes in src/ast, src/builtin, src/parser, modules/dasLLVM and tests/module_cache

…e five tests carried verbatim - das_exe, trace_prefix, run_child and report_child - a test whose child needs another spawn shape keeps that helper local; utils/mcp's install-block rule binds every top-level file the shipped SDK runs, not only .das; utils/lsp's README says the subtools spawn with -ignore-manifest

…ynamic_modules clears the rows before its walk and Module::Shutdown clears both, so a host that runs several Initialize/Shutdown cycles neither replays the last environment's rows twice nor answers a guard from a tree it no longer has; -log-compile-time's compile, simulate and run laps accumulate across the input files the way the total does, so the teardown residual no longer swallows every file but the last; has_module's page names the manifest file and the bare-name form of builtin_module_exists

…ule_exists loads a waiting module with no require naming it, rtti has_module answers true for it and loads nothing, a lazy start and an eager start count the same functions in $ (the vector-home invariant, red at 2485 vs 2501 with it reverted), and a module cache an eager start wrote serves a lazy start with -log-compile-time printing the reads and the timeline; has_deferred_dynamic_modules had no caller and is gone

…ts a deferred module at the parser's own require and loads it there, and the loader puts the parse's program back as the bound one, since a module's builtin das part parses under a program of its own (tests/language/reflection's sample requires UnitTest through compile()); the module-cache spawn helpers live in _mc_common - a sweep worker keeps every shared module it met under one name and tests/linq owns _common, so the five tests resolved to the wrong module there

…the part after '::' - a vector of a handled element now describes under the element's module (ast::dasvector`ptr`Expression), so the $:: comparison printed every list empty and the MCP ast_dump test lost its ExprReturn

…s descriptors register no shared module, so nothing is deferred there and the arms had nothing to observe

…e process's modules, so -ignore-manifest cost every keystroke a quarter second for nothing (the MCP server keeps it); the vector-home sentence claims what holds and is tested - a load adds nothing to $, and a vector's home may be a module that exists already; the dasLLVM codegen-bump rule's criterion is the outcome again with the surfaces as its content; utils/REVIEW.md names the two CMakeLists.txt its gate reads; the module-cache weakening rule names the timing as the field that leaves the compared text

…klists carried it where it could never fire, since its trigger is utils/CMakeLists.txt; utils/lsp/REVIEW.md had nothing else and is gone; the gate skips dot-directories, so a JIT's .jitted_scripts cache under utils/ is not read as a tool; the MCP and LSP roadmaps ledger the custom-modules fixture, the return to an exe form, and the watchdog supervising both

…live: include/daScript/ast (a bind's home module or name - vectorHomeModule, the vector functions, an annotation's name), src/ast (how a mangled name forms, which module addFunction files a builtin under) and src/builtin (a bind moved between modules) - the JIT's DLL cache key folds the codegen version and each function's AST hash, never the module an extern lives in, so without the bump a cached DLL binds the old name and crashes on the hit

…s proposed - utils, doc, dasLLVM, handmade - for a docs-only PR at the end of the startup chain

Copilot

🔵 Needs a closer look

It changes core module-loading and GC-root behavior across startup/require paths and introduces new global deferred-module state that warrants careful human validation (incl. concurrency considerations).

Pull request overview

This PR changes daslang’s dynamic module startup behavior to defer loading C++ .shared_module artifacts recorded in warm .das_module.manifest files until the first require that names them, and extends -log-compile-time to report a process-level startup timeline plus module-cache read timings. It also adjusts builtin type registration so handled-element vector<T> annotations register into the element’s owning module (stabilizing caches), and updates tests/docs/tools to match the new semantics.

Changes:

Defer manifest-replayed C++ module loading to first require (with fallback “load all deferred” on failure) and ensure prerequisite walks / parser require can trigger the lazy load.
Add -ignore-manifest (tooling mode) and expand -log-compile-time to include startup timeline and cache-read timing output.
Update module-cache + JIT cache versioning, tests, and docs for deferred-module behavior and new type-name/module ownership.

File summaries

File	Description
utils/REVIEW.md	Clarifies which CMake files define tool membership/shipping.
utils/REVIEW.das	Adds a gate banning supervised tools from being built/shipped as exes; skips dot-directories in dir listing.
utils/mcp/ROADMAP.md	Documents follow-ups for manifest/eager-load fixture and future exe form.
utils/mcp/REVIEW.md	Tightens shipped-SDK top-level file install rule wording.
utils/mcp/README.md	Requires `-ignore-manifest` in MCP server launch instructions and explains why.
utils/mcp/mcp_supervisor.py	Adds `-ignore-manifest` to default launcher and generated `.mcp.json` entries.
utils/mcp/daslang-mcp-msvc.cmd	Adds `-ignore-manifest` to Windows launcher paths.
utils/lsp/ROADMAP.md	Notes watchdog follow-up enabling future exe form again.
utils/lsp/REVIEW.md	Removes now-redundant local checklist (supervised-tool exe ban moved to utils gate).
utils/daslang/main.cpp	Adds `-ignore-manifest` flag handling and startup timeline timing for `-log-compile-time`.
tests/module_cache/test_macro_dep_invalidate.das	Switches to shared spawn helpers (`_mc_common`).
tests/module_cache/test_generic_instance_origin.das	Switches to shared spawn helpers (`_mc_common`).
tests/module_cache/test_descriptor_manifest.das	Updates trace parsing/expectations to accommodate new replay timing/deferred fields and shared helpers.
tests/module_cache/test_deferred_modules.das	New test covering deferred manifest C++ module load behavior, guards, fallback path, and cache/timeline output.
tests/module_cache/test_default_cache_path.das	Switches to shared spawn helpers (`_mc_common`).
tests/module_cache/REVIEW.md	Tightens “don’t weaken output comparisons” rule to cover helper-produced text and ignore new timing fields.
tests/module_cache/ARCHITECTURE.md	Documents new deferred-modules test and `_mc_common.das` helper.
tests/module_cache/_mc_common.das	New shared module providing child spawn helpers and failure reporting for module-cache tests.
src/parser/REVIEW.md	Adds requirement to extend grammar canary test when parser accepts new syntax.
src/parser/parser_impl.cpp	Makes optional-guard availability match deferred-module semantics; loads deferred module in parser-only require when needed.
src/builtin/REVIEW.md	Updates checklist to reflect broader “bytes/meaning change” version bump rules and JIT codegen version bump triggers.
src/builtin/module_builtin_rtti.cpp	Updates `has_module` to consider deferred manifest rows without loading them.
src/builtin/module_builtin_fio.cpp	Adds deferred-module registry and loader helpers (defer/load/load-all/is/clear).
src/ast/REVIEW.md	Updates checklist for cache-read diagnostics + deferred-loader invariants and JIT version bump triggers.
src/ast/REVIEW.das	Generalizes gate to ensure cache-read diagnostics are guarded by `quietCache` or timing flag.
src/ast/dyn_modules.cpp	Implements manifest ignore flag, replay timing/deferred stats, and installs deferred-loader + GC-root parking load path.
src/ast/ast_parse.cpp	Adds deferred-loader hook when prerequisite walk/parser require sees missing module; reports cache-read timing and macro-sim share.
src/ast/ast_module.cpp	Extracts `InitializeDependencies`, introduces deferred-loader plumbing, and ensures shutdown clears deferred state/loader.
src/ast/ast_infer_type.cpp	Makes `typeinfo builtin_module_exists` consider deferred modules via `guardModuleAvailable`.
src/ast/ARCHITECTURE.md	Documents deferred module mechanics, guard semantics, GC-root handling, and type/module ownership changes.
skills/mcp_tools.md	Updates MCP configuration guidance to include `-ignore-manifest`.
skills/internal/environment_variables.md	Updates `DAS_TRACE_MODULE_LOAD` description for new replay/defer logging.
skills/dynamic_modules.md	Documents deferred C++ load behavior, guard meaning, and tooling `-ignore-manifest` mode.
plans/review_md_splits.md	Adds plan document for future checklist splits (docs-only follow-up).
modules/dasLLVM/REVIEW.md	Clarifies when LLVM JIT codegen version must be bumped.
modules/dasLLVM/daslib/llvm_jit_run.das	Bumps `LLVM_JIT_CODEGEN_VERSION` for moved extern ownership.
include/daScript/ast/REVIEW.md	New checklist covering module/name ownership changes requiring JIT codegen version bumps.
include/daScript/ast/dyn_modules.h	Exposes deferred-module APIs and `ignore_dynamic_module_manifests`.
include/daScript/ast/ast.h	Adds deferred-loader API and `InitializeDependencies` declaration.
include/daScript/ast/ast_serializer.h	Bumps serializer version for module-cache format/meaning change.
include/daScript/ast/ast_handle.h	Adds `vectorHomeModule` so handled-element vectors register into the element’s module.
doc/source/stdlib/handmade/function-rtti-has_module-0x2f9e9a6e19be1ef0.rst	Updates `has_module` documentation to include deferred-manifest semantics and non-loading behavior.
doc/REVIEW.md	Broadens checklist wording for “things readers are told to fetch/run/type”.
daslib/ast_boost.das	Updates AST debug printer to match module-qualified vector type names (suffix match).

Review details

Files reviewed: 44/44 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…it under the same file-io guard as the reader, so a build with no file io does not carry functions nothing calls (the no_fileio lane builds with -Werror=unused-function)

… load, load-all, the deferred query and the clear - a run-time has_module on one thread reads the list while another thread's compile loads from it

Copilot

🔵 Needs a closer look

It modifies core module loading/initialization, manifest replay behavior, and GC-root handling in ways that are performance- and correctness-critical and warrant final human review.

Review details

Files reviewed: 44/44 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

🔵 Needs a closer look

It changes core module loading/initialization semantics across runtime, parser, and caching paths, which is high-impact and warrants final human review despite added tests and documentation.

Review details

Files reviewed: 44/44 changed files
Comments generated: 0 new
Review effort level: Lite

…, so its module loaded on start, beside replayed ones whose rows wait - brings every deferred module in when the eager fixed point fails and runs it once more; two processes warming one tree make it half-warm, which the parallel AOT batches of a cold CI checkout did (Linux Debug: 'Unable to initialize some modules: imgui_app'); the test copies dasImgui and dasGlfw under its fixture and removes the imgui copy's manifest

Copilot

🟡 Changes recommended

There are a few correctness/API issues in the new deferred-module plumbing (notably a missing nullptr guard and output-parameter hygiene) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/ast/ast_module.cpp:203

Typo/grammar in a user-facing fatal error: "No modules founds. You should add modules before call that function." This reads poorly and is easy to fix while this code is being touched.
utils/REVIEW.das:184
check_supervised_tools only checks utils/CMakeLists.txt, but the surrounding rules (and this function’s intent) treat both utils/CMakeLists.txt and the repo root CMakeLists.txt as places a tool could be built/shipped. If mcp/lsp are ever added via the root CMake, this gate would miss it.

Consider scanning both files and attributing the finding to whichever file contains the offending list entry.

Files reviewed: 44/44 changed files
Comments generated: 2
Review effort level: Lite

…ferred query does; InitializeDependencies clears the names it reports before it runs, so a caller reusing the string never reads a stale one - the two callers' own clears go

Copilot

🔵 Needs a closer look

It modifies core module loading/initialization and GC-rooting behavior across multiple subsystems, which warrants final human review despite strong test/documentation updates.

Review details

Files reviewed: 44/44 changed files
Comments generated: 0 new
Review effort level: Lite

…e does - a stream written where a C++ module had loaded lazily is read where nothing required it yet, so Program::serialize's builtin list and the module-cache record header ask the deferred loader when Module::require answers null (the darwin modules lane's ser/deser sweep: program 123 referenced sqlite, loaded by a require ?sqlite guard in an earlier test, and the reader silently skipped a null module); the test writes a dastest --ser stream of a test requiring UnitTest and reads it with --deser in a child that nothing made require it

Copilot

🔵 Needs a closer look

It changes core module loading/initialization and cache-key behavior across runtime, tooling, and tests, which warrants final human review despite the added coverage.

Review details

Files reviewed: 45/45 changed files
Comments generated: 0 new
Review effort level: Lite

…long, and where the copied imguiApp still cannot load - its libraries sit relative to the tree, as on the darwin CI box - the arm says so and returns instead of asserting on a state it never reached

Copilot

🔵 Needs a closer look

It changes core dynamic-module loading, initialization, GC/root handling, and cache/JIT invalidation behavior across many subsystems and warrants final human review plus broad CI/runtime validation.

Review details

Files reviewed: 45/45 changed files
Comments generated: 0 new
Review effort level: Lite

…re the copied imguiApp fails to dlopen there, initialize had nothing to bring in and the arm says so with the child's output, instead of asserting on the line that start could never print

Copilot

🔵 Needs a closer look

It changes core module-loading and initialization behavior (manifest replay, deferred dlopen, GC/rooting, and cache/version semantics) across multiple subsystems, so it warrants final human review despite strong test/doc updates.

Review details

Files reviewed: 45/45 changed files
Comments generated: 0 new
Review effort level: Lite

borisbat added 15 commits September 8, 2026 07:41

the diff's added C++ comments settle into the documents: the loader-b…

055bbf5

…efore-the-walk ordering is a duty in src/ast/REVIEW.md, the load's single-root gc mechanism is a sentence of src/ast/ARCHITECTURE.md sec.2, the rest are one-line pointers to that section or gone

test_deferred_modules returns on a static host - the AOT test binary'…

f9cdc93

…s descriptors register no shared module, so nothing is deferred there and the arms had nothing to observe

plans/review_md_splits.md ledgers the four checklist splits the audit…

f9c0fe3

…s proposed - utils, doc, dasLLVM, handmade - for a docs-only PR at the end of the startup chain

Copilot AI lite review requested due to automatic review settings September 8, 2026 14:44

Copilot started reviewing on behalf of borisbat September 8, 2026 14:45 View session