From 5b3c1622f8ba77924fd03dd41ee438d903cd0ffe Mon Sep 17 00:00:00 2001 From: Churkin Aleksey Date: Wed, 9 Sep 2026 02:00:12 +0300 Subject: [PATCH 1/4] daslib: a standalone context speaks C, and C++ through it [skip ci] A standalone context spoke C++ only, through methods on its generated class, and its header was written in the same interleaved pass as its source - two nested build_string blocks, every emitter writing into both writers, nothing able to produce a header on its own. A C host had nothing to include. One module writes the header now, and the C API is the only calling path: daslib/c_api_header.das the whole generated header, C first CppApi the C++ half, appended under #ifdef __cplusplus Each C++ function is an `inline` call to the C entry point declared above it, so a C host and a C++ host include the same file, neither sees the other's half, and one implementation carries both. That is also why the module needs no aot_cpp: a proxy only ever spells a signature C can spell, and for those the C++ form is mechanical - a struct or enum by its das name, everything else by its C one. aot_standalone emits the matching extern "C" bodies from the same describer, so the header and the source cannot disagree about what crosses. The C++ methods stay methods, and their bodies are now the call: `ctx.dot(a,b)` reaches the same extern "C" entry point a C host calls. So no host changed - the examples, the nano and standalone context tests, the tutorial and the watchdog all keep master's text, and test_standalone_emit passes unmodified. Selection is one bit, Function.flags.exports, which [export] and the new [export_c] both set - so the C API takes what the C++ one takes, which is what makes the proxy total. An export C cannot spell is skipped with a warning naming the type; one that ASKED for C with [export_c] and cannot cross is an error. Its name = "..." override renames both APIs at once, so a host calls one name. Two things link found rather than reading: the extern "C" block sits inside namespace das, because describeCppType spells a type relative to it and C linkage survives a namespace anyway; and the C surface spells an enum as its base integer and an opaque pointee as void *, which C++ converts back from neither on its own - so every such argument is cast at the boundary. CTypeSet.structs is borrowed from the program, so it is @do_not_delete - the set's teardown was freeing the AST's own Structures. tests-cpp/big/standalone_ctx/test_standalone_capi.c is the proof: the same fixture the C++ twin drives, through the C API, with the generated header compiled by a C compiler. A header that is not valid C, a layout that disagrees with das, or a thunk that loses a value fails there - not in a string compare. --- daslib/ARCHITECTURE.md | 3 +- daslib/ARCHITECTURE_CAPI.md | 60 ++ daslib/aot_standalone.das | 198 ++-- daslib/ast_boost.das | 12 + daslib/c_api_header.das | 977 ++++++++++++++++++ daslib/export_c.das | 49 + doc/source/reference/embedding/advanced.rst | 55 + doc/source/reference/language/annotations.rst | 34 + examples/standalone/CMakeLists.txt | 2 + skills/cpp_integration.md | 43 + .../daslang/references/modules-and-stdlib.md | 5 +- tests-cpp/big/nano_ctx/CMakeLists.txt | 1 + tests-cpp/big/standalone_ctx/CMakeLists.txt | 24 + .../standalone_init_fixture.das | 6 + .../big/standalone_ctx/test_standalone_capi.c | 65 ++ utils/CMakeLists.txt | 1 + 16 files changed, 1424 insertions(+), 111 deletions(-) create mode 100644 daslib/ARCHITECTURE_CAPI.md create mode 100644 daslib/c_api_header.das create mode 100644 daslib/export_c.das create mode 100644 tests-cpp/big/standalone_ctx/test_standalone_capi.c diff --git a/daslib/ARCHITECTURE.md b/daslib/ARCHITECTURE.md index 9dba5c40eb..86b61fb306 100644 --- a/daslib/ARCHITECTURE.md +++ b/daslib/ARCHITECTURE.md @@ -3,10 +3,11 @@ Design rationale a maintainer cannot recover from the code alone. One numbered section per module; entries are anchored to symbols. -Three companions carry a concern each; a section number is unique across all four files. +Four companions carry a concern each; a section number is unique across all five files. - `ARCHITECTURE_LINT.md` - sec. 1-4: perf_lint, lint_config, lint, style_lint. - `ARCHITECTURE_EMIT.md` - sec. 5-7, 28-29: aot_cpp, aot_standalone, flatten, the shader rails. +- `ARCHITECTURE_CAPI.md` - sec. 30: c_api_header, the C surface both backends emit. - `ARCHITECTURE_LINQ.md` - sec. 11-17, 33, 37: the linq family, sql_linq, sql_migrate. ## 8. ast_verify diff --git a/daslib/ARCHITECTURE_CAPI.md b/daslib/ARCHITECTURE_CAPI.md new file mode 100644 index 0000000000..6f26849ef1 --- /dev/null +++ b/daslib/ARCHITECTURE_CAPI.md @@ -0,0 +1,60 @@ +# daslib architecture notes - the generated C API + +Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across the family. +The header this section describes is the one a standalone context ships. + +## 30. c_api_header + +- **One module writes every generated header, and the C API is the only calling path.** The C + declarations always come first; the C++ half (`CppApi`) is appended under `#ifdef __cplusplus`, + and each of its functions is an `inline` call to the C entry point beside it - so a C host and a + C++ host include one file, neither sees the other's half, and there is exactly one + implementation to keep correct, and no `aot_cpp` dependency: a proxy only ever spells a + C-representable signature, whose C++ form is mechanical - a struct or enum by its das name, + everything else by its C one. `aot_standalone` emits the matching `extern "C"` bodies off + this describer, so the header and the source cannot disagree about what crosses or what it is + called. +- **`Function.flags.exports` is the whole selection truth.** `[export_c]` (`ExportCAnnotation`, + `daslib/export_c.das` - a das `[function_macro]`, so the C surface is decided entirely in + daslang) sets it, and so does `[export]`, so a standalone context takes the bit however it was + set. Accepting the bit however it was set is what licenses skipping an unspellable signature with + a warning; a function that ASKED for C with `[export_c]` and cannot cross is an error. No second + list can drift from the bit. Whether a signature CAN cross is decided here instead, after infer, + because argument types do not exist when an annotation applies. +- **Refusal is per stage, not per module**: `collect_c_exports` returns its rejections and + logs its skips. An `[export_c]` that cannot cross comes back for the caller to report - + `macro_error` during compilation, the jit error log during codegen - so this module needs + no `ProgramPtr` and no reporting policy of its own. A merely-public function is skipped + with a warning naming it and the type. +- **The scalar widths and the vector layouts are C++-side facts this emitter mirrors.** + `bool` is one byte (the `static_assert(sizeof(bool)==1)` in `getTypeBaseSize`, + `src/simulate/debug_info.cpp`), so das `bool` meets C as `bool`. `float3` is `{x, y, z}` at + 12 bytes and 4-byte alignment, because `vec3` (`include/daScript/misc/vectypes.h`) + is a plain three-field struct with no `alignas` - the 16-byte vec4f shape is the JIT's + register ABI, not the memory layout a header has to mirror. So no vector or structure carries + an alignment attribute, and every declared struct carries a size assert plus one offset assert + per field. `Structure.sizeOf` is already rounded to the struct's alignment + (`Structure::getSizeOf`, `src/ast/ast.cpp`), so `sizeof` in C matches it directly. +- **A bound value type crosses as the wrap type its annotation carries.** A + `ManagedValueAnnotation` is not a ref type and holds `makeValueType()` - the `WrapType::type` + das moves the value through, reachable from das as `get_underlying_value_type`. The header + typedefs that shape under the das type's name and the body assigns the das type through it; a C++ + host still gets the real type, because the C++ half carries the module's `aotRequire` include. It + is the one C type with an alignment attribute - a handle's alignment is not its wrap type's + (`BigEntityId` is 16-aligned, four floats are 4-aligned) - on ONE declarator, since the attribute + applies per declarator. Size and alignment are both asserted. A ref-type handle stays `void *`. +- **An enumeration is a typedef of its base integer plus loose enumerators, never a C + `enum`** - a C enum's underlying type is implementation-defined, which would break the + size assert on the 8/16/64-bit bases and on negative values. The values are read off the + entry's folded constant, so no smart pointer is needed to reach `find_enum_value`. +- **Types are emitted only when a signature reaches them, in post-order.** A by-value + field's structure is defined before the structure holding it; every structure also gets a + forward typedef ahead of all definitions, which is what lets a self-referential + (`Node?`) field compile. A pointer's target only has to be NAMEABLE, so one whose fields C + cannot spell is forward-declared and never DEFINED - defining it emits fields with no type at + all - while a representable target is defined, so a host can read through the pointer. A cycle + terminates either way, and a pointer to something C cannot even name degrades to `void *` + rather than refusing the function. +- **A `fixed_array` argument crosses (as `const T *`, which is what the das ABI already + passes) but a `fixed_array` result does not** - that would be a CMRES of an array, a + pointer nothing on the C side sizes. diff --git a/daslib/aot_standalone.das b/daslib/aot_standalone.das index c24d064760..a5ef5653dc 100644 --- a/daslib/aot_standalone.das +++ b/daslib/aot_standalone.das @@ -10,67 +10,15 @@ require daslib/match require daslib/strings_boost require daslib/ast_boost require daslib/templates_boost -require daslib/functional require daslib/ast_print_flags require daslib/aot_constants require daslib/aot_cpp +require daslib/c_api_header options strict_smart_pointers = false -struct StandaloneContextCfg { - //! Configuration for standalone context generation. - context_name : string; - class_name : string; - cpp_output_dir : string; - cross_platform : bool; - //! the context links C++ modules beyond the builtin one, so its constructor registers them - registers_modules : bool -}; - -def aotFunctionName(str : string) { - return replace(str, "`", "__") -} - -def writeStandaloneContextMethods(var prog : ProgramPtr; var logs : StringBuilderWriter; prefix : string; declare_only : bool; cfg : StandaloneContextCfg) { - let fnn = collectProgramUsedFunctions(prog, false, false); - - var coll = new BlockVariableCollector(); - - for (fn in fnn) { - if (!fn.flags.exports || fn._module != prog.getThisModule) continue; - if (declare_only) { - write(logs, " "); - } - write(logs, "auto {prefix}{aotFunctionName(string(fn.origin != null ? fn.origin.name : fn.name))} ( "); - var vars : array - vars |> reserve(length(fn.arguments)) - for (variable in fn.arguments) { - let type_str = build_string() $(var wr) { - if (isLocalVec(variable._type)) { - describeLocalCppType(unsafe(addr(wr)), variable._type, cfg.cross_platform); - } else { - write(wr, describeCppType(variable._type, DescribeConfig(cross_platform = cfg.cross_platform))); - } - } - vars.push("{type_str} {variable._type.isRefType ? "& " : ""}{coll.getVarName(variable)}") - } - let vars_str = join(vars, ", "); - write(logs, "{vars_str}{!fn.arguments.empty() ? " " : ""}) -> ") - describeLocalCppType(unsafe(addr(logs)), fn.result, cfg.cross_platform, CpptSubstitureRef.no, CpptSkipConst.yes); - if (declare_only) { - write(logs, ";\n"); - } else { - let args_str = (fn.arguments |> each() - |> map(@(v : VariablePtr) { return coll.getVarName(v); }) - |> join(", ")) - let maybe_comma = !empty(args_str) ? ", " : ""; - write(logs, " \{\n return {aotFuncName(fn)}(this{maybe_comma}{args_str});\n\}\n\n"); - } - } -} - def collectInitFunctionAotNames(program : ProgramPtr; var context : Context) : array { //! In the simulated context's run order (late init last). var initMnhOrder : array @@ -143,6 +91,7 @@ def private makeGlobalVarInfos(var helper : AotDebugInfoHelper?; globals : array def private writeGlobalVarInfos(var helper : AotDebugInfoHelper?; infos : array>; var tw : StringBuilderWriter) { for (entry in infos) { let suffix = "_gvar_{entry.index}" + helper->registerHandledAnnotation(unsafe(addr(tw)), entry.info) helper->writeDim(unsafe(addr(tw)), entry.info, suffix) helper->writeArgTypes(unsafe(addr(tw)), entry.info, suffix) helper->writeArgNames(unsafe(addr(tw)), entry.info, suffix) @@ -304,19 +253,7 @@ def writeStandaloneCtor(cfg : StandaloneContextCfg; initFunctions : string; var write(tw, "\}\n"); } -def writeStandaloneContext(var program : ProgramPtr, initFunctions : string, var header : StringBuilderWriter, var source : StringBuilderWriter; cfg : StandaloneContextCfg; globals : array; var context : Context) { - - write(header, "\n\n"); - //! the module scope is the FIRST base: constructed before Context, destroyed after it, so the - //! registry outlives the context's own teardown - let bases = cfg.registers_modules ? "public StandaloneModuleScope, public Context" : "public Context" - write(header, "class {cfg.class_name} : {bases} \{\n"); - write(header, "public:\n"); - write(header, " {cfg.class_name}();\n"); - writeStandaloneContextMethods(program, header, "", true, cfg); - write(header, "\};\n"); - - writeStandaloneContextMethods(program, source, "{cfg.class_name}::", false, cfg); +def writeStandaloneContext(var program : ProgramPtr, initFunctions : string, var source : StringBuilderWriter; cfg : StandaloneContextCfg; globals : array; var context : Context) { writeStandaloneCtor(cfg, initFunctions, source, program, globals, context); } @@ -377,11 +314,8 @@ class StandaloneContextGen : CppAot { }; -def writeModuleDeclarations(var header, source : StringBuilderWriter; registrations : array) { - //! the header carries the registry include (the class derives from its scope base); the - //! source carries the module declarations its registration table takes addresses of +def writeModuleDeclarations(var source : StringBuilderWriter; registrations : array) { if (empty(registrations)) return - write(header, "#include \"daScript/simulate/standalone_modules.h\"\n"); write(source, "#include \"daScript/simulate/standalone_modules.h\"\n"); for (entry in registrations) { write(source, "DECLARE_MODULE({entry.mod.cppClassName});\n"); @@ -401,8 +335,7 @@ def writeModuleRegistration(var source : StringBuilderWriter; registrations : ar write(source, "static const bool das_standalone_modules_added = standaloneAddModules(das_standalone_modules, {length(registrations)});\n\n"); } -def writeRegistration(var header : StringBuilderWriter; - var source : StringBuilderWriter; +def writeRegistration(var source : StringBuilderWriter; initFunctions : string; var program : ProgramPtr; cfg : StandaloneContextCfg; @@ -410,12 +343,10 @@ def writeRegistration(var header : StringBuilderWriter; globals : array; var context : Context) { write(source, "using namespace {program.thisNamespace};\n"); - write(header, "namespace {cfg.context_name} \{\n"); write(source, "namespace {cfg.context_name} \{\n"); dumpRegisterAot(unsafe(addr(source)), program, context, true, cfg.cross_platform); writeModuleRegistration(source, registrations); - writeStandaloneContext(program, initFunctions, header, source, cfg, globals, context); - write(header, "\} // namespace {cfg.context_name}\n"); + writeStandaloneContext(program, initFunctions, source, cfg, globals, context); write(source, "\} // namespace {cfg.context_name}\n"); } @@ -457,7 +388,6 @@ def addFunctionInfo(fnn : array; var helper : AotDebugInfoHelper?) { def genStandaloneSrc(var program : ProgramPtr; - var header : StringBuilderWriter; var source : StringBuilderWriter; cfg : StandaloneContextCfg; var coll : BlockVariableCollector?; globals : array) { var initFunctions : string; @@ -477,7 +407,7 @@ def genStandaloneSrc(var program : ProgramPtr; var gen = new StandaloneContextGen(program, unsafe(addr(tmp_writer)), coll, cfg.cross_platform); make_visitor(*gen) $(adapter) { gen.adapter := adapter - program |> visit_module(adapter, program.getThisModule); + program |> visit_module(adapter, program.getThisModule, true); // a generic instance several modules instantiated is one C++ function: emit the first copy only var emitted : table for (pfun in collectProgramUsedFunctions(program, false, false)) { @@ -504,20 +434,10 @@ def genStandaloneSrc(var program : ProgramPtr; } } } - write(header, type_defs); write(source, "namespace {program.thisNamespace} \{\n"); write(source, "{ctx_generated}"); write(source, "\} // namespace {program.thisNamespace}\n"); - return initFunctions -} - -def private isMarkedNoAot(pfun : Function?) : bool { - for (ann in pfun.annotations) { - if (ann.annotation.name == "no_aot") { - return true - } - } - return false + return <- (init_functions = initFunctions, type_defs = type_defs) } def private checkAllUsedFunctionsCanAot(program : ProgramPtr) { @@ -527,7 +447,7 @@ def private checkAllUsedFunctionsCanAot(program : ProgramPtr) { return } if (pfun.flags.noAot) { - if (isMarkedNoAot(pfun)) { + if ((pfun |> find_annotation("no_aot")) != null) { macro_error(program, pfun.at, "standalone AOT cannot emit function {pfun.name}: it is [no_aot], and a standalone context has no interpreter") } else { macro_error(program, pfun.at, "standalone AOT cannot emit function {pfun.name}: it uses a type AOT cannot express, and a standalone context has no interpreter") @@ -591,6 +511,54 @@ def private prepareProgramForEmission(var program : ProgramPtr; context : Contex return coll } +def private writeCApiBodies(var writer : StringBuilderWriter; var exports : array; names : CNames; cfg : StandaloneContextCfg) { + if (exports |> empty()) { + return + } + let ns = "das::{cfg.context_name}" + write(writer, "\n// C API - the entry points a C host links\n") + write(writer, "extern \"C\" \{\n") + write(writer, "{names.prefix}_ctx * {names.prefix}_create(void) \{\n") + write(writer, " return ({names.prefix}_ctx *) new {ns}::{cfg.class_name}();\n\}\n") + write(writer, "void {names.prefix}_destroy({names.prefix}_ctx * ctx) \{\n") + write(writer, " delete ({ns}::{cfg.class_name} *) ctx;\n\}\n") + write(writer, "const char * {names.prefix}_last_error({names.prefix}_ctx * ctx) \{\n") + write(writer, " return ctx ? (({ns}::{cfg.class_name} *) ctx)->getException() : nullptr;\n\}\n") + for (e in exports) { + var fn = e.fn + write(writer, "{c_declaration(e, names)} \{\n") + var args : array + args |> reserve(length(e.params) + 1) + args |> push("({ns}::{cfg.class_name} *) ctx") + for (p, variable in e.params, fn.arguments) { + let cpp = describeCppType(variable._type, DescribeConfig(cross_platform = cfg.cross_platform)) + if (p.by_pointer) { + args |> push("*({cpp} *) {p.name}") + } elif (variable._type.isString) { + args |> push("(char *) {p.name}") + } elif (variable._type.enumType != null || variable._type.isPointer) { + args |> push("({cpp}) {p.name}") + } else { + args |> push(p.name) + } + } + let call = "{aotFuncName(fn)}({args |> join(", ")})" + if (e.result.via_out) { + let cpp_res = describeCppType(fn.result, DescribeConfig(cross_platform = cfg.cross_platform)) + write(writer, " *({cpp_res} *) out = {call};\n") + } elif (fn.result.isVoid) { + write(writer, " {call};\n") + } elif (fn.result.enumType != null || fn.result.isPointer) { + write(writer, " return ({e.result.c_type}) {call};\n") + } else { + write(writer, " return {call};\n") + } + write(writer, "\}\n") + } + write(writer, "\} // extern \"C\"\n") +} + + def public runStandaloneVisitor(var program : ProgramPtr, modules : array; registrations : array; var pctx : smart_ptr; cfg : StandaloneContextCfg) : bool { //! Runs the standalone AOT visitor on the program to generate C++ source and header files. //! `registrations` are the C++ modules the generated constructor registers, in order. @@ -603,44 +571,55 @@ def public runStandaloneVisitor(var program : ProgramPtr, modules : array empty())) { return false } - let cur_mod = mod.name.empty() ? cfg.context_name : string(mod.name); let outputFile = "{cfg.cpp_output_dir}/{cur_mod}.das"; fopen("{outputFile}.h", "wb") $(fw) { if (fw != null) { @@ -671,8 +650,9 @@ def public standalone_aot(input : string; output_dir : string; cross_platform : //! `before_emit` receives the compiled program, so a driver can read annotations //! without compiling the program a second time. let file_name = input |> split_by_chars("/\\") |> back() - let ctx_name = (file_name |> split("."))[0] - var cfg = StandaloneContextCfg(context_name = ctx_name, + let ctx_stem = (file_name |> split("."))[0] + var cfg = StandaloneContextCfg(context_name = cpp_context_ident(ctx_stem), + file_stem = ctx_stem, class_name = "Standalone", cpp_output_dir = output_dir, cross_platform = cross_platform) diff --git a/daslib/ast_boost.das b/daslib/ast_boost.das index a300be9e42..5d98c7b153 100644 --- a/daslib/ast_boost.das +++ b/daslib/ast_boost.das @@ -925,6 +925,18 @@ def convert_to_expression(value : auto ==const) { return <- convert_to_expression(value, LineInfo()) } +def find_annotation(var fn : Function?; ann_name : string) : AnnotationDeclaration? { + if (fn == null) { + return null + } + for (ann in fn.annotations) { + if (ann != null && ann.annotation.name == ann_name) { + return ann + } + } + return null +} + def find_annotation(mod_name, ann_name : string) : Annotation const? { //! Finds an annotation by module name and annotation name in the compiling program. var mod = find_compiling_module(mod_name) diff --git a/daslib/c_api_header.das b/daslib/c_api_header.das new file mode 100644 index 0000000000..ffeb75f104 --- /dev/null +++ b/daslib/c_api_header.das @@ -0,0 +1,977 @@ +options gen2 +options indenting = 4 + +module c_api_header shared private + +require daslib/ast_boost +require daslib/rtti +require daslib/fio +require daslib/strings_boost +require strings + + +let C_KEYWORDS <- {"restrict", "_Bool", "_Complex", "_Imaginary", "_Alignas", "_Alignof", + "_Atomic", "_Generic", "_Noreturn", "_Static_assert", "_Thread_local", "typeof"} + + +let private CPP_RESULT_VAR = "__result" + +struct public StandaloneContextCfg { + context_name : string; + file_stem : string; + class_name : string; + cpp_output_dir : string; + cross_platform : bool; + registers_modules : bool +}; + + +def public c_ident(name : string) : string { + if (name |> empty()) { + return "" + } + var out = build_string() $(writer) { + name |> peek_data() $(bytes) { + for (b in bytes) { + let ch = int(b) + let keep = (is_alpha(ch) || is_number(ch) || ch == '_') + write_char(writer, keep ? ch : '_') + } + } + } + if (is_number(first_character(out))) { + out = "_{out}" + } + if (is_cpp_keyword(out) || (C_KEYWORDS |> key_exists(out))) { + out = "{out}_" + } + return out +} + + +def public cpp_context_ident(stem : string) : string { + let id = c_ident(stem) + return id |> empty() ? "das_ctx" : id +} + + +def public lib_prefix_from_path(input : string) : string { + let stem = input |> base_name() |> split(".") + return c_ident(stem |> empty() ? "" : stem[0]) +} + + +def public export_c_name_override(var fn : Function?) : string { + var ann = fn |> find_annotation("export_c") + if (ann == null) { + return "" + } + for (arg in ann.arguments) { + if (arg.name == "name" && arg.basicType == Type.tString) { + return string(arg.sValue) + } + } + return "" +} + + +def public has_export_c_annotation(var fn : Function?) : bool { + return (fn |> find_annotation("export_c")) != null +} + + +def private aotFunctionName(str : string) { + return replace(str, "`", "__") +} + +def public standalone_function_name(var fn : Function?) : string { + let renamed = fn |> export_c_name_override() + if (!(renamed |> empty())) { + return renamed + } + return aotFunctionName(string(fn.origin != null ? fn.origin.name : fn.name)) +} + + +def public c_export_symbol(prefix : string; var fn : Function?; name_override : string) : string { + if (!(name_override |> empty())) { + return "{prefix}_{name_override}" + } + let das_name = string(fn.origin != null ? fn.origin.name : fn.name) + return "{prefix}_{c_ident(das_name)}" +} + + +struct public CType { + ok : bool + spelling : string + by_pointer : bool +} + +struct CVector { + ok : bool + name : string + elem : string + lanes : int + span : bool + bytes : int +} + + +def private is_unsigned_type(t : Type) : bool { + return t == Type.tUInt || t == Type.tUInt8 || t == Type.tUInt16 || t == Type.tUInt64 +} + + +def private c_int_name(is_signed : bool; bytes : int) : string { + return "{is_signed ? "" : "u"}int{bytes * 8}_t" +} + + +def private scalar_of(t : TypeDeclPtr) : string { + if (t.isBool) { + return "bool" + } + if (t.isInteger) { + return c_int_name(t.isSignedInteger, t.sizeOf) + } + if (t.isBitfield) { + return c_int_name(false, t.sizeOf) + } + if (t.isFloatOrDouble) { + return t.sizeOf == 4 ? "float" : "double" + } + return "" +} + + +def private vector_of(var t : TypeDeclPtr) : CVector { + if (t == null || !t.isVectorType) { + return CVector(ok = false) + } + let lanes = t.vectorDim + let bytes = t.sizeOf / lanes + let base = t.isRange ? t.rangeBaseType : t.vectorBaseType + let is_unsigned = is_unsigned_type(base) + var elem = c_int_name(!is_unsigned, bytes) + var name = "{is_unsigned ? "uint" : "int"}{lanes}" + if (base == Type.tFloat || base == Type.tDouble) { + elem = bytes == 4 ? "float" : "double" + name = "{elem}{lanes}" + } + if (t.isRange) { + name = "{is_unsigned ? "u" : ""}range{bytes == 8 ? "64" : ""}" + } + return CVector(ok = true, name = name, elem = elem, lanes = lanes, span = t.isRange, bytes = t.sizeOf) +} + + +let VEC_LANE_NAMES <- ["x", "y", "z", "w"] +let SPAN_LANE_NAMES <- ["from", "to"] + + +def private vector_field_names(v : CVector) : array { + return [for (i in range(v.lanes)); v.span ? SPAN_LANE_NAMES[i] : VEC_LANE_NAMES[i]] +} + + +struct public CNames { + prefix : string + @do_not_delete this_module : Module? +} + + +def private c_type_name(names : CNames; var owner : Module?; name : das_string) : string { + if (owner == null || owner == names.this_module || owner.name |> empty()) { + return "{names.prefix}_{c_ident(string(name))}" + } + return "{names.prefix}_{c_ident(string(owner.name))}_{c_ident(string(name))}" +} + + +def private peel(var t : TypeDeclPtr) : TypeDeclPtr { + if (t.baseType == Type.tDistinct && t.firstType != null) { + return peel(t.firstType) + } + return t +} + + +def private is_pod_struct(var st : Structure?) : bool { + return st != null && !st.flags.isClass && !(st.fields |> empty()) +} + + +def private pointee_ok(var t : TypeDeclPtr) : bool { + if (t == null) { + return true + } + var pt = peel(t) + if (pt.isStructure) { + return is_pod_struct(pt.structType) + } + if (pt.isEnumT) { + return pt.enumType != null && !pt.enumType.external + } + return !(scalar_of(pt) |> empty()) || vector_of(pt).ok || pt.isVoid +} + + +def private pointer_c_type(var t : TypeDeclPtr; names : CNames) : CType { + if (t.flags.smartPtr) { + return CType(ok = false) + } + var target = t.firstType + if (target == null || target.isVoid || !pointee_ok(target)) { + return CType(ok = true, spelling = "void *") + } + var pt = peel(target) + var inner = "" + if (pt.isStructure) { + inner = c_type_name(names, pt.structType._module, pt.structType.name) + } elif (pt.isEnumT) { + inner = c_type_name(names, pt.enumType._module, pt.enumType.name) + } else { + let v = vector_of(pt) + inner = v.ok ? "{names.prefix}_{v.name}" : scalar_of(pt) + } + let qual = pt.flags.constant ? "const " : "" + return CType(ok = true, spelling = "{qual}{inner} *") +} + + +def private field_c_type(var t : TypeDeclPtr; names : CNames; var visiting : table) : CType { + var ft = peel(t) + if (ft.baseType == Type.tFixedArray) { + let inner = ft.firstType == null ? CType(ok = false) : field_c_type(ft.firstType, names, visiting) + return inner.ok && !inner.by_pointer ? CType(ok = true, spelling = inner.spelling) : CType(ok = false) + } + if (ft.isStructure) { + if (!struct_c_ok(ft.structType, names, visiting)) { + return CType(ok = false) + } + return CType(ok = true, spelling = c_type_name(names, ft.structType._module, ft.structType.name)) + } + return value_c_type(ft, names) +} + + +def private struct_c_ok(var st : Structure?; names : CNames; var visiting : table) : bool { + if (!is_pod_struct(st)) { + return false + } + let key = "{st._module.name}::{st.name}" + if (visiting |> key_exists(key)) { + return true + } + visiting |> insert(key) + var ok = true + for (fld in st.fields) { + if (!field_c_type(fld._type, names, visiting).ok) { + ok = false + break + } + } + visiting |> erase(key) + return ok +} + + +def private handle_wrap_type(var t : TypeDeclPtr) : TypeDeclPtr { + if (t.baseType != Type.tHandle || t.annotation == null || t.annotation.isRefType) { + return TypeDeclPtr() + } + return <- get_underlying_value_type(t) +} + + +def private handle_c_type(var t : TypeDeclPtr; names : CNames) : CType { + var wrap <- handle_wrap_type(t) + if (wrap == null) { + return CType(ok = false) + } + let inner = value_c_type(wrap, names) + if (!inner.ok) { + return CType(ok = false) + } + return CType(ok = true, spelling = c_type_name(names, t.annotation._module, t.annotation.name), + by_pointer = inner.by_pointer) +} + + +def private value_c_type(var t : TypeDeclPtr; names : CNames) : CType { + if (t.baseType == Type.tHandle) { + return handle_c_type(t, names) + } + if (t.isPointer) { + return pointer_c_type(t, names) + } + if (t.isString) { + return CType(ok = true, spelling = "const char *") + } + if (t.isEnumT) { + if (t.enumType == null || t.enumType.external) { + return CType(ok = false) + } + return CType(ok = true, spelling = c_type_name(names, t.enumType._module, t.enumType.name)) + } + let v = vector_of(t) + if (v.ok) { + return CType(ok = true, spelling = "{names.prefix}_{v.name}", by_pointer = true) + } + let sc = scalar_of(t) + return sc |> empty() ? CType(ok = false) : CType(ok = true, spelling = sc) +} + + +def public c_type_of(var t : TypeDeclPtr; names : CNames) : CType { + if (t == null) { + return CType(ok = false) + } + var pt = peel(t) + if (pt.isVoid) { + return CType(ok = true, spelling = "void") + } + if (pt.isStructure) { + var visiting : table + if (!struct_c_ok(pt.structType, names, visiting)) { + return CType(ok = false) + } + return CType(ok = true, spelling = c_type_name(names, pt.structType._module, pt.structType.name), by_pointer = true) + } + if (pt.baseType == Type.tFixedArray) { + var visiting : table + let inner = pt.firstType == null ? CType(ok = false) : field_c_type(pt.firstType, names, visiting) + return inner.ok ? CType(ok = true, spelling = inner.spelling, by_pointer = true) : CType(ok = false) + } + return value_c_type(pt, names) +} + + +struct public CParam { + name : string + c_type : string + by_pointer : bool + is_const : bool + pointer_in_impl : bool +} + + +struct public CResult { + c_type : string + via_out : bool + cmres : bool +} + + +struct public CExport { + @do_not_delete fn : Function? + c_name : string + das_signature : string + params : array + result : CResult +} + + +struct CReject { + ok : bool + what : string + why : string +} + + +def private param_name_of(raw : string; index : int) : string { + let n = c_ident(raw) + if (n |> empty()) { + return "a{index}" + } + return n == "ctx" || n == "out" ? "{n}_" : n +} + + +def private describe_params(var fn : Function?; names : CNames; var out : array) : CReject { + for (arg, i in fn.arguments, count()) { + let ct = c_type_of(arg._type, names) + if (!ct.ok || arg._type.isVoid) { + return CReject(what = "parameter `{arg.name}`", why = arg._type.describe()) + } + let by_ptr = ct.by_pointer || arg._type.flags.ref + out |> emplace(CParam(name = param_name_of(string(arg.name), i), + c_type = ct.spelling, + by_pointer = by_ptr, + is_const = by_ptr && arg._type.flags.constant, + pointer_in_impl = arg._type.isRef)) + } + return CReject(ok = true) +} + + +def private returns_cmres(var fn : Function?) : bool { + return fn.flags.copyOnReturn || fn.flags.moveOnReturn +} + + +def public describe_c_signature(var fn : Function?; names : CNames; c_name : string) : tuple { + var params : array + let bad = describe_params(fn, names, params) + if (!bad.ok) { + return (exp = CExport(), reject = bad) + } + let rt = c_type_of(fn.result, names) + if (!rt.ok || peel(fn.result).baseType == Type.tFixedArray) { + return (exp = CExport(), reject = CReject(what = "result", why = fn.result.describe())) + } + let res = CResult(c_type = rt.spelling, via_out = rt.by_pointer, cmres = returns_cmres(fn)) + return (exp = CExport(fn = fn, c_name = c_name, das_signature = das_signature_of(fn), + params <- params, result = res), reject = CReject(ok = true)) +} + + +def private das_signature_of(var fn : Function?) : string { + let args = [for (a in fn.arguments); "{a.name} : {a._type.describe() |> replace(" const", "")}"] + return "def {fn.name}({args |> join("; ")}) : {fn.result.describe() |> replace(" const", "")}" +} + + +struct CEnum { + @do_not_delete en : Enumeration? + base : string +} + + +struct CHandle { + c_name : string + cpp_name : string + wrap : string + lanes : int + bytes : int + align : int + by_pointer : bool +} +struct CTypeSet { + enums : array + vectors : array + handles : array + @do_not_delete structs : array + @do_not_delete opaque : array + seen : table +} + + +def private collect_type(var t : TypeDeclPtr; names : CNames; var set : CTypeSet) { + if (t == null) { + return + } + var pt = peel(t) + if (pt.isPointer) { + var pointee = pt.firstType == null ? TypeDeclPtr() : peel(pt.firstType) + if (pointee != null && pointee.isStructure) { + var visiting : table + if (struct_c_ok(pointee.structType, names, visiting)) { + collect_struct(pointee.structType, names, set) + } else { + collect_opaque_struct(pointee.structType, set) + } + } else { + collect_type(pt.firstType, names, set) + } + return + } + if (pt.baseType == Type.tFixedArray) { + collect_type(pt.firstType, names, set) + return + } + if (pt.isEnumT && pt.enumType != null && !pt.enumType.external) { + let key = "e:{pt.enumType._module.name}::{pt.enumType.name}" + if (!(set.seen |> key_exists(key))) { + set.seen |> insert(key) + set.enums |> push(CEnum(en = pt.enumType, + base = c_int_name(!is_unsigned_type(pt.enumType.baseType), pt.sizeOf))) + } + return + } + let v = vector_of(pt) + if (v.ok) { + let key = "v:{v.name}" + if (!(set.seen |> key_exists(key))) { + set.seen |> insert(key) + set.vectors |> push(v) + } + return + } + if (pt.baseType == Type.tHandle) { + collect_handle(pt, names, set) + return + } + if (pt.isStructure) { + collect_struct(pt.structType, names, set) + } +} + + +def private collect_handle(var t : TypeDeclPtr; names : CNames; var set : CTypeSet) { + var wrap <- handle_wrap_type(t) + if (wrap == null) { + return + } + let inner = value_c_type(wrap, names) + if (!inner.ok) { + return + } + let key = "h:{t.annotation._module.name}::{t.annotation.name}" + if (set.seen |> key_exists(key)) { + return + } + set.seen |> insert(key) + let v = vector_of(wrap) + set.handles |> push(CHandle( + c_name = c_type_name(names, t.annotation._module, t.annotation.name), + cpp_name = string(t.annotation.cppName |> empty() ? t.annotation.name : t.annotation.cppName), + wrap = v.ok ? v.elem : inner.spelling, lanes = v.ok ? v.lanes : 0, + bytes = t.sizeOf, align = t.alignOf, by_pointer = inner.by_pointer)) +} + + +def private collect_opaque_struct(var st : Structure?; var set : CTypeSet) { + if (st == null) { + return + } + let key = "o:{st._module.name}::{st.name}" + if (set.seen |> key_exists(key)) { + return + } + set.seen |> insert(key) + set.opaque |> push(st) +} + + +def private collect_struct(var st : Structure?; names : CNames; var set : CTypeSet) { + if (st == null) { + return + } + let key = "s:{st._module.name}::{st.name}" + if (set.seen |> key_exists(key)) { + return + } + set.seen |> insert(key) + for (fld in st.fields) { + collect_type(fld._type, names, set) + } + set.structs |> push(st) +} + + +def public collect_c_types(var exports : array; names : CNames) : CTypeSet { + var set : CTypeSet + for (e in exports) { + for (arg in e.fn.arguments) { + collect_type(arg._type, names, set) + } + collect_type(e.fn.result, names, set) + } + return <- set +} + + +def private enum_value_of(var en : Enumeration?; name : das_string; base : string) : string { + let v = en |> find_enum_value(string(name)) + let is_unsigned = base |> starts_with("u") + let digits = is_unsigned ? "{uint64(v):d}" : "{v:d}" + if (base == "int64_t" || base == "uint64_t") { + return "{is_unsigned ? "UINT64_C" : "INT64_C"}({digits})" + } + return is_unsigned ? "{digits}u" : digits +} + + +def private write_enum(var writer : StringBuilderWriter; var e : CEnum; names : CNames) { + var en = e.en + let cname = c_type_name(names, en._module, en.name) + write(writer, "\n// das: enum {en.name}\ntypedef {e.base} {cname};\n") + write(writer, "enum \{\n") + let entries = [for (ee in en.list); " {cname}_{c_ident(string(ee.name))} = {enum_value_of(en, ee.name, e.base)}"] + write(writer, "{entries |> join(",\n")}\n\};\n") +} + + +def private needs_alignas(handles : array) : bool { + for (h in handles) { + if (h.lanes != 0) return true + } + return false +} + + +def private write_handle(var writer : StringBuilderWriter; h : CHandle; names : CNames) { + let guard = names.prefix |> to_upper() + let body = h.lanes == 0 ? h.wrap : "struct \{ {guard}_ALIGNAS({h.align}) {h.wrap} lanes[{h.lanes}]; \}" + write(writer, "\n// das: {h.cpp_name} - a bound value type, carried as its ABI wrap type. Pass it\n") + write(writer, "// back as you received it; the lanes are the ABI's, not the type's own fields.\ntypedef {body} {h.c_name};\n") + write(writer, "{guard}_STATIC_ASSERT(sizeof({h.c_name}) == {h.bytes}, \"{h.c_name}: size differs from the daslang layout\");\n") + write(writer, "{guard}_STATIC_ASSERT({guard}_ALIGNOF({h.c_name}) == {h.align}, \"{h.c_name}: alignment differs from the daslang layout\");\n") +} + + +def private write_vector(var writer : StringBuilderWriter; v : CVector; names : CNames) { + let cname = "{names.prefix}_{v.name}" + let fields = vector_field_names(v) |> join(", ") + write(writer, "\n// das: {v.name}\ntypedef struct \{ {v.elem} {fields}; \} {cname};\n") + write(writer, "{names.prefix |> to_upper()}_STATIC_ASSERT(sizeof({cname}) == {v.bytes}, \"{cname}: size differs from the daslang layout\");\n") +} + + +def private field_decl(var fld : FieldDeclaration; names : CNames) : string { + var visiting : table + var ft = peel(fld._type) + var dims : array + while (ft.baseType == Type.tFixedArray && ft.firstType != null) { + dims |> push("[{ft.fixedDim}]") + ft = peel(ft.firstType) + } + let ct = field_c_type(fld._type, names, visiting) + return " {ct.spelling} {c_ident(string(fld.name))}{dims |> join("")};" +} + + +def private write_struct(var writer : StringBuilderWriter; var st : Structure?; names : CNames) { + let cname = c_type_name(names, st._module, st.name) + let guard = names.prefix |> to_upper() + write(writer, "\n// das: struct {st.name}\nstruct {cname} \{\n") + for (fld in st.fields) { + write(writer, "{field_decl(fld, names)}\n") + } + write(writer, "\};\n") + write(writer, "{guard}_STATIC_ASSERT(sizeof({cname}) == {st.sizeOf}, \"{cname}: size differs from the daslang layout\");\n") + for (fld in st.fields) { + write(writer, "{guard}_STATIC_ASSERT(offsetof({cname}, {c_ident(string(fld.name))}) == {fld.offset}, \"{cname}.{fld.name}: offset differs from the daslang layout\");\n") + } +} + + +def private param_decl(p : CParam) : string { + if (!p.by_pointer) { + return "{p.c_type} {p.name}" + } + return "{p.is_const ? "const " : ""}{p.c_type} * {p.name}" +} + + +def public c_declaration(e : CExport; names : CNames) : string { + var args = ["{names.prefix}_ctx * ctx"] + args |> reserve(length(e.params) + 2) + for (p in e.params) { + args |> push(param_decl(p)) + } + if (e.result.via_out) { + args |> push("{e.result.c_type} * out") + } + let ret = e.result.via_out ? "void" : e.result.c_type + return "{ret} {e.c_name}({args |> join(", ")})" +} + + +def private write_prologue(var writer : StringBuilderWriter; names : CNames; generated_by : string; needs_alignas : bool) { + let guard = names.prefix |> to_upper() + write(writer, "// Code generated by `{generated_by}`. DO NOT EDIT.\n") + write(writer, "// Layouts are asserted for the daslang that generated this header; regenerate it for another target.\n") + write(writer, "#pragma once\n\n#include \n#include \n#include \n\n") + write(writer, "#if defined(__cplusplus)\n#define {guard}_STATIC_ASSERT(cond, msg) static_assert(cond, msg)\n") + write(writer, "#elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L\n") + write(writer, "#define {guard}_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)\n") + write(writer, "#else\n#define {guard}_STATIC_ASSERT(cond, msg)\n#endif\n\n") + if (needs_alignas) { + write(writer, "#if defined(__cplusplus)\n#define {guard}_ALIGNAS(n) alignas(n)\n") + write(writer, "#elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L\n") + write(writer, "#define {guard}_ALIGNAS(n) _Alignas(n)\n") + write(writer, "#elif defined(_MSC_VER)\n#define {guard}_ALIGNAS(n) __declspec(align(n))\n") + write(writer, "#else\n#define {guard}_ALIGNAS(n) __attribute__((aligned(n)))\n#endif\n") + write(writer, "#if defined(__cplusplus)\n#define {guard}_ALIGNOF(t) alignof(t)\n") + write(writer, "#elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L\n") + write(writer, "#define {guard}_ALIGNOF(t) _Alignof(t)\n") + write(writer, "#else\n#define {guard}_ALIGNOF(t) __alignof__(t)\n#endif\n\n") + } + write(writer, "#ifdef __cplusplus\nextern \"C\" \{\n#endif\n") +} + + +def private write_fixed_api(var writer : StringBuilderWriter; names : CNames) { + let p = names.prefix + write(writer, "\n// One library instance and everything it owns: globals, heap, string heap. Drive one\n") + write(writer, "// instance from one thread at a time, or make one per thread.\n") + write(writer, "typedef struct {p}_ctx {p}_ctx;\n") + write(writer, "\n// Creates an instance, then runs the script's global initializers and its [init] functions.\n") + write(writer, "// Callable from any thread, and from a process that already carries a daslang runtime -\n") + write(writer, "// another such library, or a host that registered the daslang modules itself. Returns NULL\n") + write(writer, "// when the initializers raised; {p}_last_error(NULL) reports why.\n") + write(writer, "{p}_ctx * {p}_create(void);\n") + write(writer, "\n// Runs the script's [finalize] functions, then frees the instance and everything it\n") + write(writer, "// allocated - every string and every pointer it returned included.\n") + write(writer, "void {p}_destroy({p}_ctx * ctx);\n") + write(writer, "\n// NULL when the last call on `ctx` completed normally; otherwise that call's exception text,\n") + write(writer, "// owned by the instance and valid until the next call on it. Pass NULL to read a failed\n") + write(writer, "// {p}_create. On a failed call the returned value is 0/false/NULL and `out` is left untouched.\n") + write(writer, "const char * {p}_last_error({p}_ctx * ctx);\n") + write(writer, "\n// Drains the daslang runtime for the whole process. Optional, and final: no instance of any\n") + write(writer, "// daslang library may be created or called afterwards.\n") + write(writer, "void {p}_shutdown_runtime(void);\n") + write(writer, "\n// A `const char *` an exported function returns points into the instance's string heap, and a\n") + write(writer, "// returned `T *` into its object heap: copy what you need past the next call on that instance,\n") + write(writer, "// because a later call may collect. The empty daslang string comes back NULL. A pointer you\n") + write(writer, "// pass in, `const char *` included, is borrowed for the duration of the call.\n") +} + + +def private write_c_section(var writer : StringBuilderWriter; var exports : array; names : CNames; + generated_by, link_note : string) { + var types <- collect_c_types(exports, names) + { + write_prologue(writer, names, generated_by, needs_alignas(types.handles)) + write_fixed_api(writer, names) + for (en in types.enums) { + write_enum(writer, en, names) + } + for (vt in types.vectors) { + write_vector(writer, vt, names) + } + for (h in types.handles) { + write_handle(writer, h, names) + } + var defined : table + for (st in types.structs) { + defined |> insert(c_type_name(names, st._module, st.name)) + } + if (!(types.structs |> empty()) || !(types.opaque |> empty())) { + write(writer, "\n") + for (st in types.structs) { + let cname = c_type_name(names, st._module, st.name) + write(writer, "typedef struct {cname} {cname};\n") + } + for (st in types.opaque) { + let cname = c_type_name(names, st._module, st.name) + if (!(defined |> key_exists(cname))) { + write(writer, "typedef struct {cname} {cname};\n") + } + } + } + for (st in types.structs) { + write_struct(writer, st, names) + } + write(writer, "\n") + for (e in exports) { + write(writer, "\n// {e.das_signature}\n{c_declaration(e, names)};\n") + } + if (!(link_note |> empty())) { + write(writer, "\n// Linking:\n") + for (line in link_note |> split("\n")) { + write(writer, "// {line}\n") + } + } + write(writer, "\n#ifdef __cplusplus\n\} // extern \"C\"\n#endif\n") + } +} + + +def public build_c_header(var exports : array; names : CNames; input_path, output_path, link_note : string) : string { + return build_string() $(var writer) { + write_c_section(writer, exports, names, "daslang -lib {input_path} -output {output_path}", link_note) + } +} + + +def public emit_c_header(var exports : array; names : CNames; input_path, output_path, link_note : string) : bool { + let text = build_c_header(exports, names, input_path, output_path, link_note) + let path = "{output_path}.h" + var ok = false + fopen(path, "wb") $(f) { + if (f != null) { + f |> fwrite(text) + ok = true + } + } + if (!ok) { + to_log(LOG_ERROR, "daslang -lib: can't write the C header {path}\n") + } + return ok +} + + +struct public CRejection { + fn_name : string + message : string +} + + +def public is_c_export_candidate(var fn : Function?; prog : Program?; export_all : bool) : bool { + if (fn._module != prog.getThisModule || !fn.flags.exports + || fn.flags.builtIn || fn.flags.generated || fn.moreFlags.isTemplate || fn.fromGeneric != null + || fn.flags.init || fn.flags.shutdown || fn.flags.macroInit || fn.moreFlags.macroFunction) { + return false + } + return export_all || has_export_c_annotation(fn) +} + + +def private reserved_c_names(names : CNames) : array { + let p = names.prefix + return ["{p}_create", "{p}_destroy", "{p}_last_error", "{p}_shutdown_runtime", "{p}_ctx"] +} + + +def public collect_c_exports(prog : Program?; names : CNames; export_all : bool) : tuple; errors : array> { + var accepted : array + var errors : array + var claimed <- {for (r in reserved_c_names(names)); r => "the fixed library API"} + var this_mod = prog.getThisModule + this_mod |> for_each_function("") $(var fn) { + if (!is_c_export_candidate(fn, prog, export_all)) { + return + } + let is_explicit = has_export_c_annotation(fn) + let c_name = c_export_symbol(names.prefix, fn, export_c_name_override(fn)) + var why = "" + let owner = claimed?[c_name] ?? "" + var sig <- describe_c_signature(fn, names, c_name) + if (!(owner |> empty())) { + why = "C symbol `{c_name}` is already taken by {owner}; C has no overloading - give one of them [export_c(name = \"...\")]" + } elif (!sig.reject.ok) { + why = ("{sig.reject.what} has type {sig.reject.why}, which has no C representation " + + "(allowed: bool, int8..uint64, int, uint, float, double, string, pointers, enums, " + + "POD structs, bound value types, float2..uint4, range/urange/range64/urange64, " + + "fixed_array arguments)") + } + if (!(why |> empty())) { + if (is_explicit) { + errors |> emplace(CRejection(fn_name = string(fn.name), + message = "{describe(fn.at)}: [export_c] {fn.name}: {why}")) + } else { + to_log(LOG_WARNING, "{describe(fn.at)}: skipping {fn.name} in the C API: {why}\n") + } + return + } + claimed[c_name] = "{fn.name} at {describe(fn.at)}" + accepted |> emplace(sig.exp) + } + return (exports <- accepted, errors <- errors) +} + + +struct public CppApi { + class_name : string + namespace_name : string + includes : string + type_defs : string + registers_modules : bool +} + + +struct private CppArg { + decl : string + value : string +} + + +def private cpp_type_name(ns : string; names : CNames; var owner : Module?; name : das_string) : string { + if (owner == null || owner == names.this_module || owner.name |> empty()) { + return "{ns}::{name}" + } + return "{owner.name}::{name}" +} + + +def private cpp_enum_of(var t : TypeDeclPtr; names : CNames; ns : string) : string { + return t.enumType == null ? "" : "DAS_COMMENT(enum) {cpp_type_name(ns, names, t.enumType._module, t.enumType.name)}" +} + + +def private cpp_by_ref_type(var t : TypeDeclPtr; names : CNames; ns : string) : string { + if (t.baseType == Type.tStructure && t.structType != null) { + return "struct {cpp_type_name(ns, names, t.structType._module, t.structType.name)}" + } + if (t.baseType == Type.tHandle && t.annotation != null) { + return string(t.annotation.cppName |> empty() ? t.annotation.name : t.annotation.cppName) + } + let v = vector_of(t) + return v.ok ? v.name : "" +} + + +def private cpp_arg_of(p : CParam; var raw : TypeDeclPtr; names : CNames; ns : string) : CppArg { + var t = peel(raw) + let en = cpp_enum_of(t, names, ns) + if (!(en |> empty())) { + return CppArg(decl = "{en} {p.name}", value = "({p.c_type}) {p.name}") + } + + let cst = p.is_const ? "const " : "" + if (p.by_pointer) { + let cpp = cpp_by_ref_type(t, names, ns) + if (cpp |> empty()) { + return CppArg(decl = "{cst}{p.c_type} * {p.name}", value = p.name) + } + return CppArg(decl = "{cpp} {cst}& {p.name}", value = "({cst}{p.c_type} *) &{p.name}") + } + return CppArg(decl = "{p.c_type} {p.name}", value = p.name) +} + + +def private write_cpp_proxy(var writer : StringBuilderWriter; var e : CExport; names : CNames; cpp : CppApi) { + var fn = e.fn + let ns = cpp.namespace_name + var decls : array + var args = ["({names.prefix}_ctx *) this"] + decls |> reserve(length(e.params)) + args |> reserve(length(e.params) + 2) + for (p, variable in e.params, fn.arguments) { + let a = cpp_arg_of(p, variable._type, names, ns) + decls |> push(a.decl) + args |> push(a.value) + } + let out_type = e.result.via_out ? cpp_by_ref_type(peel(fn.result), names, ns) : "" + let enum_res = cpp_enum_of(peel(fn.result), names, ns) + var ret = e.result.c_type + if (e.result.via_out) { + ret = out_type + } elif (!(enum_res |> empty())) { + ret = enum_res + } + if (e.result.via_out) { + args |> push("({e.result.c_type} *) &{CPP_RESULT_VAR}") + } + write(writer, " auto {standalone_function_name(fn)} ( {decls |> join(", ")} ) -> {ret} \{\n") + let call = "::{e.c_name}({args |> join(", ")})" + if (e.result.via_out) { + write(writer, " {ret} {CPP_RESULT_VAR};\n {call};\n return {CPP_RESULT_VAR};\n") + } elif (fn.result.isVoid) { + write(writer, " {call};\n") + } elif (!(enum_res |> empty())) { + write(writer, " return ({ret}) {call};\n") + } else { + write(writer, " return {call};\n") + } + write(writer, " \}\n") +} + + +def private write_cpp_section(var writer : StringBuilderWriter; var exports : array; names : CNames; cpp : CppApi) { + write(writer, "\n#ifdef __cplusplus\n\n") + write(writer, cpp.includes) + write(writer, "namespace das \{\n") + write(writer, cpp.type_defs) + write(writer, "namespace {cpp.namespace_name} \{\n\n") + let bases = cpp.registers_modules ? "public StandaloneModuleScope, public Context" : "public Context" + write(writer, "class {cpp.class_name} : {bases} \{\npublic:\n {cpp.class_name}();\n") + for (e in exports) { + write_cpp_proxy(writer, e, names, cpp) + } + write(writer, "\};\n\n") + write(writer, "\} // namespace {cpp.namespace_name}\n\} // namespace das\n\n#endif // __cplusplus\n") +} + + +def public build_standalone_header(var exports : array; names : CNames; cpp : CppApi; + generated_by, link_note : string) : string { + return build_string() $(var writer) { + write_c_section(writer, exports, names, generated_by, link_note) + write_cpp_section(writer, exports, names, cpp) + } +} diff --git a/daslib/export_c.das b/daslib/export_c.das new file mode 100644 index 0000000000..65d68a7726 --- /dev/null +++ b/daslib/export_c.das @@ -0,0 +1,49 @@ +options gen2 +options indenting = 4 + +module export_c shared private !inscope + +require daslib/ast_boost +require strings + + +def private is_c_identifier(name : string) : bool { + if (name |> empty()) { + return false + } + var first = true + for (ch in name) { + let alpha = is_alpha(ch) || ch == '_' + if (!(alpha || (is_number(ch) && !first))) { + return false + } + first = false + } + return true +} + + +[function_macro(name = "export_c")] +class private ExportCAnnotation : AstFunctionAnnotation { + def override apply(var func : FunctionPtr; var group : ModuleGroup; + args : AnnotationArgumentList; var errors : das_string) : bool { + for (arg in func.arguments) { + if (arg._type.baseType == Type.autoinfer && arg.init == null) { + errors := "[export_c] can't export generic function `{func.name}`: it has auto or template arguments, and C needs one concrete signature" + return false + } + } + for (arg in args) { + if (arg.name != "name") { + errors := "[export_c] unknown argument `{arg.name}`; the only argument is name=\"\"" + return false + } + if (arg.basicType != Type.tString || !is_c_identifier(string(arg.sValue))) { + errors := "[export_c] name must be a valid C identifier ([A-Za-z_][A-Za-z0-9_]*), got `{arg.sValue}`" + return false + } + } + func.flags.exports = true + return true + } +} diff --git a/doc/source/reference/embedding/advanced.rst b/doc/source/reference/embedding/advanced.rst index 2058b2ced3..fafae64bd6 100644 --- a/doc/source/reference/embedding/advanced.rst +++ b/doc/source/reference/embedding/advanced.rst @@ -4,6 +4,7 @@ .. index:: single: Embedding; AOT single: Embedding; Advanced Topics + single: Embedding; C Libraries single: Embedding; Class Adapters single: Embedding; Coroutines single: Embedding; Standalone Contexts @@ -274,6 +275,60 @@ See :ref:`tutorial_integration_cpp_standalone_contexts` for a complete example. +C libraries +=========== + +``daslang -lib`` compiles a daslang program to a native library with a C +API, so a host that only *calls* one script needs no daslang headers, no +``Module``, and no compiler. The LLVM backend emits the library; a +standalone context (above) emits C++ source your build compiles instead. + +Pipeline: + +1. ``daslang -lib script.das -output build/script`` +2. This writes ``build/script.so`` (``.dylib`` / ``.dll``) and + ``build/script.h``. Add ``-- --jit-lib-static`` for a ``.a`` / ``.lib`` + archive instead; the header records what a host then links. +3. Include the header, link the library, call it. + +Which functions cross the boundary is your choice: ``[export_c]`` marks +them one at a time, and ``-lib-export-all`` offers every public function +of the entry module whose signature C can spell, naming the ones it skips. + +Every library gets the same four entry points, prefixed with the output +name — one instance owns one daslang context, with its own globals and +heap: + +.. code-block:: c + + #include "script.h" + + script_ctx * ctx = script_create(); + if ( !ctx ) { + printf("%s\n", script_last_error(NULL)); + return 1; + } + script_Vec3 v = { 1.0f, 2.0f, 3.0f }; + script_Vec3 out; + script_scale(ctx, &v, 2.0f, &out); /* a struct result uses a trailing out pointer */ + script_destroy(ctx); + +A daslang panic never unwinds into C: the call returns zero and leaves +``out`` untouched, and ``script_last_error(ctx)`` reports the text until +the next call on that instance clears it. A ``const char *`` a function +returns lives in that instance's string heap, so copy it if you need it +past the next call. + +The generated header asserts the layout of every structure it declares +(``_Static_assert`` in C11, ``static_assert`` in C++), so a host built +for a different target fails to compile rather than misreading memory. + +Several such libraries coexist in one process, and so does a library inside a +host that registered the daslang modules itself: whoever gets there first +registers the runtime, and the rest bind to it. Several instances of one +library are fine, on any thread. + + Serialization ============= diff --git a/doc/source/reference/language/annotations.rst b/doc/source/reference/language/annotations.rst index 15ec5828ff..31111aa93f 100644 --- a/doc/source/reference/language/annotations.rst +++ b/doc/source/reference/language/annotations.rst @@ -65,6 +65,40 @@ Lifecycle print("hello\n") } +``[export_c]`` + An ``[export]`` that also names the function to C: a standalone context declares it in the C + half of the header it generates, so a C host calls it with no daslang API of its own. Needs + ``require daslib/export_c``. The function keeps working in every tier. Its signature + has to be one C can spell: scalars, ``string``, raw pointers, enumerations, plain structures, + the ``float2``..``uint4`` and ``range`` families, and ``fixed_array`` arguments; ``array``, + ``table``, ``tuple``, ``variant``, lambdas, blocks, iterators and bound C++ types are not, and + the build names the parameter it could not spell. A generic function cannot carry it. + ``name="..."`` renames the C symbol, which is how two overloads reach C at all: + + .. das-doc: alt + .. code-block:: das + + struct Vec3 { + x : float + y : float + z : float + } + + [export_c] + def dot(a, b : Vec3) : float { + return a.x * b.x + a.y * b.y + a.z * b.z + } + + [export_c(name = "scale_by")] + def scale(v : Vec3; k : float) : Vec3 { + return Vec3(x = v.x * k, y = v.y * k, z = v.z * k) + } + + A scalar result comes back directly, so ``dot`` becomes + ``float p_dot(p_ctx *, const p_Vec3 *, const p_Vec3 *)``; a structure or vector result travels + through a trailing out pointer, so ``scale`` becomes + ``void p_scale_by(p_ctx *, const p_Vec3 *, float, p_Vec3 *)``. + ``[init]`` Marks a function to run automatically during context initialization. The function must take no arguments and return ``void``: diff --git a/examples/standalone/CMakeLists.txt b/examples/standalone/CMakeLists.txt index 256ad82b7d..dfcd8648f5 100644 --- a/examples/standalone/CMakeLists.txt +++ b/examples/standalone/CMakeLists.txt @@ -33,6 +33,7 @@ function(das_nano_example name dir das_file) ${ARGN} "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT: ${das_file}" @@ -69,6 +70,7 @@ if(NOT DAS_HV_DISABLED) "${CMAKE_CURRENT_SOURCE_DIR}/06_full_runtime/service_probe.das" "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT (full runtime): service_probe.das" diff --git a/skills/cpp_integration.md b/skills/cpp_integration.md index 6d12d75e9c..e7ec84f724 100644 --- a/skills/cpp_integration.md +++ b/skills/cpp_integration.md @@ -346,6 +346,49 @@ link. Worked example: `examples/standalone/06_full_runtime/` - read it for the s it needs the daslang repository, since the bundle carries no dasHV headers or archive. The recipe above works from a bundle for any C++ module you build yourself. +## Calling daslang from C - `daslang -lib` + +When the host is C, or wants no daslang API at all, compile the script to a native library with a +generated C header instead: + +```sh +daslang -lib script.das -output build/script # build/script.so (.dylib/.dll) + build/script.h +daslang -lib script.das -output build/script -- --jit-lib-static # build/script.a instead +``` + +Three ways to pick what crosses: mark each function `[export_c]` (an `[export]` the library also +surfaces in C); pass `-- --jit-lib-export-marked` to take whatever the program already marks +`[export]`; or pass `-lib-export-all` for every public function of the entry module whose signature +C can spell - only export-all skips an unspellable one with a warning, the other two make it a hard +error. `-lib` carries the annotation itself; add `require daslib/export_c` to compile that same +source without the JIT, which the linter and the AOT pass both do. `examples/c_api_library/` builds +one library each way and binds all three at once. Then: + +```c +#include "script.h" + +script_ctx * ctx = script_create(); /* one instance = one context, globals and heap */ +script_Vec3 v = { 1.0f, 2.0f, 3.0f }, out; +script_scale(ctx, &v, 2.0f, &out); /* a struct or vector result uses a trailing out ptr */ +if ( script_last_error(ctx) ) { /* the call raised; out is untouched */ } +script_destroy(ctx); +``` + +Scalars, `string` (as `const char *`), pointers and enums cross by value; everything else +representable crosses as `const T *`. A daslang panic returns zero and reports through +`script_last_error(ctx)` - it never unwinds into C. A returned `const char *` lives in that +instance's string heap, so copy it if you need it past the next call. The header asserts the +layout of every structure it declares, so a host built for a different target fails to compile. +Several such libraries coexist in one process, as does a library inside a host that registered +the daslang modules itself - the first one there registers the runtime and the rest bind to it. +Several instances of one library are fine, created and driven on any thread. + +Choosing between the four: **nano** when the host is C++ and you want the smallest runtime; a +**standalone context on the full runtime** when the host is C++ and the script reaches a C++ +module beyond `builtin`; **`-lib`** when the host is C, or wants a plain ABI boundary and no +daslang headers; the **C API** (`daScriptC.h`) when the host has to compile daslang itself at run +time. + ## Diagnostics - `TextPrinter`, never `fprintf(stderr, ...)` ```cpp diff --git a/skills/daslang/references/modules-and-stdlib.md b/skills/daslang/references/modules-and-stdlib.md index 3d1562f449..7da35ea4fb 100644 --- a/skills/daslang/references/modules-and-stdlib.md +++ b/skills/daslang/references/modules-and-stdlib.md @@ -22,7 +22,10 @@ def main { print("ok\n") } The only enforced ordering is `module` before any type declaration; `options` / `module` / `require` otherwise interleave. A file with no `module` line is a program, named by its file stem. -`[export]` makes a function callable from the host by name; `[init]` / `[finalize]` run at context +`[export]` makes a function callable from the host by name, and `[export_c]` is an `[export]` that +also crosses to C - a standalone context declares it in the C half of the header it generates, so a +C host needs no daslang API (`require daslib/export_c`; `[export_c(name = "...")]` picks the C +symbol, which is how two overloads both reach C); `[init]` / `[finalize]` run at context init / shutdown (no arguments, no return). `main` is a convention, not a keyword: it returns `void` unless declared `def main() : int`, whose return value is the process exit code (do not `panic` to force one). diff --git a/tests-cpp/big/nano_ctx/CMakeLists.txt b/tests-cpp/big/nano_ctx/CMakeLists.txt index 9cef9fba7c..6f8727cc54 100644 --- a/tests-cpp/big/nano_ctx/CMakeLists.txt +++ b/tests-cpp/big/nano_ctx/CMakeLists.txt @@ -44,6 +44,7 @@ foreach(_pair "01_pure/pure_math.das" "02_heap/heap_demo.das" "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT (nano): ${_das_name}" VERBATIM diff --git a/tests-cpp/big/standalone_ctx/CMakeLists.txt b/tests-cpp/big/standalone_ctx/CMakeLists.txt index 4879c2f353..fbffdad707 100644 --- a/tests-cpp/big/standalone_ctx/CMakeLists.txt +++ b/tests-cpp/big/standalone_ctx/CMakeLists.txt @@ -17,6 +17,7 @@ add_custom_command( "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT: standalone_init_fixture.das" VERBATIM @@ -36,6 +37,28 @@ add_test(NAME standalone_ctx COMMAND test_standalone_ctx set_tests_properties(standalone_ctx PROPERTIES LABELS "big") add_dependencies(test-big test_standalone_ctx) +# The same fixture driven through its C API, with the C half of the generated header compiled by +# a C compiler: a header that is not valid C, a layout that disagrees with das, or a thunk that +# loses a value fails here. The C++ twin above shares the generated source, so both APIs are +# proven against one emission. +add_executable(test_standalone_capi + test_standalone_capi.c + "${STANDALONE_CTX_GEN}/standalone_init_fixture.das.cpp") +target_link_libraries(test_standalone_capi PRIVATE + libDaScript ${SRC_LIBRARIES} ${DAS_MODULES_LIBS}) +target_include_directories(test_standalone_capi PRIVATE "${STANDALONE_CTX_GEN}" ${NEED_MODULES_PATH}) +set_target_properties(test_standalone_capi PROPERTIES + FOLDER "tests-cpp/big" + C_STANDARD 11 + C_STANDARD_REQUIRED ON + LINKER_LANGUAGE CXX) +SETUP_CPP11(test_standalone_capi) + +add_test(NAME standalone_capi COMMAND test_standalone_capi + WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) +set_tests_properties(standalone_capi PROPERTIES LABELS "small") +add_dependencies(test-small test_standalone_capi) + # Two contexts with different C++ module sets in one binary - the dasHV + fio example and a # fio-only fixture - share one module registry lifetime. Small-labelled: a second owner of the # shutdown, or an unbalanced Initialize, fails at exit and nothing else would notice. @@ -54,6 +77,7 @@ if(NOT DAS_HV_DISABLED) "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT: ${_das_name}" VERBATIM diff --git a/tests-cpp/big/standalone_ctx/standalone_init_fixture.das b/tests-cpp/big/standalone_ctx/standalone_init_fixture.das index 75ed20f363..2e76d7bc9b 100644 --- a/tests-cpp/big/standalone_ctx/standalone_init_fixture.das +++ b/tests-cpp/big/standalone_ctx/standalone_init_fixture.das @@ -2,6 +2,7 @@ options gen2 options stack = 262144 require standalone_init_dep +require daslib/export_c var g_stamp = 0 @@ -115,3 +116,8 @@ def sum_generator(n : int) : int { } return s } + +[export_c(name = "renamed_sum")] +def sum_under_another_name(a, b : int) : int { + return a + b +} diff --git a/tests-cpp/big/standalone_ctx/test_standalone_capi.c b/tests-cpp/big/standalone_ctx/test_standalone_capi.c new file mode 100644 index 0000000000..e7854c0cd6 --- /dev/null +++ b/tests-cpp/big/standalone_ctx/test_standalone_capi.c @@ -0,0 +1,65 @@ +#include "standalone_init_fixture.das.h" + +#include +#include + +static int failures = 0; + +static void expect ( const char * what, int have, int want ) { + if ( have != want ) { + printf("%s = %d, expected %d\n", what, have, want); + failures ++; + } +} + +int main ( void ) { + standalone_init_fixture_ctx * ctx = standalone_init_fixture_create(); + if ( !ctx ) { + printf("create failed: %s\n", standalone_init_fixture_last_error(NULL)); + return 1; + } + + expect("get_first()", standalone_init_fixture_get_first(ctx), 31); + expect("get_second()", standalone_init_fixture_get_second(ctx), 2); + expect("get_init_fn_stamp()", standalone_init_fixture_get_init_fn_stamp(ctx), 3); + expect("get_reads_forward()", standalone_init_fixture_get_reads_forward(ctx), 100); + expect("get_later()", standalone_init_fixture_get_later(ctx), 7); + expect("get_shared_total()", standalone_init_fixture_get_shared_total(ctx), 6); + + standalone_init_fixture_Pair made; + memset(&made, 0xAA, sizeof(made)); + standalone_init_fixture_make_pair(ctx, 3, 4, &made); + expect("make_pair(3,4).a", made.a, 3); + expect("make_pair(3,4).b", made.b, 4); + expect("pair_sum(make_pair(3,4))", standalone_init_fixture_pair_sum(ctx, &made), 7); + + standalone_init_fixture_Pair mine; + mine.a = 40; + mine.b = 2; + expect("pair_sum(a Pair this C host built)", standalone_init_fixture_pair_sum(ctx, &mine), 42); + + expect("flip(on)", standalone_init_fixture_flip(ctx, standalone_init_fixture_Mode_on), + standalone_init_fixture_Mode_off); + expect("flip(off)", standalone_init_fixture_flip(ctx, standalone_init_fixture_Mode_off), + standalone_init_fixture_Mode_on); + + standalone_init_fixture_Outer nested; + nested.inner.weight = 9; + nested.tag = 5; + expect("outer_weight(a struct holding a struct)", + standalone_init_fixture_outer_weight(ctx, &nested), 14); + + expect("head_value() on a null safe-navigation", standalone_init_fixture_head_value(ctx), 41); + + expect("renamed_sum(20,22) through the [export_c(name=...)] symbol", + standalone_init_fixture_renamed_sum(ctx, 20, 22), 42); + + if ( standalone_init_fixture_last_error(ctx) ) { + printf("a call raised: %s\n", standalone_init_fixture_last_error(ctx)); + failures ++; + } + + standalone_init_fixture_destroy(ctx); + printf(failures ? "standalone_capi: %d failure(s)\n" : "standalone_capi: ok\n", failures); + return failures ? 1 : 0; +} diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index 6534ff120f..8f7eaf421c 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -37,6 +37,7 @@ if(TARGET libDasModuleHV AND TARGET libDasModuleStdDlg AND TARGET libDasModuleSt "${PROJECT_SOURCE_DIR}/utils/aot/main.das" "${PROJECT_SOURCE_DIR}/daslib/aot_standalone.das" "${PROJECT_SOURCE_DIR}/daslib/aot_cpp.das" + "${PROJECT_SOURCE_DIR}/daslib/c_api_header.das" WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}" COMMENT "Standalone AOT (full runtime): watchdog" VERBATIM From aa2d03a5486b00cba0fcbd36044a78695fb978da Mon Sep 17 00:00:00 2001 From: Churkin Aleksey Date: Wed, 9 Sep 2026 17:43:08 +0300 Subject: [PATCH 2/4] aot: a standalone context names its namespace, its globals and its structures [skip ci] Compiling the standalone output of every test in the tree - 2031 files - found four emitter defects the AOT build never sees, because only a standalone context names its main module and visits that module by itself. The context name was the file stem verbatim, and a stem is not an identifier: while.das opened `namespace while {`, struct.das `namespace struct {`. It goes through cpp_context_ident now, the sanitizer the C prefix already used. The stem is still needed - a build system predicts the generated file names from the input path and cannot be told they were sanitized - so StandaloneContextCfg carries both: context_name is the identifier, file_stem is what the files are called. A safe-navigation field read emits a member pointer beside the type it belongs to, and the two disagreed: das_safe_navigation The type comes from describeCppType, which resolves the entry module's types through g_aot_main_module_name; the member pointer read _module.name, which is empty for every type a script declares itself. Both go through aotModuleName. A global of handled type refers to its AnnotationInfo by the name the debug-info dump minted, and the dump cleared that table on the way OUT - so the reference came back as a bare `&`, since the globals are written after it. The table resets on the way in instead, and the globals' walk registers what only it reaches: writeHandledAnnotations iterates types, structs and functions, never globals. Structures were emitted in declaration order, so a by-value field could name a structure not yet defined. Program::visitModule already sorts - topoSortStructures, which regular AOT asks for through visit(program, adapter, true) - and the visit_module binding did not expose the flag. It does now, and the standalone emitter passes it. Eighteen test files can be turned into a standalone context that compiles as a result. The sweep reports no regression: nothing that compiled before stopped, and the set of files standalone AOT refuses is unchanged. standalone_init_fixture carries both shapes now, so the standalone_capi and standalone_ctx builds are the gate: Outer embeds an Inner declared after it, and a safe-navigation read emits a member pointer. Revert either fix and neither binary compiles. --- daslib/ARCHITECTURE_EMIT.md | 22 ++++++++++++++++ daslib/aot_cpp.das | 5 ++-- src/builtin/module_builtin_ast_adapters.cpp | 14 ++++++++++ .../standalone_init_fixture.das | 26 +++++++++++++++++++ 4 files changed, 65 insertions(+), 2 deletions(-) diff --git a/daslib/ARCHITECTURE_EMIT.md b/daslib/ARCHITECTURE_EMIT.md index 23819ff21d..906cd73ffd 100644 --- a/daslib/ARCHITECTURE_EMIT.md +++ b/daslib/ARCHITECTURE_EMIT.md @@ -52,8 +52,30 @@ Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across dasHV takes `rtti_core`). A module missing from the daslib list still registers, only in the dependencies-first pass that follows; a module added to the C++ side joins the list. +- **The AnnotationInfo table resets at the START of the debug-info dump, not its end.** The + globals' `VarInfo`s are written after that dump and a handled global's info refers to an + `AnnotationInfo` by the name the dump minted, so clearing on the way out left `&` with nothing + after it. Only that walk reaches a global's annotation - `writeHandledAnnotations` iterates + types, structs and functions. +- **A member pointer is qualified with `aotModuleName`, never the raw module name.** The main + module is unnamed, so `_module.name` is empty for every type a script declares itself, while + `describeCppType` resolves those types through `g_aot_main_module_name`. Where one emitter writes + both - `das_safe_navigation` - they must agree, or the type argument names the + context's namespace while the member pointer names nothing. Only a standalone context sets a + main-module name, so regular AOT never sees it. + ## 6. aot_standalone +- **The entry module's structures are visited SORTED.** `visitModule` takes `sortStructures`, + which runs `topoSortStructures` so a by-value field's structure is complete before the structure + holding it; regular AOT passes it through `visit(program, adapter, true)` and a standalone + context, which visits the entry module by itself, has to ask for it too. Declaration order is the + author's, and nothing else re-derives it. +- **The context name is an identifier; the file stem is not the same string.** A stem reaches C++ + as a namespace and C as a symbol prefix, so `while.das` or `3d-math.das` would open + `namespace while {`. `context_name` is the stem through `cpp_context_ident`, while `file_stem` + keeps the raw name - a build system predicts the generated file names from the input path and + cannot be told they were sanitized. - **The generated constructor IS the init protocol** - a standalone context never calls `Context::runInitScript`, so the ctor reproduces its observable semantics inline: `memset(context.globals, 0, context.getGlobalSize())` mirrors runInitScript's globals diff --git a/daslib/aot_cpp.das b/daslib/aot_cpp.das index 9ce915f1f9..83f194ec09 100644 --- a/daslib/aot_cpp.das +++ b/daslib/aot_cpp.das @@ -854,6 +854,7 @@ class public AotDebugInfoHelper { def str() { return build_string() $(var writer) { verify(info2Name.empty() && info2TypeName.empty()) + annInfoNames |> clear() helper |> debug_helper_iter_structs($(_name, ti) { write(writer, "extern StructInfo {structInfoName(ti)};\n"); }); @@ -896,7 +897,6 @@ class public AotDebugInfoHelper { write(writer, "\}\n\n") info2Name.clear(); info2TypeName.clear(); - annInfoNames |> clear() } } @@ -2479,7 +2479,8 @@ class public CppAot : AstVisitor { } write(*ss, ", {vtype.get_variant_field_offset(field.fieldIndex)}, {field.fieldIndex}>::get("); } else { - let mod_name = (vtype.structType._module.name.empty() ? "" : string(vtype.structType._module.name) + "::"); + let ns = aotModuleName(vtype.structType._module); + let mod_name = (ns |> empty() ? "" : "{ns}::"); write(*ss, ",&{mod_name}{aotStructName(vtype.structType)}::{aotFieldName(string(field.name))}>::get("); } } diff --git a/src/builtin/module_builtin_ast_adapters.cpp b/src/builtin/module_builtin_ast_adapters.cpp index 42615a8a55..8239ebce5e 100644 --- a/src/builtin/module_builtin_ast_adapters.cpp +++ b/src/builtin/module_builtin_ast_adapters.cpp @@ -2541,6 +2541,17 @@ namespace das { program->visitModule(*adapter, module); } + void astVisitModuleWithSort ( smart_ptr_raw program, VisitorAdapter * adapter, + Module* module, bool sortStructures, Context * context, LineInfoArg * line_info ) { + if (!adapter) + context->throw_error_at(line_info, "adapter is required"); + if (!program) + context->throw_error_at(line_info, "program is required"); + if (!module) + context->throw_error_at(line_info, "module is required"); + program->visitModule(*adapter, module, false, sortStructures); + } + void astVisitModulesInOrder ( smart_ptr_raw program, VisitorAdapter * adapter, Context * context, LineInfoArg * line_info ) { if (!adapter) context->throw_error_at(line_info, "adapter is required"); @@ -2611,6 +2622,9 @@ namespace das { addExtern(*this, lib, "visit_modules", SideEffects::accessExternal, "astVisitModulesInOrder") ->args({"program","adapter","context","line"}); + addExtern(*this, lib, "visit_module", + SideEffects::accessExternal, "astVisitModuleWithSort") + ->args({"program","adapter","module","sortStructures","context","lineInfo"}); addExtern(*this, lib, "visit_module", SideEffects::accessExternal, "astVisitModule") ->args({"program","adapter","module","context","line"}); diff --git a/tests-cpp/big/standalone_ctx/standalone_init_fixture.das b/tests-cpp/big/standalone_ctx/standalone_init_fixture.das index 2e76d7bc9b..af4e807d0f 100644 --- a/tests-cpp/big/standalone_ctx/standalone_init_fixture.das +++ b/tests-cpp/big/standalone_ctx/standalone_init_fixture.das @@ -86,6 +86,32 @@ def flip(m : Mode) : Mode { return m == Mode.on ? Mode.off : Mode.on } +struct Outer { + inner : Inner + tag : int +} + +struct Inner { + weight : int +} + +[export] +def outer_weight(o : Outer) : int { + return o.inner.weight + o.tag +} + +struct Node { + value : int + next : Node? +} + +var private g_node : Node? + +[export] +def head_value : int { + return g_node?.value ?? 41 +} + [export] def apply_lambda(x : int) : int { let add3 = @(v : int) : int => v + 3 From 385b1b8465a1b798d8c8b6803b1b53be69b6c85b Mon Sep 17 00:00:00 2001 From: Churkin Aleksey Date: Wed, 9 Sep 2026 02:00:23 +0300 Subject: [PATCH 3/4] dasbind: a bound extern is not unreachable at simulate [skip ci] `daslang -aot script.das out.cpp -aot-macros` core dumped whenever the script's module graph carried a dasbind [extern], which covers every dasLLVM binding and any user's own: Should be unreachable. We handled it in transformCall. Failed on: c_strlen. -aot-macros sets export_all beside the aot_module it already sets, which takes the force-marking branch in parseDaScript: every function of every module is marked used and removeUnusedSymbols is skipped. That reaches the [extern] declarations, which have no body of their own, and simulating one hit a DAS_FATAL_ERROR. The annotation claimed a stronger invariant than it needs. transformCall rewrites every CALL to a bound extern, which is what matters; the function itself being simulated is harmless, because nothing calls it. The base FunctionAnnotation::simulate already returns nullptr, so the override - and the 35 commented-out lines under it - just go. --- src/builtin/module_builtin_dasbind.cpp | 37 -------------------------- 1 file changed, 37 deletions(-) diff --git a/src/builtin/module_builtin_dasbind.cpp b/src/builtin/module_builtin_dasbind.cpp index 574eb8c9e8..92d9cdf354 100644 --- a/src/builtin/module_builtin_dasbind.cpp +++ b/src/builtin/module_builtin_dasbind.cpp @@ -563,43 +563,6 @@ FastCallWrapper getExtraWrapper ( int nargs, int res, int perm ) { } return newCallExpr; } - virtual SimNode * simulate ( Context * /*context*/, Function * fun, const AnnotationArgumentList & /*args*/, string & /*err*/ ) override { - if (is_in_completion()) return nullptr; - DAS_FATAL_ERROR("Should be unreachable. We handled it in transformCall. Failed on: %s.", fun->name.c_str()); - // // All validation is in apply(). This path is only reached for late/opengl functions. - // auto [is_ok, ba] = parseExternArgs(args, err); - // DAS_ASSERTF(is_ok, "Should have failed in apply"); - // void * libhandle = nullptr; - // if ( !ba.library.empty() ) { - // libhandle = bindDynamicLibrary(ba.library); - // if ( !libhandle && !ba.late && ba.api!=ApiType::api_opengl ) { - // err = "can't load library " + ba.library; - // return nullptr; - // } - // } - // void * funptr = nullptr; - // if ( !ba.late ) { - // if ( ba.api==ApiType::api_opengl ) { - // funptr = openGlGetFunctionAddress(ba.fn_name.c_str()); - // } - // if ( !funptr ) { - // funptr = getFunctionAddress(libhandle, ba.fn_name.c_str()); - // } - // if ( !funptr ) { - // err = "can't find function " + ba.fn_name + " in library " + ba.library; - // return nullptr; - // } - // } - // uint64_t code = lateBind(ba.fn_name, ba.library, funptr); - // auto wrp = computeWrapper(fun); - // if ( ba.api==ApiType::api_opengl ) { - // return context->code->makeNode(fun->at,code,wrp,funptr); - // } - // if ( ba.late ) { - // return context->code->makeNode(fun->at,code,wrp,funptr); - // } - // return context->code->makeNode(fun->at,code,wrp,funptr); - } #endif }; From 82adf418b384efaa7f3b5ff58e672454b348e375 Mon Sep 17 00:00:00 2001 From: Churkin Aleksey Date: Wed, 9 Sep 2026 02:00:58 +0300 Subject: [PATCH 4/4] jit: daslang -lib emits a C-ABI native library [skip ci] The JIT could already produce an exe and an AOT object; a host that only calls one script had neither. -lib writes a native shared library (or a static archive under --jit-lib-static) plus a C header, and links no daslang API into the host. Nothing about the C surface is new here: daslib/c_api_header already decides what may cross and writes the header for a standalone context, so -lib reuses that describer whole and only emits the other side. A JIT library and an AOT standalone context therefore declare the same C API, name the same functions and skip the same signatures, and the C++ half a standalone context appends stays a proxy over C entry points either backend can supply. Which functions cross is one bit, Function.flags.exports, read three ways. The default accepts only what carries [export_c]; --jit-lib-export-marked accepts the bit however it was set, so a script already marking its host API with [export] needs no second annotation; -lib-export-all is that same acceptance plus policies.export_public_functions, which has to be a policy because MarkSymbolUse consumes it before infer, where no option can reach. Only export-all may skip an unspellable signature with a warning - the other two name a function on purpose. [export_c] reaches a library source with no require because daslib/export_c is !inscope and daslib/just_in_time requires it, so -lib, -jit and -exe all carry the annotation; a compile with no JIT - the linter, the AOT pass - asks for it by name. Codegen stays in LlvmJitMode.EXE, which is what makes the exe startup shareable: this hoists it into emitters -exe and -lib both call, and only the entry shape and the artifact differ. Each export gets an extern "C" thunk that packs its arguments into a frame and hands it to a trampoline through jit_lib_invoke_guarded, so a das panic reaches the caller as a zero return with its text in

_last_error, never as an unwind through the C boundary. A structure result is built in an alloca of its own alignment, not in a frame field: an in-struct [N x i8] slot aligns to 1, and the body then stores through it at the structure's real alignment. Symbols carry the prefix of the OUTPUT name, so they always match the header they are declared in. The context reaches a library's init as a parameter rather than as the create call's result, so the sealed name lookups are adopted at the entry block's head - still before every registration call. Three facts about a library's runtime shape the entry points, and ARCHITECTURE.md#lib-runtime-scope records them. The environment is thread-local while the registration behind it happens once, so every later thread binds the first one's environment. One daslang runtime fits in a process and every artifact shares it: registrations go through jit_register_module_once, and a library that finds a populated environment is a guest - it neither re-initializes the runtime nor drains what it did not create, so a library loads into a daslang host and beside another library. And nothing finds a jitted SimFunction's shutdown functions, so

_destroy emits them itself. -lib carries no CodeOfPolicies field of its own: llvm_jit declares jit_lib and jit_lib_export_marked as module options, run_jit reads them off prog._options the way it already reads jit_split_modules, and main.cpp pushes jit_lib between compile and simulate. examples/c_api_library builds one library each way and drives all three at once through dasbind, which is also the coexistence proof. Its host.c includes nothing but the generated header and links nothing but the library. tests/jit_tests/jit_lib runs that example, so the -jit sweep covers it. Loading each emitted library from a das host through dasbind - the whole test tree, one library at a time - found two ways a guest still fought its host. Registration went through jit_register_module_once at every site the require walk reaches, but not at the one the extern collector reaches: a module named by no require, pulled in because a function calls one of its externs, got a direct jit_register_Module_* and aborted with `Module 'ast_core' already created`. Both sites use the shim now. And jit_finalize_dynamic_modules fatals on anything still unloadable, which is right for an exe that owns its process and wrong for a library: the pending list it inherits is the HOST's, so a module the host could not load killed the process over a library that never asked for it. A guest retries and reports instead; a module the library does need still surfaces, as a null

_create with the reason. sec. 1.3 moves to ARCHITECTURE_LIB.md - with the library's runtime rules the main document was past the 300-line cap. The startup is shared rather than copied: inject_main keeps its body and calls the four emitters this hoists out, which the library calls too. What has no exe counterpart is the C boundary itself - an exe emits one main that takes argc/argv and calls one entry, a library emits a thunk per export, each with an argument frame, a trampoline and a guarded call, because runWithCatch takes one fixed signature and a das panic must not unwind into C. The library's own codegen lives in jit_standalone.das: the thunks, their argument frames and trampolines, the four C entry points and inject_lib. It requires llvm_exe for the four startup emitters both backends share, so the dependency runs one way and llvm_exe keeps inject_main and nothing about libraries. tests/jit_tests/jit_lib drives the AOT standalone fixture through -lib with --jit-lib-export-marked and asserts the values its C twin asserts, so one file proves both tiers: the two backends declare the same C API from the same source. --- .gitignore | 3 + CLAUDE.md | 2 +- daslib/ARCHITECTURE_CAPI.md | 56 ++- daslib/just_in_time.das | 1 + doc/source/reference/embedding/c_api.rst | 6 + doc/source/reference/language/annotations.rst | 5 +- ...ructure_annotation-rtti-CodeOfPolicies.rst | 1 + examples/c_api_library/greetings.das | 17 + examples/c_api_library/main.das | 139 ++++++ examples/c_api_library/shapes.das | 24 + examples/c_api_library/units.das | 17 + include/daScript/simulate/code_of_policies.h | 1 + install/CLAUDE.md | 2 +- modules/dasLLVM/.das_module | 3 +- modules/dasLLVM/ARCHITECTURE.md | 21 +- modules/dasLLVM/ARCHITECTURE_LIB.md | 60 +++ modules/dasLLVM/README.md | 41 ++ modules/dasLLVM/REVIEW.md | 8 + modules/dasLLVM/daslib/jit_standalone.das | 405 ++++++++++++++++ modules/dasLLVM/daslib/llvm_dll_utils.das | 30 +- modules/dasLLVM/daslib/llvm_exe.das | 230 +++++---- modules/dasLLVM/daslib/llvm_jit.das | 4 +- modules/dasLLVM/daslib/llvm_jit_cli.das | 12 + modules/dasLLVM/daslib/llvm_jit_common.das | 57 +-- modules/dasLLVM/daslib/llvm_jit_run.das | 52 +- skills/daslang/references/cli-and-config.md | 3 + .../daslang/references/modules-and-stdlib.md | 6 +- src/ast/ast_export.cpp | 9 + src/builtin/module_builtin_rtti.cpp | 1 + src/builtin/module_jit.cpp | 119 +++++ tests-cpp/small/test_jit_lib_guard.cpp | 110 +++++ tests/jit_tests/jit_lib.das | 446 ++++++++++++++++++ utils/daslang/main.cpp | 24 + 33 files changed, 1740 insertions(+), 175 deletions(-) create mode 100644 examples/c_api_library/greetings.das create mode 100644 examples/c_api_library/main.das create mode 100644 examples/c_api_library/shapes.das create mode 100644 examples/c_api_library/units.das create mode 100644 modules/dasLLVM/ARCHITECTURE_LIB.md create mode 100644 modules/dasLLVM/daslib/jit_standalone.das create mode 100644 tests-cpp/small/test_jit_lib_guard.cpp create mode 100644 tests/jit_tests/jit_lib.das diff --git a/.gitignore b/.gitignore index 6782e960da..ce6987d140 100644 --- a/.gitignore +++ b/.gitignore @@ -199,3 +199,6 @@ modules/dasLLAMA/benchmarks/asr/_pybench_rows.txt # python bytecode, anywhere __pycache__/ site/files/profile_results_*.json + +# examples/c_api_library builds its three libraries here +examples/c_api_library/_out/ diff --git a/CLAUDE.md b/CLAUDE.md index 73be925bd5..2480a11c91 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -96,7 +96,7 @@ Task-specific instructions are split into skill files under `skills/`. You MUST | `skills/internal/documentation_rst.md` | Editing RST in `doc/source/`, `//!` doc-comments in `daslib/*.das`, tutorial RST pages | | `skills/internal/tutorials.md` | Anything that looks like a tutorial - they live under `/tutorials//`, NEVER `modules//tutorial/` | | `skills/internal/tutorial_prose.md` | WRITING or revising general-reader doc/tutorial prose (`documentation_rst.md` is mechanics, this is the words) | -| `skills/cpp_integration.md` | Embedding daslang in C++; binding types/functions/enums; shipping without the compiler (`libDaScriptNano`, or a standalone context on the full runtime) | +| `skills/cpp_integration.md` | Embedding daslang in C++; binding types/functions/enums; shipping without the compiler (`libDaScriptNano`, or a standalone context on the full runtime); calling daslang from C (`daslang -lib`) | | `skills/internal/cpp_codebase_notes.md` | Working on daslang's own C++ - where inference/builtins/errors/parser live, AST function flags | | `skills/internal/clang_bind_build.md` | Enabling `dasClangBind` / bumping the libclang SDK / running any `bind_*.das` self-binder | | `skills/daslib_modules.md` | Working with `daslib/` modules or extending the stdlib | diff --git a/daslib/ARCHITECTURE_CAPI.md b/daslib/ARCHITECTURE_CAPI.md index 6f26849ef1..5a4c12743e 100644 --- a/daslib/ARCHITECTURE_CAPI.md +++ b/daslib/ARCHITECTURE_CAPI.md @@ -1,31 +1,40 @@ # daslib architecture notes - the generated C API Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across the family. -The header this section describes is the one a standalone context ships. +The header this section describes is the one BOTH backends write: `aot_standalone` for a +standalone context, `inject_lib` for `daslang -lib`. ## 30. c_api_header - **One module writes every generated header, and the C API is the only calling path.** The C - declarations always come first; the C++ half (`CppApi`) is appended under `#ifdef __cplusplus`, - and each of its functions is an `inline` call to the C entry point beside it - so a C host and a - C++ host include one file, neither sees the other's half, and there is exactly one - implementation to keep correct, and no `aot_cpp` dependency: a proxy only ever spells a - C-representable signature, whose C++ form is mechanical - a struct or enum by its das name, - everything else by its C one. `aot_standalone` emits the matching `extern "C"` bodies off - this describer, so the header and the source cannot disagree about what crosses or what it is - called. -- **`Function.flags.exports` is the whole selection truth.** `[export_c]` (`ExportCAnnotation`, - `daslib/export_c.das` - a das `[function_macro]`, so the C surface is decided entirely in - daslang) sets it, and so does `[export]`, so a standalone context takes the bit however it was - set. Accepting the bit however it was set is what licenses skipping an unspellable signature with - a warning; a function that ASKED for C with `[export_c]` and cannot cross is an error. No second - list can drift from the bit. Whether a signature CAN cross is decided here instead, after infer, - because argument types do not exist when an annotation applies. -- **Refusal is per stage, not per module**: `collect_c_exports` returns its rejections and - logs its skips. An `[export_c]` that cannot cross comes back for the caller to report - - `macro_error` during compilation, the jit error log during codegen - so this module needs - no `ProgramPtr` and no reporting policy of its own. A merely-public function is skipped - with a warning naming it and the type. + declarations come first; the C++ half (`CppApi`) is appended under `#ifdef __cplusplus`, each of + its functions an `inline` call to the C entry point beside it - so both hosts include one file, + neither sees the other's half, one implementation stays correct, and no `aot_cpp` is needed: a + proxy only spells a C-representable signature, whose C++ form is mechanical. `aot_standalone` + emits the matching `extern "C"` bodies and `inject_lib` the matching thunks, both off this + describer, so JIT and AOT cannot disagree about what crosses or what it is called. +- **`Function.flags.exports` is the whole selection truth.** `[export_c]` + (`ExportCAnnotation`, `daslib/export_c.das` - a das `[function_macro]`, so the C surface is + decided entirely in daslang) sets it, and + `policies.export_public_functions` sets it for every public entry-module function under + `-lib-export-all` (`MarkSymbolUse::exportPublicFunctions`, `src/ast/ast_export.cpp`). The + selection forms are therefore one bit read several ways: `-lib` alone accepts only what carries + the annotation, `--jit-lib-export-marked` and a standalone context accept the bit however it was + set (so `[export]` selects), and `-lib-export-all` is that same acceptance plus the marking + policy. Accepting the bit however set is what licenses skipping an unspellable signature with a + warning; one that ASKED for C with `[export_c]` and cannot cross is an error. Whether a signature + CAN cross is decided here, after infer, because argument types do not exist at annotation time. +- **`[export_c]` reaches a library source with no `require` because `daslib/export_c` is + `!inscope`.** That marker sets `visibleEverywhere`, which `Module::isVisibleDirectly` honors + ahead of the require map - the mechanism that makes `daslib/builtin.das` universal. Being visible + still needs the module LOADED, and `daslib/just_in_time.das` - injected whenever the JIT is on - + requires it, so `-lib`, `-jit` and `-exe` carry the annotation for free; a compile with no JIT + needs `require daslib/export_c`. It lives in its own module rather than in `c_api_header` because + that require costs `ast_boost` alone, not the header emitter's whole graph. +- **Refusal is per stage, not per module**: `collect_c_exports` returns its rejections and logs + its skips. An `[export_c]` that cannot cross comes back for the caller to report - `macro_error` + during compilation, the jit error log during codegen - so this module needs no `ProgramPtr` and + no reporting policy of its own. - **The scalar widths and the vector layouts are C++-side facts this emitter mirrors.** `bool` is one byte (the `static_assert(sizeof(bool)==1)` in `getTypeBaseSize`, `src/simulate/debug_info.cpp`), so das `bool` meets C as `bool`. `float3` is `{x, y, z}` at @@ -55,6 +64,5 @@ The header this section describes is the one a standalone context ships. all - while a representable target is defined, so a host can read through the pointer. A cycle terminates either way, and a pointer to something C cannot even name degrades to `void *` rather than refusing the function. -- **A `fixed_array` argument crosses (as `const T *`, which is what the das ABI already - passes) but a `fixed_array` result does not** - that would be a CMRES of an array, a - pointer nothing on the C side sizes. +- **A `fixed_array` argument crosses (as `const T *`, which the das ABI already passes) but a + `fixed_array` result does not** - that is a CMRES of an array, a pointer nothing in C sizes. diff --git a/daslib/just_in_time.das b/daslib/just_in_time.das index 2750391988..7524b94d5d 100644 --- a/daslib/just_in_time.das +++ b/daslib/just_in_time.das @@ -13,3 +13,4 @@ module just_in_time shared private */ require llvm/daslib/llvm_macro +require daslib/export_c diff --git a/doc/source/reference/embedding/c_api.rst b/doc/source/reference/embedding/c_api.rst index 12bfc739ea..68e81e1e78 100644 --- a/doc/source/reference/embedding/c_api.rst +++ b/doc/source/reference/embedding/c_api.rst @@ -37,6 +37,12 @@ functionality. The C API covers the most common embedding scenarios but does not expose every C++ feature (e.g. class adapters, custom annotations). +This API embeds the daslang *compiler*: the host loads sources, compiles +them, and calls what it finds by name. A host that only needs to call +one fixed script wants ``daslang -lib`` instead — it compiles that script +to a native library with its own generated C header, and the host links +no daslang API at all. See :ref:`embedding_advanced` (C libraries). + Linking ======= diff --git a/doc/source/reference/language/annotations.rst b/doc/source/reference/language/annotations.rst index 31111aa93f..b06c15f9a3 100644 --- a/doc/source/reference/language/annotations.rst +++ b/doc/source/reference/language/annotations.rst @@ -66,8 +66,9 @@ Lifecycle } ``[export_c]`` - An ``[export]`` that also names the function to C: a standalone context declares it in the C - half of the header it generates, so a C host calls it with no daslang API of its own. Needs + An ``[export]`` that ``daslang -lib`` additionally surfaces as a C function in the header it + generates. ``-lib``, ``-jit`` and ``-exe`` carry the annotation themselves, so a library source + needs no ``require``; a plain interpreter or AOT compile of the same file needs ``require daslib/export_c``. The function keeps working in every tier. Its signature has to be one C can spell: scalars, ``string``, raw pointers, enumerations, plain structures, the ``float2``..``uint4`` and ``range`` families, and ``fixed_array`` arguments; ``array``, diff --git a/doc/source/stdlib/handmade/structure_annotation-rtti-CodeOfPolicies.rst b/doc/source/stdlib/handmade/structure_annotation-rtti-CodeOfPolicies.rst index 084f1782e1..3a5b89501b 100644 --- a/doc/source/stdlib/handmade/structure_annotation-rtti-CodeOfPolicies.rst +++ b/doc/source/stdlib/handmade/structure_annotation-rtti-CodeOfPolicies.rst @@ -14,6 +14,7 @@ Whether we are in lint-check mode (standalone linters set this so modules can ad Skip Program::lint() entirely (as if every module set ``options lint = false``). Skip the Module::Initialize() assert in compileDaScript (for environments initialized later, e.g. dynamic-module discovery). Export all functions and global variables. +Treat every public, non-generic function of the entry module as [export] (daslang -lib -lib-export-all). If not set, we recompile main module each time. Keep context alive after main function. Whether to use very safe context (delete of data is delayed, to avoid table[foo]=table[bar] lifetime bugs). diff --git a/examples/c_api_library/greetings.das b/examples/c_api_library/greetings.das new file mode 100644 index 0000000000..71243a32c6 --- /dev/null +++ b/examples/c_api_library/greetings.das @@ -0,0 +1,17 @@ +options gen2 +options indenting = 4 + + +[export] +def greet(who : string) : string { + return "hola {who}" +} + +[export] +def bump(n : int) : int { + return n + 1 +} + +def internal_seed() : int { + return 41 +} diff --git a/examples/c_api_library/main.das b/examples/c_api_library/main.das new file mode 100644 index 0000000000..521bf74fad --- /dev/null +++ b/examples/c_api_library/main.das @@ -0,0 +1,139 @@ +options gen2 +options indenting = 4 + + +require dasbind +require daslib/fio +require daslib/safe_addr +require strings + +let DIR = "examples/c_api_library" +let OUT = "examples/c_api_library/_out" + + +struct Vec2 { + x : float + y : float +} + +[extern(cdecl, late, name="shapes_create", + linux_library="examples/c_api_library/_out/shapes.so", + macos_library="examples/c_api_library/_out/shapes.dylib", + windows_library="examples/c_api_library/_out/shapes.dll")] +def shapes_create() : void? {} + +[extern(cdecl, late, name="shapes_destroy", + linux_library="examples/c_api_library/_out/shapes.so", + macos_library="examples/c_api_library/_out/shapes.dylib", + windows_library="examples/c_api_library/_out/shapes.dll")] +def shapes_destroy(ctx : void?) : void {} + +[extern(cdecl, late, name="shapes_last_error", + linux_library="examples/c_api_library/_out/shapes.so", + macos_library="examples/c_api_library/_out/shapes.dylib", + windows_library="examples/c_api_library/_out/shapes.dll")] +def shapes_last_error(ctx : void?) : string {} + +[extern(cdecl, late, name="shapes_dot", + linux_library="examples/c_api_library/_out/shapes.so", + macos_library="examples/c_api_library/_out/shapes.dylib", + windows_library="examples/c_api_library/_out/shapes.dll")] +def shapes_dot(ctx : void?; a, b : Vec2?#) : float {} + +[extern(cdecl, late, name="units_create", + linux_library="examples/c_api_library/_out/units.so", + macos_library="examples/c_api_library/_out/units.dylib", + windows_library="examples/c_api_library/_out/units.dll")] +def units_create() : void? {} + +[extern(cdecl, late, name="units_destroy", + linux_library="examples/c_api_library/_out/units.so", + macos_library="examples/c_api_library/_out/units.dylib", + windows_library="examples/c_api_library/_out/units.dll")] +def units_destroy(ctx : void?) : void {} + +[extern(cdecl, late, name="units_celsius_to_fahrenheit", + linux_library="examples/c_api_library/_out/units.so", + macos_library="examples/c_api_library/_out/units.dylib", + windows_library="examples/c_api_library/_out/units.dll")] +def units_celsius_to_fahrenheit(ctx : void?; c : float) : float {} + +[extern(cdecl, late, name="greetings_create", + linux_library="examples/c_api_library/_out/greetings.so", + macos_library="examples/c_api_library/_out/greetings.dylib", + windows_library="examples/c_api_library/_out/greetings.dll")] +def greetings_create() : void? {} + +[extern(cdecl, late, name="greetings_destroy", + linux_library="examples/c_api_library/_out/greetings.so", + macos_library="examples/c_api_library/_out/greetings.dylib", + windows_library="examples/c_api_library/_out/greetings.dll")] +def greetings_destroy(ctx : void?) : void {} + +[extern(cdecl, late, name="greetings_greet", + linux_library="examples/c_api_library/_out/greetings.so", + macos_library="examples/c_api_library/_out/greetings.dylib", + windows_library="examples/c_api_library/_out/greetings.dll")] +def greetings_greet(ctx : void?; who : string) : string {} + +def private run(cmd : string; var lines : array) : int { + var rc : int + unsafe { + rc = popen_timeout("{cmd} 2>&1", 600.0) $(f) { + if (f == null) { + return + } + while (!feof(f)) { + let ln = strip(fgets(f)) + if (!(ln |> empty())) { + lines |> push("{ln}") + } + } + } + } + return rc +} + + +def private build_lib(bin, stem, extra : string) : bool { + var lines : array + if (run("\"{bin}\" -lib \"{DIR}/{stem}.das\" -output \"{OUT}/{stem}\" {extra}", lines) != 0) { + for (ln in lines) { + print("{ln}\n") + } + print("FAILED to build {stem}\n") + return false + } + print("built {OUT}/{stem} ({extra |> empty() ? "[export_c] only" : extra})\n") + return true +} + + +[export] +def main() { + let args <- get_command_line_arguments() + let bin = args[0] + mkdir_rec(OUT) + + if (!build_lib(bin, "shapes", "") || !build_lib(bin, "units", "-lib-export-all") + || !build_lib(bin, "greetings", "-- --jit-lib-export-marked")) { + return + } + + var sh = shapes_create() + var un = units_create() + var gr = greetings_create() + if (sh == null || un == null || gr == null) { + print("a library refused to create an instance: {shapes_last_error(null)}\n") + return + } + var a = Vec2(x = 3.0, y = 4.0) + var b = Vec2(x = 1.0, y = 2.0) + print("shapes.dot((3,4),(1,2)) = {shapes_dot(sh, safe_addr(a), safe_addr(b))}\n") + print("units.celsius_to_fahrenheit(100) = {units_celsius_to_fahrenheit(un, 100.0)}\n") + print("greetings.greet(\"mundo\") = {greetings_greet(gr, "mundo")}\n") + greetings_destroy(gr) + units_destroy(un) + shapes_destroy(sh) + print("three daslang libraries, one process - ok\n") +} diff --git a/examples/c_api_library/shapes.das b/examples/c_api_library/shapes.das new file mode 100644 index 0000000000..1f2ac59a14 --- /dev/null +++ b/examples/c_api_library/shapes.das @@ -0,0 +1,24 @@ +options gen2 +options indenting = 4 + + +require daslib/export_c + +struct Vec2 { + x : float + y : float +} + +[export_c] +def dot(a, b : Vec2) : float { + return a.x * b.x + a.y * b.y +} + +[export_c(name = "scaled")] +def scale(v : Vec2; k : float) : Vec2 { + return Vec2(x = v.x * k, y = v.y * k) +} + +def length_squared(v : Vec2) : float { + return dot(v, v) +} diff --git a/examples/c_api_library/units.das b/examples/c_api_library/units.das new file mode 100644 index 0000000000..8acede2724 --- /dev/null +++ b/examples/c_api_library/units.das @@ -0,0 +1,17 @@ +options gen2 +options indenting = 4 + +require math + + +def celsius_to_fahrenheit(c : float) : float { + return c * 1.8 + 32.0 +} + +def clamp_int(v, lo, hi : int) : int { + return clamp(v, lo, hi) +} + +def private rounding_bias() : float { + return 0.5 +} diff --git a/include/daScript/simulate/code_of_policies.h b/include/daScript/simulate/code_of_policies.h index 78bb940b67..2d27daf29f 100644 --- a/include/daScript/simulate/code_of_policies.h +++ b/include/daScript/simulate/code_of_policies.h @@ -41,6 +41,7 @@ namespace das { bool no_lint = false; // skip Program::lint() entirely bool no_init_check = false; // skip the Module::Initialize() assert, most of the time should be false (except maybe dynamic-module discovery) bool export_all = false; // when user compiles, export all (public?) functions + bool export_public_functions = false; bool serialize_main_module = true; // if false, then we recompile main module each time bool keep_alive = false; // produce keep-alive noodes /*option*/ bool very_safe_context = false; // context is very safe (does not release old memory from array or table grow, leaves it to GC) diff --git a/install/CLAUDE.md b/install/CLAUDE.md index 9d83ff71a1..16a77746cc 100644 --- a/install/CLAUDE.md +++ b/install/CLAUDE.md @@ -53,7 +53,7 @@ Task-specific instructions are in skill files under `skills/`. Read the relevant | `skills/mcp_tools.md` | Full MCP tool table + live-API reference | | `skills/das_formatting.md` | Creating or modifying any `.das` file | | `skills/comment_style_hygiene.md` | Writing or reviewing comments, names, or local code shape in ANY language | -| `skills/cpp_integration.md` | Embedding daslang in C++; binding types/functions/enums; shipping without the compiler (`libDaScriptNano`, or a standalone context on the full runtime) | +| `skills/cpp_integration.md` | Embedding daslang in C++; binding types/functions/enums; shipping without the compiler (`libDaScriptNano`, or a standalone context on the full runtime); calling daslang from C (`daslang -lib`) | | `skills/daslib_modules.md` | Using `daslib/` modules (linq, json, regex, etc.) | | `skills/das_macros.md` | Compile-time macros, AST manipulation, qmacro/quote, gc_node patterns | | `skills/daspkg.md` | Creating `.das_package` manifests, daspkg commands | diff --git a/modules/dasLLVM/.das_module b/modules/dasLLVM/.das_module index 311cdd1c8c..05007a6b11 100644 --- a/modules/dasLLVM/.das_module +++ b/modules/dasLLVM/.das_module @@ -8,7 +8,8 @@ def initialize(project_path : string) { "llvm_boost", "llvm_debug", "llvm_jit", "llvm_targets", "llvm_dsl", "llvm_jit_intrin", "llvm_jit_common", "llvm_jit_lower", "llvm_dll_utils", - "llvm_exe", "llvm_macro", "llvm_jit_cli", "llvm_jit_run", "llvm_aot", + "llvm_exe", + "jit_standalone", "llvm_macro", "llvm_jit_cli", "llvm_jit_run", "llvm_aot", "llvm_env", // [EnvConfig] environment-knob registry (ENVIRONMENT.md generates from it) "llvm_cpu_class", // the CPU classes profiles are keyed by, and the DAS_JIT_BASELINE target "llvm_code", // [llvm_code] user-side annotation diff --git a/modules/dasLLVM/ARCHITECTURE.md b/modules/dasLLVM/ARCHITECTURE.md index 8e166b38ab..593f17713d 100644 --- a/modules/dasLLVM/ARCHITECTURE.md +++ b/modules/dasLLVM/ARCHITECTURE.md @@ -13,7 +13,8 @@ cached DLL, compare per-function hashes), **irgen** (the das IR emitter over eve **optimize** (the LLVM pass pipeline at the requested level, plus the opt-in IR dump and the post-opt verify), **emit+link** (artifact production - for the DLL path `write_dll` in `llvm_jit_common.das`, which itself splits into **emit-obj**, machine-code emission, and -**link**, the lld-link spawn), **install** (resolve externs, instrument sim nodes), and +**link**, the lld-link spawn; for `-lib` `write_lib`, which links a shared library or archives a +static one and writes the C header beside it), **install** (resolve externs, instrument sim nodes), and **finalize** (engine teardown / state install). On a cache hit irgen, optimize, and emit+link are skipped and read as zero. @@ -53,13 +54,17 @@ one of those files visible (re-pin `LLVM_JIT_EMITTER_HASH`), and the bump is owe emitted code for identical inputs can differ - a comment, a nolint, or a same-value rewrite inside an emitter file re-pins without a bump. -`--jit-opt-level` (CLI, over `policies.jit_opt_level`, default 3) drives both the optimize -pipeline and the DLL path's codegen-side target machine. `write_exe` and AOT-object emission -(`emit_object_only`) deliberately stay at codegen level 3: shipped artifacts are not -content-addressed, so a tier change there has no cache guard to catch it. At level 0 the -injected tune-policy default becomes `fallback` (`jit_cli_opt_level()` in `llvm_tune.das`): -tune winners are raced under O3 codegen, so an O0 run cannot represent them and must not block -on the tuner to mint them. +The per-artifact entry emitters are OUTSIDE that surface: `llvm_exe.das` (a standalone exe's +`main`) and its `-lib` half (the C entry points and thunks) emit startup glue for artifacts +nothing content-addresses, so neither is in `EMITTER_FILES` and neither owes a version bump. Both +artifacts run in `LlvmJitMode.EXE`, which is what makes the exe startup shareable: the four +`emit_standalone_*` helpers in `llvm_exe.das` are the shared halves, split so a library can put the +process-global half behind a once guard and the per-context half behind its own catch boundary. + +### 1.3 A library's runtime + +`daslang -lib` emits an artifact that loads into a process it does not own; its runtime, +environment and shutdown rules are `ARCHITECTURE_LIB.md` sec. 1.3. ## 2. Codegen identity - the DLL cache diff --git a/modules/dasLLVM/ARCHITECTURE_LIB.md b/modules/dasLLVM/ARCHITECTURE_LIB.md new file mode 100644 index 0000000000..f7deb488e1 --- /dev/null +++ b/modules/dasLLVM/ARCHITECTURE_LIB.md @@ -0,0 +1,60 @@ +# dasLLVM Architecture - the emitted library + +Companion to `ARCHITECTURE.md` in this folder; section numbers are unique across both files. + +### 1.3 A library's runtime is process-global, its environment is per-thread {#lib-runtime-scope} + +An exe owns its process: one thread runs `main`, registers the modules, and drains them on the way +out. A library owns none of that, and three consequences shape its entry points. + +**The environment is thread-local** (`daScriptEnvironment::bound` / `owned`, +`include/daScript/ast/ast.h`), but the module registration behind it happens once. So +`jit_lib_run_once` does both jobs: the first caller registers, and every later caller - on any +thread - is BOUND to that first registration's environment. Without the binding a second thread's +`

_create` dereferences a null `getBound()->modules` inside `jit_init_extern_function`. The +create-failure message (`g_jitLibCreateError`) is process-global for the same reason read from the +other side: a C host reads it through `

_last_error(nullptr)`, and a per-thread copy would answer +null on every thread but the one whose create failed. + +**One daslang runtime fits in a process, and every artifact in it shares that one.** +`jit_register_Module_*` (`REGISTER_MODULE_IN_NAMESPACE`) carries no already-created guard and +aborts on a second call, so every emitted registration goes through +`jit_register_module_once(dasName, reg)`, which registers only when `Module::require(dasName)` +finds nothing and otherwise hands back the module already there. EVERY site means both of them - +the require walk and the extern collector's `ensure_module`, which reaches a module no `require` +names (an `ast_core` extern called from a library aborted the host until it did). A library that finds a populated +environment is a GUEST (`g_jitLibGuest`, decided in `jit_lib_run_once` before the init call): it +skips `Module::Initialize` so `g_envTotal` stays balanced, and it never drains what it did not +create. Ownership is tracked separately (`g_jitLibOwner`, set by `jit_lib_arm_shutdown`) because +the guest flag is process-global: a second library arriving later must not erase the first one's +duty to drain. + +**A guest reports a failed dynamic module; it does not abort.** `jit_finalize_dynamic_modules` +fatals on anything still unloadable, which is right for an exe that owns its process and wrong for +a library: the pending list it inherits is the HOST's, and a module the host could not load is not +this library's to kill the process over. A guest calls `jit_lib_finalize_dynamic_modules` instead - +retry, report, carry on - and a module this library actually needs still surfaces, as a null +`

_create` carrying the reason. + +**Nothing calls the shutdown functions for a library.** A jitted `SimFunction` gets a zeroed +`FuncInfo` (`jit_lib`'s `registerJitFunction`, `src/builtin/module_jit.cpp`), so +`Context::runShutdownScript` - which selects on `FuncInfo::flag_shutdown`, set only by the +interpreter's debug-info builder - finds none. `

_destroy` therefore emits the program's +`[finalize]` / `[shutdown]` calls itself, the same way `

_create` emits its `[init]` calls. The +process-level drain is an `atexit` hook armed beside the registration it balances, and it rebinds +the recorded environment first so the draining thread need not be the creating one. + +Every linked artifact leaves through one emitter, `write_artifact` (`llvm_jit_common.das`): the +`JitArtifact` kind picks the file name, the linker flavor (an archiver for a static library, no +`-shared` for an exe) and the target CPU. A `jit_dll` always targets this host - it only ever runs +on the box that emitted it - while a shipped artifact takes the caller's `use_host_cpu`: generic +and redistributable by default, host-specific only when the build carries `[llvm_code]` kernels, +whose tuner-generated IR a generic target refuses to legalize. + +`--jit-opt-level` (CLI, over `policies.jit_opt_level`, default 3) drives both the optimize +pipeline and the DLL path's codegen-side target machine. A shipped artifact and AOT-object +emission (`emit_object_only`) deliberately stay at codegen level 3: shipped artifacts are not +content-addressed, so a tier change there has no cache guard to catch it. At level 0 the +injected tune-policy default becomes `fallback` (`jit_cli_opt_level()` in `llvm_tune.das`): +tune winners are raced under O3 codegen, so an O0 run cannot represent them and must not block +on the tuner to mint them. diff --git a/modules/dasLLVM/README.md b/modules/dasLLVM/README.md index cd29e509c4..4cd78de53e 100644 --- a/modules/dasLLVM/README.md +++ b/modules/dasLLVM/README.md @@ -81,6 +81,47 @@ cached DLL for instant execution. - By default, the `dll` is stored in `.jitted_scripts/`. - This can be changed using `jit_output_path`. +## Native library with a C API (`-lib`) +`-lib` emits a native library plus the C header a host calls it through, so a +program that only needs to *call* one script links no daslang API: + +```sh +./bin/daslang -lib script.das -output build/script +``` + +writes `build/script.so` (`.dylib` / `.dll`) and `build/script.h`. Add +`-- --jit-lib-static` for a `.a` / `.lib` archive instead; the generated header +records what a host then links. + +Which functions cross the boundary, in three forms: `[export_c]` marks them one +at a time; `-- --jit-lib-export-marked` takes whatever the program already marks +`[export]`, so a script with a host API needs no new annotation; and +`-lib-export-all` offers every public function of the entry module whose +signature C can spell (naming the ones it skips). The first two are a selection +only - a signature C cannot spell stays a hard error - while export-all skips it +with a warning. `examples/c_api_library/` builds one library each way and loads +all three at once through dasbind. Every library gets the same four entry points, +prefixed with the output name, so they always match the header they are declared in: + +```c +script_ctx * script_create(void); +void script_destroy(script_ctx * ctx); +const char * script_last_error(script_ctx * ctx); +void script_shutdown_runtime(void); +``` + +One instance owns one daslang context - its own globals, heap and string heap - and `_create` +works on any thread. Several such libraries coexist in one process, as does one inside a host that +registered the daslang modules itself: the first there registers the runtime, the rest bind to it. +Scalars, `string`, pointers and enums cross by value; structures and the +`float2`..`uint4` / `range` families cross as `const T *` and return through a +trailing `T * out`. A daslang panic returns zero, leaves `out` untouched, and +shows up in `script_last_error(ctx)` until the next call clears it. + +Codegen is the same as `-exe`: the same private wrappers, the same +load-resolved globals, no DLL cache. Cross-compilation is not supported here - +build the library on its target host. + ## Cross-compilation (WebAssembly) The JIT pipeline can emit a non-host target instead of running on the host. The supported cross-target is `wasm32-unknown-emscripten` (the default when no diff --git a/modules/dasLLVM/REVIEW.md b/modules/dasLLVM/REVIEW.md index 81531e8f6a..65fcc9188f 100644 --- a/modules/dasLLVM/REVIEW.md +++ b/modules/dasLLVM/REVIEW.md @@ -56,6 +56,14 @@ (`ARCHITECTURE.md` sec.5). An unpinned compile-time file read serves stale macro output from the module cache until an unrelated source file changes - silently. +- **A diff that adds a `-lib` entry point, or work to one, keeps the C boundary's three promises: + a raise reaches the caller as a return value and never an unwind, `

_create` answers null + rather than aborting, and `

_destroy` runs what the runtime's own shutdown cannot find** + (`ARCHITECTURE_LIB.md#lib-runtime-scope`). A library is called by code that cannot catch anything. + +- **A `-lib` build that writes no artifact exits non-zero.** A build rule reads the exit code, and + a silent success lets it link the previous run's library against this run's header. + - **A change to a `[tune]`-family annotation is reviewed with `skills/tune.md`.** - **A change to the tune framework - `daslib/llvm_tune.das` or its tests - is reviewed with diff --git a/modules/dasLLVM/daslib/jit_standalone.das b/modules/dasLLVM/daslib/jit_standalone.das new file mode 100644 index 0000000000..925a38cab4 --- /dev/null +++ b/modules/dasLLVM/daslib/jit_standalone.das @@ -0,0 +1,405 @@ +options gen2 +options indenting = 4 +options no_unused_block_arguments = false +options no_unused_function_arguments = false +options strict_smart_pointers = false +options relaxed_pointer_const +options unsafe_table_lookup = false +options no_global_variables = false + +module jit_standalone shared private + +require llvm/daslib/llvm_boost +require llvm/daslib/llvm_jit +require llvm/daslib/llvm_jit_common +require llvm/daslib/llvm_dll_utils +require llvm/daslib/llvm_dsl +require llvm/daslib/llvm_exe +require daslib/ast_boost +require daslib/c_api_header +require daslib/safe_addr +require daslib/defer + +let private LIB_GUARD_GLOBAL = "__das_lib_init_guard" +let private LIB_RUNTIME_INIT = "__das_lib_runtime_init" +let private LIB_CTX_INIT = "__das_lib_ctx_init" + +var public g_lib_exports : array + + +def public lib_link_note(static_lib : bool; link_whole_lib : bool) : string { + let windows = get_platform_name() == "windows" + let rt_static = windows ? "libDaScript_runtime.lib" : "liblibDaScript_runtime.a" + let cc_static = windows ? "libDaScript.lib" : "liblibDaScript.a" + let rt_shared = windows ? "libDaScriptDyn_runtime.dll" : (get_platform_name() == "darwin" ? "liblibDaScriptDyn_runtime.dylib" : "liblibDaScriptDyn_runtime.so") + if (static_lib) { + let whole = link_whole_lib ? " Add /lib/{cc_static} too - this library\nregisters a module that lives in the compiler library." : "" + let sys = windows ? "" : " Also link the platform libraries the runtime needs:\n-lpthread -ldl -lm -lstdc++." + return "Link this archive plus /lib/{rt_static}.{whole}{sys}" + } + let whole = link_whole_lib ? " It also needs the compiler library beside it - this library\nregisters a module that lives there." : "" + return "Link this shared library. At load it needs {rt_shared}, which it looks for beside itself,\nin ../lib, and in /lib; otherwise put it on the loader path.{whole}" +} + + +def private new_function(ctx : LLVMContextRef; name : string; typ : LLVMTypeRef; exported : bool) : tuple { + let fn = LLVMAddFunctionWithType(g_mod, name, typ) + if (exported) { + set_public_linkage(fn) + } else { + set_private_linkage(fn) + } + let b = LLVMCreateBuilderInContext(ctx) + LLVMPositionBuilderAtEnd(b, LLVMAppendBasicBlockInContext(ctx, fn, "entry")) + return (fn = fn, builder = b) +} + + +struct private ThunkAbi { + arg_types : array + arg_loads : array + c_types : array + slot_type : LLVMOpaqueType? + has_slot : bool +} + + +def private thunk_abi(var e : CExport; var types : PrimitiveTypes?) : ThunkAbi { + var abi : ThunkAbi + abi.c_types |> push(types.LLVMVoidPtrType()) + for (arg, p in e.fn.arguments, e.params) { + abi.arg_types |> push(type_to_llvm_abi_type(arg._type)) + abi.arg_loads |> push(p.by_pointer && !p.pointer_in_impl) + abi.c_types |> push(p.by_pointer ? types.LLVMVoidPtrType() + : (arg._type.isBool ? types.t_int8 : type_to_llvm_abi_type(arg._type))) + } + if (e.result.via_out) { + abi.c_types |> push(types.LLVMVoidPtrType()) + } + if (!e.fn.result.isVoid) { + abi.has_slot = true + abi.slot_type = (e.result.cmres ? types.LLVMVoidPtrType() : type_to_llvm_abi_type(e.fn.result)) + } + return <- abi +} + + +def private frame_type(var abi : ThunkAbi; ctx : LLVMContextRef) : LLVMOpaqueType? { + var fields <- [for (t in abi.arg_types); t] + if (abi.has_slot) { + fields |> push(abi.slot_type) + } + if (fields |> empty()) { + fields |> push(g_prim_t.t_int32) + } + return LLVMStructTypeInContext(ctx, array_data_ptr(fields), uint(length(fields)), 0) +} + + +def private emit_trampoline(ctx : LLVMContextRef; var e : CExport; var uids : UidNodes?; + var types : PrimitiveTypes?; var abi : ThunkAbi; ft : LLVMOpaqueType?) : LLVMOpaqueValue? { + let impl_name = uids.get_dll_fn_name_ptr(e.fn).impl() + var impl = LLVMGetNamedFunction(g_mod, impl_name) + if (impl == null) { + return null + } + let tramp_type = jit_fn_type($(ctx, frame : void?) : void {}) + let made = new_function(ctx, "__das_lib_tramp_{e.c_name}", tramp_type, false) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + } + var ctx_arg = LLVMGetParam(made.fn, 0u) + var frame = LLVMBuildPointerCast(b, LLVMGetParam(made.fn, 1u), LLVMPointerType(ft, 0u), "frame") + var args : array + args |> reserve(length(abi.arg_types) + 2) + for (at, i in abi.arg_types, count()) { + let slot = LLVMBuildStructGEP2(b, ft, frame, uint(i), "arg_{i}") + args |> push(LLVMBuildLoad2(b, at, slot, "argv_{i}")) + } + args |> push(ctx_arg) + let slot_index = uint(length(abi.arg_types)) + if (e.result.cmres) { + var slot_ptr = LLVMBuildStructGEP2(b, ft, frame, slot_index, "res_ptr") + args |> push(LLVMBuildLoad2(b, types.LLVMVoidPtrType(), slot_ptr, "res")) + } + var ret = LLVMBuildCall2(b, g_fn_types[impl_name], impl, args, "") + if (abi.has_slot && !e.result.cmres) { + LLVMBuildStore(b, ret, LLVMBuildStructGEP2(b, ft, frame, slot_index, "res")) + } + LLVMBuildRetVoid(b) + return made.fn +} + + +def private c_return_type(var e : CExport; var types : PrimitiveTypes?) : LLVMTypeRef { + if (e.result.via_out || e.fn.result.isVoid) { + return types.t_void + } + return e.fn.result.isBool ? types.t_int8 : type_to_llvm_abi_type(e.fn.result) +} + + +def private store_thunk_args(b : LLVMBuilderRef; var e : CExport; var abi : ThunkAbi; + fn : LLVMOpaqueValue?; ft : LLVMOpaqueType?; var types : PrimitiveTypes?; frame : LLVMOpaqueValue?) { + for (arg, p, i in e.fn.arguments, e.params, count()) { + var incoming = LLVMGetParam(fn, uint(i + 1)) + var slot = LLVMBuildStructGEP2(b, ft, frame, uint(i), "in_{i}") + if (abi.arg_loads[i]) { + var typed = LLVMBuildPointerCast(b, incoming, LLVMPointerType(abi.arg_types[i], 0u), "") + incoming = LLVMBuildLoadData2Aligned(b, abi.arg_types[i], typed, arg._type.alignOf, "val_{i}") + } elif (p.by_pointer) { + incoming = LLVMBuildPointerCast(b, incoming, abi.arg_types[i], "") + } elif (arg._type.isBool) { + incoming = LLVMBuildICmp(b, LLVMIntPredicate.LLVMIntNE, incoming, types.ConstI8(int8(0)), "b_{i}") + } + LLVMBuildStore(b, incoming, slot) + } +} + + +def private emit_thunk(ctx : LLVMContextRef; var e : CExport; var uids : UidNodes?; var types : PrimitiveTypes?) : bool { + var abi <- thunk_abi(e, types) + let ft = frame_type(abi, ctx) + let tramp = emit_trampoline(ctx, e, uids, types, abi, ft) + if (tramp == null) { + to_log(LOG_ERROR, "LLVM LIB: no jitted body for `{e.fn.name}` - it cannot be exported to C\n") + return false + } + if (LLVMGetNamedFunction(g_mod, e.c_name) != null) { + to_log(LOG_ERROR, "LLVM LIB: C symbol `{e.c_name}` is already emitted; rename one with [export_c(name = \"...\")]\n") + return false + } + let made = new_function(ctx, e.c_name, LLVMFunctionType(c_return_type(e, types), abi.c_types), true) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + } + var frame = LLVMBuildAlloca(b, ft, "frame") + var cmres_slot : LLVMOpaqueValue? + if (e.result.cmres) { + cmres_slot = LLVMBuildAlloca(b, LLVMArrayType(types.t_int8, uint(e.fn.result.sizeOf)), "result") + LLVMSetAlignment(cmres_slot, uint(e.fn.result.alignOf)) + LLVMBuildStore(b, LLVMBuildPointerCast(b, cmres_slot, types.LLVMVoidPtrType(), ""), + LLVMBuildStructGEP2(b, ft, frame, uint(length(abi.arg_types)), "res_ptr")) + } + store_thunk_args(b, e, abi, made.fn, ft, types, frame) + let guard_type = jit_fn_type($(ctx, tramp, frame : void?) : int {}) + var guard = declare_extern_fn("jit_lib_invoke_guarded", guard_type) + var guard_args = array(LLVMGetParam(made.fn, 0u), + LLVMBuildPointerCast(b, tramp, types.LLVMVoidPtrType(), ""), + LLVMBuildPointerCast(b, frame, types.LLVMVoidPtrType(), "")) + var ok = LLVMBuildCall2(b, guard_type, guard, guard_args, "ok") + if (!abi.has_slot) { + LLVMBuildRetVoid(b) + return true + } + let ok_bb = LLVMAppendBasicBlockInContext(ctx, made.fn, "ok") + let fail_bb = LLVMAppendBasicBlockInContext(ctx, made.fn, "raised") + var cond = LLVMBuildICmp(b, LLVMIntPredicate.LLVMIntNE, ok, types.ConstI32(0ul), "ran") + LLVMBuildCondBr(b, cond, ok_bb, fail_bb) + let slot_index = uint(length(abi.arg_types)) + LLVMPositionBuilderAtEnd(b, ok_bb) + if (e.result.via_out) { + var out_ptr = LLVMGetParam(made.fn, uint(length(e.params) + 1)) + var source = e.result.cmres ? cmres_slot : LLVMBuildStructGEP2(b, ft, frame, slot_index, "res") + LLVMBuildMemCpy(b, out_ptr, 1u, source, uint(e.fn.result.alignOf), types.ConstI64(uint64(e.fn.result.sizeOf))) + LLVMBuildRetVoid(b) + } else { + var value = LLVMBuildLoad2(b, abi.slot_type, LLVMBuildStructGEP2(b, ft, frame, slot_index, "res"), "value") + LLVMBuildRet(b, e.fn.result.isBool ? LLVMBuildZExt(b, value, types.t_int8, "b") : value) + } + LLVMPositionBuilderAtEnd(b, fail_bb) + if (e.result.via_out || e.fn.result.isVoid) { + LLVMBuildRetVoid(b) + } else { + LLVMBuildRet(b, LLVMConstNull(c_return_type(e, types))) + } + return true +} + + +def private emit_runtime_init(ctx : LLVMContextRef; prog : Program?; var types : PrimitiveTypes?; + register_all_modules : bool; var used_modules : table&; + var dynamic_modules : table&) : tuple { + let void_fn_type = jit_fn_type($() : void {}) + let made = new_function(ctx, LIB_RUNTIME_INIT, void_fn_type, false) + let fresh_init_modules = LLVMGetNamedFunction(g_mod, "initialize_modules") == null + var init_modules = declare_extern_fn("initialize_modules", void_fn_type) + if (fresh_init_modules) { + set_private_linkage(init_modules) + } + var force_dynamic_modules : table + if (register_all_modules) { + force_dynamic_modules["UnitTest"] = true + } + let ships = emit_standalone_runtime_init(made.builder, prog, types, register_all_modules, + force_dynamic_modules, used_modules, dynamic_modules, true) + let arm_type = jit_fn_type($() : void {}) + var arm = declare_extern_fn("jit_lib_arm_shutdown", arm_type) + LLVMBuildCall2(made.builder, arm_type, arm, array(), "") + return (fn = made.fn, builder = made.builder, ships_dynamic = ships) +} + + +def private emit_create(ctx : LLVMContextRef; program_context : Context?; prog : Program?; mod : LLVMOpaqueModule?; + var types : PrimitiveTypes?; var uids : UidNodes?; prefix : string; + var sf : StandaloneFunctions; runtime_init : LLVMOpaqueValue?; + runtime_builder : LLVMBuilderRef; register_all_modules : bool; ships_dynamic : bool; + dynamic_modules : table) : tuple { + let void_ptr = types.LLVMVoidPtrType() + var guard_global = LLVMAddGlobal(g_mod, types.t_int32, LIB_GUARD_GLOBAL) + set_private_linkage(guard_global) + LLVMSetInitializer(guard_global, types.ConstI32(0ul)) + + let ctx_init = new_function(ctx, LIB_CTX_INIT, jit_fn_type($(ctx, frame : void?) : void {}), false) + let made = new_function(ctx, "{prefix}_create", jit_fn_type($() : void? {}), true) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + LLVMDisposeBuilder(ctx_init.builder) + } + let run_once_type = jit_fn_type($(guard, init : void?) : int {}) + var run_once = declare_extern_fn("jit_lib_run_once", run_once_type) + var registered = LLVMBuildCall2(b, run_once_type, run_once, + array(LLVMBuildPointerCast(b, guard_global, void_ptr, ""), + LLVMBuildPointerCast(b, runtime_init, void_ptr, "")), "registered") + let ready_bb = LLVMAppendBasicBlockInContext(ctx, made.fn, "ready") + let declined_bb = LLVMAppendBasicBlockInContext(ctx, made.fn, "declined") + var can_run = LLVMBuildICmp(b, LLVMIntPredicate.LLVMIntNE, registered, types.ConstI32(0ul), "can_run") + LLVMBuildCondBr(b, can_run, ready_bb, declined_bb) + LLVMPositionBuilderAtEnd(b, declined_bb) + LLVMBuildRet(b, LLVMConstNull(void_ptr)) + LLVMPositionBuilderAtEnd(b, ready_bb) + var global_context = emit_create_standalone_ctx(b, program_context, prog, types, sf.any_pinvoke) + let guard_type = jit_fn_type($(ctx, tramp, frame : void?) : int {}) + var guard = declare_extern_fn("jit_lib_invoke_guarded", guard_type) + var ok = LLVMBuildCall2(b, guard_type, guard, + array(global_context, LLVMBuildPointerCast(b, ctx_init.fn, void_ptr, ""), LLVMConstNull(void_ptr)), "ok") + let finish_type = jit_fn_type($(ctx : void?; ok : int) : void? {}) + var finish = declare_extern_fn("jit_lib_create_finish", finish_type) + LLVMBuildRet(b, LLVMBuildCall2(b, finish_type, finish, array(global_context, ok), "")) + + var ctx_param = LLVMGetParam(ctx_init.fn, 0u) + let whole = emit_standalone_context_init(ctx_init.builder, runtime_builder, program_context, ctx, + prog, mod, types, uids, sf.funcs, ctx_param, register_all_modules, ships_dynamic, + sf.used_modules, dynamic_modules) + LLVMBuildRetVoid(ctx_init.builder) + return (fn = made.fn, link_whole_lib = whole) +} + + +def private emit_shim_entry(ctx : LLVMContextRef; name, shim : string; typ : LLVMTypeRef; takes_ctx : bool) { + let made = new_function(ctx, name, typ, true) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + } + var args <- takes_ctx ? array(LLVMGetParam(made.fn, 0u)) : array() + var call = LLVMBuildCall2(b, typ, declare_extern_fn(shim, typ), args, "") + if (takes_ctx) { + LLVMBuildRet(b, call) + } else { + LLVMBuildRetVoid(b) + } +} + + +def private emit_shutdown_trampoline(ctx : LLVMContextRef; var uids : UidNodes?; + var funcs : array) : LLVMOpaqueValue? { + var shutdown_fns <- [for (fn in funcs); fn; where fn.flags.shutdown] + if (shutdown_fns |> empty()) { + return null + } + let made = new_function(ctx, "__das_lib_ctx_shutdown", jit_fn_type($(ctx, frame : void?) : void {}), false) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + } + var ctx_arg = LLVMGetParam(made.fn, 0u) + for (fn in shutdown_fns) { + let impl_name = uids.get_dll_fn_name(fn).impl() + var impl = LLVMGetNamedFunction(g_mod, impl_name) + if (impl != null) { + LLVMBuildCall2(b, g_fn_types[impl_name], impl, array(ctx_arg), "") + } + } + LLVMBuildRetVoid(b) + return made.fn +} + + +def private emit_destroy(ctx : LLVMContextRef; var types : PrimitiveTypes?; prefix : string; + shutdown_tramp : LLVMOpaqueValue?) { + let void_ptr = types.LLVMVoidPtrType() + let made = new_function(ctx, "{prefix}_destroy", jit_fn_type($(ctx : void?) : void {}), true) + let b = made.builder + defer() { + LLVMDisposeBuilder(b) + } + var ctx_arg = LLVMGetParam(made.fn, 0u) + if (shutdown_tramp != null) { + var guard = declare_extern_fn("jit_lib_invoke_guarded", jit_fn_type($(ctx, tramp, frame : void?) : int {})) + LLVMBuildCall2(b, jit_fn_type($(ctx, tramp, frame : void?) : int {}), guard, + array(ctx_arg, LLVMBuildPointerCast(b, shutdown_tramp, void_ptr, ""), LLVMConstNull(void_ptr)), "") + } + let destroy_type = jit_fn_type($(ctx : void?) : void {}) + var destroy = declare_extern_fn("jit_destroy_standalone_ctx", destroy_type) + LLVMBuildCall2(b, destroy_type, destroy, array(ctx_arg), "") + LLVMBuildRetVoid(b) +} + + +def private emit_fixed_entries(ctx : LLVMContextRef; var types : PrimitiveTypes?; var uids : UidNodes?; + var funcs : array; prefix : string) { + emit_destroy(ctx, types, prefix, emit_shutdown_trampoline(ctx, uids, funcs)) + emit_shim_entry(ctx, "{prefix}_last_error", "jit_lib_last_error", jit_fn_type($(ctx : void?) : void? {}), true) + emit_shim_entry(ctx, "{prefix}_shutdown_runtime", "jit_lib_shutdown", jit_fn_type($() : void {}), false) +} + + +[arch(at="../ARCHITECTURE_LIB.md#lib-runtime-scope")] +def public inject_lib(program_context : Context?; ctx : LLVMContextRef; prog : Program?; + mod : LLVMOpaqueModule?; var types : PrimitiveTypes?; var uids : UidNodes?; + strict : bool; export_all : bool; output_path : string; + register_all_modules : bool = false) : tuple { + g_lib_exports |> clear() + let prefix = lib_prefix_from_path(output_path) + if (prefix |> empty()) { + to_log(LOG_ERROR, "LLVM LIB: cannot derive a C symbol prefix from the output path `{output_path}` - name it after a C identifier\n") + return (fn = null, link_whole_lib = false) + } + let names = CNames(prefix = prefix, this_module = prog.getThisModule) + var selected <- collect_c_exports(prog, names, export_all) + for (r in selected.errors) { + failed(r.message) + } + if (!(selected.errors |> empty())) { + return (fn = null, link_whole_lib = false) + } + var exports <- selected.exports + if (exports |> empty()) { + to_log(LOG_ERROR, "LLVM LIB: nothing to export - annotate functions with [export_c], or pass -lib-export-all\n") + return (fn = null, link_whole_lib = false) + } + var sf <- collect_standalone_functions(prog, strict) + if (!sf.ok) { + return (fn = null, link_whole_lib = false) + } + var dynamic_modules : table + let rt = emit_runtime_init(ctx, prog, types, register_all_modules, sf.used_modules, dynamic_modules) + let created = emit_create(ctx, program_context, prog, mod, types, uids, prefix, sf, rt.fn, + rt.builder, register_all_modules, rt.ships_dynamic, dynamic_modules) + LLVMBuildRetVoid(rt.builder) + LLVMDisposeBuilder(rt.builder) + emit_fixed_entries(ctx, types, uids, sf.funcs, prefix) + for (e in exports) { + if (!emit_thunk(ctx, e, uids, types)) { + return (fn = null, link_whole_lib = false) + } + } + to_log(LOG_INFO, "LLVM LIB: C entry points generated: {prefix}_create/destroy/last_error plus {length(exports)} exports, {length(sf.funcs)} functions\n") + g_lib_exports |> clear() + g_lib_exports |> push_clone_from(exports) + return (fn = created.fn, link_whole_lib = created.link_whole_lib) +} diff --git a/modules/dasLLVM/daslib/llvm_dll_utils.das b/modules/dasLLVM/daslib/llvm_dll_utils.das index f00d265e99..270bbec452 100644 --- a/modules/dasLLVM/daslib/llvm_dll_utils.das +++ b/modules/dasLLVM/daslib/llvm_dll_utils.das @@ -301,17 +301,31 @@ class public DLLHandle { } -def public add_obj_extension(path : string) { - return "{path}.o" -} -def public add_dll_extension(path : string) { - return "{path}.dll" +enum public JitArtifact { + object + jit_dll + exe + shared_lib + static_lib } -def public add_exe_extension(path : string) { - return "{path}.exe" + +def public artifact_path(path : string; kind : JitArtifact) : string { + let plat = get_platform_name() + var suffix = "o" + if (kind == JitArtifact.jit_dll) { + suffix = "dll" + } elif (kind == JitArtifact.exe) { + suffix = "exe" + } elif (kind == JitArtifact.static_lib) { + suffix = plat == "windows" ? "lib" : "a" + } elif (kind == JitArtifact.shared_lib) { + suffix = plat == "windows" ? "dll" : (plat == "darwin" ? "dylib" : "so") + } + return "{path}.{suffix}" } + def public get_dll_by_path(path : string) : DLLHandle? { - return new DLLHandle(handle = load_dynamic_library(add_dll_extension(path))) + return new DLLHandle(handle = load_dynamic_library(artifact_path(path, JitArtifact.jit_dll))) } diff --git a/modules/dasLLVM/daslib/llvm_exe.das b/modules/dasLLVM/daslib/llvm_exe.das index dc2481ce61..6a74765b20 100644 --- a/modules/dasLLVM/daslib/llvm_exe.das +++ b/modules/dasLLVM/daslib/llvm_exe.das @@ -125,19 +125,19 @@ class public CollectExternVisitor : AstVisitor { registered_modules[m] = true } } else { - // Register $ and strings unless inject_main's static sweep already emitted the (non-idempotent) call. - if (!(g_exe_emitted_reg |> key_exists("jit_register_Module_BuiltIn"))) { - g_exe_emitted_reg["jit_register_Module_BuiltIn"] = true - var reg_builtin = LLVMAddFunctionWithType(g_mod, "jit_register_Module_BuiltIn", register_mod_type) - LLVMBuildCall2(ib, register_mod_type, reg_builtin, array(), "") - } - registered_modules["$"] = true - if (!(g_exe_emitted_reg |> key_exists("jit_register_Module_Strings"))) { - g_exe_emitted_reg["jit_register_Module_Strings"] = true - var reg_strings = LLVMAddFunctionWithType(g_mod, "jit_register_Module_Strings", register_mod_type) - LLVMBuildCall2(ib, register_mod_type, reg_strings, array(), "") + let once_type = jit_fn_type($(name, reg : void?) : void? {}) + var once = declare_extern_fn("jit_register_module_once", once_type) + for (pair in fixed_array(fixed_array("$", "jit_register_Module_BuiltIn"), + fixed_array("strings", "jit_register_Module_Strings"))) { + if (!(g_exe_emitted_reg |> key_exists(pair[1]))) { + g_exe_emitted_reg[pair[1]] = true + var reg_fn = declare_extern_fn(pair[1], register_mod_type) + LLVMBuildCall2(ib, once_type, once, + array(get_string_constant_ptr(ib, pair[0]), + LLVMBuildPointerCast(ib, reg_fn, g_prim_t.LLVMVoidPtrType(), "")), "") + } + registered_modules[pair[0]] = true } - registered_modules["strings"] = true } return ib } @@ -313,14 +313,19 @@ class public CollectExternVisitor : AstVisitor { if (mod_name == "ast_core" || mod_name == "ast" || mod_name == "network_core" || mod_name == "network") { needs_whole_lib = true } - // One call per thunk process-wide (not idempotent); get-or-add avoids a silent rename to an undefined symbol. return if (g_exe_emitted_reg |> key_exists(reg_fn_name)) g_exe_emitted_reg[reg_fn_name] = true var reg_fn = LLVMGetNamedFunction(g_mod, reg_fn_name) if (reg_fn == null) { reg_fn = LLVMAddFunctionWithType(g_mod, reg_fn_name, register_mod_type) } - LLVMBuildCall2(init_builder, register_mod_type, reg_fn, array(), "") + let once_type = jit_fn_type($(name, reg : void?) : void? {}) + var once = LLVMGetNamedFunction(g_mod, "jit_register_module_once") + if (once == null) { + once = LLVMAddFunctionWithType(g_mod, "jit_register_module_once", once_type) + } + LLVMBuildCall2(init_builder, once_type, once, + array(get_string_constant_ptr(init_builder, mod_name), reg_fn), "") } def make_call(expr : ExprCallFunc?) { @@ -1005,24 +1010,25 @@ def private emit_module_registration(m : Module?; dynamic_modules : table(), "") + var reg_fn = declare_extern_fn(reg_fn_name, register_mod_type) + let once_type = jit_fn_type($(name, reg : void?) : void? {}) + var once = declare_extern_fn("jit_register_module_once", once_type) + LLVMBuildCall2(builder, once_type, once, + array(get_string_constant_ptr(builder, mod_name), + reg_fn), "") } // Creates main function that initializes a standalone JIT context, registers // all compiled functions, runs init scripts, then calls the program entry point. -def public inject_main(program_context : Context?; ctx : LLVMContextRef; // nolint:STYLE037,STYLE038 — the standalone-exe entry emitter: one emission block per runtime feature, in startup order - prog : Program ?; entry_point : string; mod : LLVMOpaqueModule?; - var types : PrimitiveTypes?, var uids : UidNodes?; strict : bool; - register_all_modules : bool = false) : tuple { - g_exe_emitted_reg |> clear() // per-codegen-run: one registration call per module thunk - let builder = LLVMCreateBuilder() - defer() { - LLVMDisposeBuilder(builder) - } +struct public StandaloneFunctions { + funcs : array + used_modules : table + any_pinvoke : bool + ok : bool +} + + +def public collect_standalone_functions(prog : Program?; strict : bool) : StandaloneFunctions { // Collect ALL used functions from the program (not just JIT-compiled ones from `funcs`). // There shouldn't be any `das` functions in standalone_exe, otherwise it // will crash. @@ -1048,56 +1054,39 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli } if (strict && has_no_jit) { to_log(LOG_ERROR, "Cannot build standalone exe: some functions are no_jit in strict mode\n") - return (fn = null, link_whole_lib = false) + return <- StandaloneFunctions(funcs <- funcs, used_modules <- used_modules, any_pinvoke = any_pinvoke) } + return <- StandaloneFunctions(funcs <- funcs, used_modules <- used_modules, any_pinvoke = any_pinvoke, ok = true) +} - // --jit-register-all-modules: compiler-driver exes may recompile targets needing UnitTest at - // runtime. Force it only in main()'s dynamic-module loop below — NOT via used_modules, else - // initialize_modules() emits a second register call ("Module 'UnitTest' already created"). - var force_dynamic_modules : table - if (register_all_modules) { - force_dynamic_modules["UnitTest"] = true - } +def private find_exe_entry(var funcs : array; var uids : UidNodes?; entry_point : string) : tuple { var start_fn_name = "" var no_return = true var bool_return = false for (fn in funcs) { if (fn.name == entry_point && fn.arguments.empty()) { assume fnmna = uids.get_dll_fn_name(fn).impl() - if (fn.result.isVoid) { - start_fn_name = fnmna - no_return = true - } elif (fn.result.baseType == Type.tInt) { - start_fn_name = fnmna - no_return = false - } elif (fn.result.baseType == Type.tBool) { - start_fn_name = fnmna - no_return = false - bool_return = true - } + let is_void = fn.result.isVoid + let is_bool = fn.result.baseType == Type.tBool + let takes_it = is_void || is_bool || fn.result.baseType == Type.tInt + start_fn_name = takes_it ? fnmna : start_fn_name + no_return = takes_it ? is_void : no_return + bool_return = takes_it ? is_bool : bool_return } } - if (start_fn_name |> empty()) { - to_log(LOG_ERROR, "entrypoint `{entry_point}()` not found in input file.\n") - return (fn = null, link_whole_lib = false) - } - let main_fn_type = LLVMFunctionType(types.t_int32, - fixed_array( - types.t_int32, // argc - types.LLVMVoidPtrType() // argv - ) - ) - // wasm32-emscripten: emcc's libstandalonewasm already defines `main`, - // emit `__main_argc_argv` instead and let crt1 chain through. - let main_sym = g_target_is_wasm ? "__main_argc_argv" : "main" - let main_fn = LLVMAddFunctionWithType(mod, main_sym, main_fn_type) - let entry = LLVMAppendBasicBlockInContext(ctx, main_fn, "entry") - LLVMPositionBuilderAtEnd(builder, entry) + return (name = start_fn_name, no_return = no_return, bool_return = bool_return) +} + +def public emit_standalone_runtime_init(builder : LLVMBuilderRef; prog : Program?; var types : PrimitiveTypes?; // nolint:STYLE038 - one linear registration sequence, in startup order; a split would hide which step runs when + register_all_modules : bool; force_dynamic_modules : table; + var used_modules : table&; + var dynamic_modules : table&; + guest : bool = false) : bool { + g_exe_emitted_reg |> clear() // per-codegen-run: one registration call per module thunk // Collect dynamic module names — used by CollectExternVisitor to skip dlopen'd modules - var dynamic_modules : table for_each_registered_dynamic_module() $(_path, _mod_name, das_name) { dynamic_modules[das_name] = true } @@ -1196,30 +1185,19 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli LLVMBuildCall2(builder, init_modules_type, init_modules_fn, array(), "") // Dynamic modules load Quiet (sibling DT_NEEDED ordering): retry after the last registration - // and fatal on anything still unloadable, naming the module + dlopen error. Native only — on - // wasm those modules also register via static thunks, so the failed resolve is BY DESIGN. if (ships_dynamic && !g_target_is_wasm) { let fin_dyn_type = LLVMFunctionType(types.t_void, array()) - var fin_dyn = LLVMAddFunctionWithType(g_mod, "jit_finalize_dynamic_modules", fin_dyn_type) + let fin_dyn_name = guest ? "jit_lib_finalize_dynamic_modules" : "jit_finalize_dynamic_modules" + var fin_dyn = LLVMAddFunctionWithType(g_mod, fin_dyn_name, fin_dyn_type) LLVMBuildCall2(builder, fin_dyn_type, fin_dyn, array(), "") } - // Init argc, argv - let set_cmd_args_type = LLVMFunctionType(types.t_void, - fixed_array( - types.t_int32, // argc - types.LLVMVoidPtrType() // argv - ) - ) - var jit_set_cmd_args = LLVMAddFunctionWithType( - g_mod, "jit_set_command_line_arguments", set_cmd_args_type - ) - let argc = LLVMGetParam(main_fn, 0 |> uint); // argc - let argv = LLVMGetParam(main_fn, 1 |> uint); // argv - LLVMBuildCall2(builder, set_cmd_args_type, - jit_set_cmd_args, fixed_array(argc, argv), "") + return ships_dynamic +} +def public emit_create_standalone_ctx(builder : LLVMBuilderRef; program_context : Context?; prog : Program?; + var types : PrimitiveTypes?; any_pinvoke : bool) : LLVMOpaqueValue? { // Determine the context stack size from options (mirrors Program::getContextStackSize) var context_stack_size = uint64(prog.policies.stack) let stack_opt = find_arg(prog._options, "stack") @@ -1254,7 +1232,17 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli types.ConstI64(context_stack_size) ) let global_context = LLVMBuildCall2(builder, create_ctx_type, jit_create_context, params, "") + return global_context +} + +def public emit_standalone_context_init(builder : LLVMBuilderRef; fusion_builder : LLVMBuilderRef; // nolint:STYLE038 - one linear startup sequence sharing the context value; a split would hide which step runs when + program_context : Context ?; ctx : LLVMContextRef; prog : Program?; + mod : LLVMOpaqueModule?; var types : PrimitiveTypes?; var uids : UidNodes?; + var funcs : array; global_context : LLVMOpaqueValue?; + register_all_modules : bool; ships_dynamic : bool; + var used_modules : table&; + dynamic_modules : table) : bool { // Pre-build init_globals so its function-pointer globals exist before // collect_external_functions walks them (issue #2582). Actual init_globals(ctx) // call is emitted further down at its runtime position in main_fn. @@ -1322,9 +1310,9 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli // Whole-compiler-lib link: register the fusion engine (needed by simulate). Skip on wasm — // cross-link only sees libDaScript_runtime.a (no jit_register_fusion), and exes never read it (#2805). if (needs_whole_lib && !g_target_is_wasm) { - let reg_fusion_type = LLVMFunctionType(types.t_void, array()) - let reg_fusion_fn = LLVMAddFunctionWithType(g_mod, "jit_register_fusion", reg_fusion_type) - LLVMBuildCall2(builder, reg_fusion_type, reg_fusion_fn, array(), "") + let reg_fusion_type = jit_fn_type($() : void {}) + var reg_fusion_fn = declare_extern_fn("jit_register_fusion", reg_fusion_type) + LLVMBuildCall2(fusion_builder, reg_fusion_type, reg_fusion_fn, array(), "") } // void jit_register_standalone_variable ( Context * ctx, uint64_t index, const char * name, uint64_t mnh, uint64_t offset, int shared ) @@ -1354,10 +1342,14 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli LLVMBuildCall2(builder, init_global_var_type, init_global_var, var_params, "") } // the sealed lookups are adopted right after the context is created - before any registration - // call, so no JIT code can ever probe an unadopted table, however LLVM orders the calls { let resume_block = LLVMGetInsertBlock(builder) - LLVMPositionBuilderBefore(builder, LLVMGetNextInstruction(global_context)) + if (LLVMIsAInstruction(global_context) != null) { + LLVMPositionBuilderBefore(builder, LLVMGetNextInstruction(global_context)) + } else { + let entry = LLVMGetEntryBasicBlock(LLVMGetBasicBlockParent(resume_block)) + LLVMPositionBuilder(builder, entry, LLVMGetFirstInstruction(entry)) + } emit_exe_lookups(builder, ctx, types, global_context, collected.registered, program_context) LLVMPositionBuilderAtEnd(builder, resume_block) } @@ -1418,6 +1410,72 @@ def public inject_main(program_context : Context?; ctx : LLVMContextRef; // noli fixed_array(global_context, LLVMBuildPointerCast(builder, init_script_fn, types.LLVMVoidPtrType(), "")), "") } + return needs_whole_lib +} +def public inject_main(program_context : Context?; ctx : LLVMContextRef; // nolint:STYLE037,STYLE038 — the standalone-exe entry emitter: one emission block per runtime feature, in startup order + prog : Program ?; entry_point : string; mod : LLVMOpaqueModule?; + var types : PrimitiveTypes?, var uids : UidNodes?; strict : bool; + register_all_modules : bool = false) : tuple { + let builder = LLVMCreateBuilder() + defer() { + LLVMDisposeBuilder(builder) + } + var std_fns <- collect_standalone_functions(prog, strict) + var funcs <- std_fns.funcs + var used_modules <- std_fns.used_modules + let any_pinvoke = std_fns.any_pinvoke + if (!std_fns.ok) { + return (fn = null, link_whole_lib = false) + } + + var force_dynamic_modules : table + if (register_all_modules) { + force_dynamic_modules["UnitTest"] = true + } + + let found = find_exe_entry(funcs, uids, entry_point) + let start_fn_name = found.name + let no_return = found.no_return + let bool_return = found.bool_return + if (start_fn_name |> empty()) { + to_log(LOG_ERROR, "entrypoint `{entry_point}()` not found in input file.\n") + return (fn = null, link_whole_lib = false) + } + + let main_fn_type = LLVMFunctionType(types.t_int32, + fixed_array( + types.t_int32, // argc + types.LLVMVoidPtrType() // argv + ) + ) + let main_sym = g_target_is_wasm ? "__main_argc_argv" : "main" + let main_fn = LLVMAddFunctionWithType(mod, main_sym, main_fn_type) + let entry = LLVMAppendBasicBlockInContext(ctx, main_fn, "entry") + LLVMPositionBuilderAtEnd(builder, entry) + + var dynamic_modules : table + let ships_dynamic = emit_standalone_runtime_init(builder, prog, types, register_all_modules, + force_dynamic_modules, used_modules, dynamic_modules) + let set_cmd_args_type = LLVMFunctionType(types.t_void, + fixed_array( + types.t_int32, // argc + types.LLVMVoidPtrType() // argv + ) + ) + var jit_set_cmd_args = LLVMAddFunctionWithType( + g_mod, "jit_set_command_line_arguments", set_cmd_args_type + ) + let argc = LLVMGetParam(main_fn, 0 |> uint); // argc + let argv = LLVMGetParam(main_fn, 1 |> uint); // argv + LLVMBuildCall2(builder, set_cmd_args_type, + jit_set_cmd_args, fixed_array(argc, argv), "") + + + let global_context = emit_create_standalone_ctx(builder, program_context, prog, types, any_pinvoke) + + let needs_whole_lib = emit_standalone_context_init(builder, builder, program_context, ctx, prog, mod, + types, uids, funcs, global_context, register_all_modules, ships_dynamic, used_modules, dynamic_modules) + var main_params = fixed_array(global_context) // Match the interpreter's WebLoop: on wasm a program exporting `update` runs as init() once + // update() per browser frame + shutdown() (the browser can't block in main's while-loop). `main` // stays the desktop driver, bypassed here — so the same .das cross-compiles unchanged for web. diff --git a/modules/dasLLVM/daslib/llvm_jit.das b/modules/dasLLVM/daslib/llvm_jit.das index 038f49935d..4b3d9b4862 100644 --- a/modules/dasLLVM/daslib/llvm_jit.das +++ b/modules/dasLLVM/daslib/llvm_jit.das @@ -89,7 +89,7 @@ def set_debug_linkage(value : LLVMOpaqueValue?) { LLVMSetLinkage(value, LLVMLinkage.LLVMInternalLinkage) } -def set_public_linkage(value : LLVMOpaqueValue?) { +def public set_public_linkage(value : LLVMOpaqueValue?) { LLVMSetLinkage(value, LLVMLinkage.LLVMDLLExportLinkage) LLVMSetDLLStorageClass(value, LLVMDLLStorageClass.LLVMDLLExportStorageClass) } @@ -8384,5 +8384,7 @@ def init_llvm_jit_module_options() { this_module() |> add_module_option("jit_target", Type.tString) this_module() |> add_module_option("jit_split_modules", Type.tInt) this_module() |> add_module_option("jit_obj_cache", Type.tBool) + this_module() |> add_module_option("jit_lib", Type.tBool) + this_module() |> add_module_option("jit_lib_export_marked", Type.tBool) } } diff --git a/modules/dasLLVM/daslib/llvm_jit_cli.das b/modules/dasLLVM/daslib/llvm_jit_cli.das index 35b7eaf975..3b29a4b21f 100644 --- a/modules/dasLLVM/daslib/llvm_jit_cli.das +++ b/modules/dasLLVM/daslib/llvm_jit_cli.das @@ -74,6 +74,18 @@ struct public JitCliOptions { @clarg_doc = "JIT: explicit linker binary (overrides default c++/clang/lld-link). Windows-MSVC builds a link.exe-flavored cmd (/DLL /OUT:) — pass a link.exe-compatible linker; elsewhere match the compiler daslang was built with to avoid sanitizer runtime mismatch." path_to_linker : Option + @clarg_name = "jit-lib" + @clarg_doc = "JIT: emit a C-ABI native library plus its C header instead of running the program. Script-level pin: options jit_lib = true" + lib : Option + + @clarg_name = "jit-lib-export-marked" + @clarg_doc = "JIT (-lib): export every function the program already marks [export], not only the [export_c] ones. Selection only - unlike -lib-export-all it marks nothing itself, so a non-representable signature is still a hard error. Script-level pin: options jit_lib_export_marked = true" + lib_export_marked : Option + + @clarg_name = "jit-lib-static" + @clarg_doc = "JIT (-lib): produce a static archive (.a / .lib) instead of a shared library; the host then links libDaScript_runtime itself (the generated header records what). Under this flag --jit-path-to-linker names the ARCHIVER (llvm-ar / ar / lib), not a linker" + lib_static : Option + @clarg_name = "jit-register-all-modules" @clarg_doc = "JIT (-exe only): register all builtin native modules (math, fio, dasbind, ...) at exe startup, so a standalone compiler-driver exe can recompile arbitrary daslang at runtime" register_all_modules : Option diff --git a/modules/dasLLVM/daslib/llvm_jit_common.das b/modules/dasLLVM/daslib/llvm_jit_common.das index 1c8c1cbb78..6c9edfa2e1 100644 --- a/modules/dasLLVM/daslib/llvm_jit_common.das +++ b/modules/dasLLVM/daslib/llvm_jit_common.das @@ -562,6 +562,15 @@ def public jit_extern_type(t : Type) : LLVMTypeRef { return t == Type.tVoid ? g_prim_t.t_void : base_type_to_llvm_type(t) } +def public declare_extern_fn(name : string; typ : LLVMTypeRef) : LLVMOpaqueValue? { + var fn = LLVMGetNamedFunction(g_mod, name) + if (fn == null) { + fn = LLVMAddFunctionWithType(g_mod, name, typ) + } + return fn +} + + def public jit_add_extern(name : string; typ : LLVMTypeRef; fnAddr : void?; attrs : LLVMOpaqueAttributeRef? []) { var f = LLVMAddFunctionWithType(g_mod, name, typ) LLVMAddGlobalMapping(g_engine, f, fnAddr) @@ -1047,7 +1056,7 @@ def public emit_object_only(mod : LLVMOpaqueModule?; out_path : string; use_host LLVMSetTarget(mod, LLVMGetDefaultTargetTriple()) LLVMDisposeTargetData(dl) let error : string? - let file = add_obj_extension(out_path) + let file = artifact_path(out_path, JitArtifact.object) let filetype = LLVMCodeGenFileType.LLVMObjectFile let failed = LLVMTargetMachineEmitToFile(targetMachine, mod, file, filetype, error) if (failed != 0) { @@ -1085,7 +1094,7 @@ def public link_dll_from_objects(objs : array; out_path : string; path_t return false } let phase_tm = ref_time_ticks() - let ok = create_shared_library(objs[0], add_dll_extension(out_path), "{path_to_dascript_lib}", "{path_to_linker}", "\"@{rsp}\" {linker_string}", true, link_whole_lib, debug_info) + let ok = create_shared_library(objs[0], artifact_path(out_path, JitArtifact.jit_dll), "{path_to_dascript_lib}", "{path_to_linker}", "\"@{rsp}\" {linker_string}", true, link_whole_lib, debug_info) if (log_time) { to_log(LOG_INFO, "LLVM JIT time: link {jit_sec(get_time_usec(phase_tm))} ({length(objs)} objects)\n") } @@ -1095,44 +1104,36 @@ def public link_dll_from_objects(objs : array; out_path : string; path_t // @out_path - folder + file name, without extension. @path_to_dascript_lib - required on Windows, // no effect on Linux. @path_to_linker - linker override; Windows-MSVC defaults to lld-link (link.exe // flavor) from the LLVM package, c++/clang elsewhere. -def public write_dll(mod : LLVMOpaqueModule?; out_path : string; path_to_dascript_lib, path_to_linker, linker_string : string; link_whole_lib : bool; debug_info : bool = false; log_time : bool = false; codegen_opt_level : uint = 3u) { - // JIT DLL cache: emitted artifact only ever runs on this host. - with_default_target_machine(codegen_opt_level, true) $(targetMachine : LLVMTargetMachineRef) { +def public write_artifact(mod : LLVMOpaqueModule?; out_path : string; kind : JitArtifact; + path_to_dascript_lib, path_to_linker, linker_string : string; + link_whole_lib : bool; use_host_cpu : bool; debug_info : bool = false; + log_time : bool = false; codegen_opt_level : uint = 3u) { + with_default_target_machine(codegen_opt_level, kind == JitArtifact.jit_dll ? true : use_host_cpu) $(targetMachine : LLVMTargetMachineRef) { let error : string? - let file = add_obj_extension(out_path) - let filetype = LLVMCodeGenFileType.LLVMObjectFile + let file = artifact_path(out_path, JitArtifact.object) var phase_tm = ref_time_ticks() - let failed = LLVMTargetMachineEmitToFile(targetMachine, mod, file, filetype, error) - if (failed != 0) { - panic("write_dll: LLVMTargetMachineEmitToFile failed for {file}") + if (LLVMTargetMachineEmitToFile(targetMachine, mod, file, LLVMCodeGenFileType.LLVMObjectFile, error) != 0) { + panic("write_artifact: LLVMTargetMachineEmitToFile failed for {file}") } let t_emit = get_time_usec(phase_tm) phase_tm = ref_time_ticks() - let ok = create_shared_library(file, add_dll_extension(out_path), "{path_to_dascript_lib}", "{path_to_linker}", "{linker_string}", true, link_whole_lib, debug_info) + let target = artifact_path(out_path, kind) + var ok = false + if (kind == JitArtifact.static_lib) { + ok = create_static_library(file, target, "{path_to_linker}") + } else { + ok = create_shared_library(file, target, "{path_to_dascript_lib}", "{path_to_linker}", + "{linker_string}", kind != JitArtifact.exe, link_whole_lib, debug_info) + } if (log_time) { to_log(LOG_INFO, "LLVM JIT time: emit-obj {jit_sec(t_emit)} link {jit_sec(get_time_usec(phase_tm))}\n") } if (!ok) { - panic("write_dll: link failed for {out_path}") + panic("write_artifact: {kind == JitArtifact.static_lib ? "archive" : "link"} failed for {target}") } } } -def public write_exe(mod : LLVMOpaqueModule?; out_path : string; path_to_dascript_lib, path_to_linker, linker_string : string; link_whole_lib : bool; use_host_cpu : bool; debug_info : bool = false) { - // Standalone exe: generic (redistributable) by default; use_host_cpu targets the box so - // tuner-generated host-specific IR legalizes (the generic target aborts codegen on it). - // Codegen level 3 deliberate: shipped artifacts, no cache guard (ARCHITECTURE.md 1.2) - with_default_target_machine(3u, use_host_cpu) $(targetMachine : LLVMTargetMachineRef) { - let error : string? - let file = add_obj_extension(out_path) - let filetype = LLVMCodeGenFileType.LLVMObjectFile - LLVMTargetMachineEmitToFile(targetMachine, mod, file, filetype, error) - let ok = create_shared_library(file, add_exe_extension(out_path), "{path_to_dascript_lib}", "{path_to_linker}", "{linker_string}", false, link_whole_lib, debug_info) - if (!ok) { - panic("write_exe: link failed for {out_path}") - } - } -} // Locate the wasm64 (memory64) libDaScript_runtime.a. Non-empty `override_path` wins (e.g. // --jit-runtime-lib CLI flag; a missing file logs a warning); otherwise auto-locate at @@ -1173,7 +1174,7 @@ def public write_wasm(mod : LLVMOpaqueModule?; out_path : string; triple : strin let error : string? // emit_object_only: write the object verbatim to out_path (no `.o.o` // double extension) so the caller has a predictable, linkable artifact. - let obj = emit_object_only ? out_path : add_obj_extension(out_path) + let obj = emit_object_only ? out_path : artifact_path(out_path, JitArtifact.object) let filetype = LLVMCodeGenFileType.LLVMObjectFile LLVMTargetMachineEmitToFile(targetMachine, mod, obj, filetype, error) if (emit_object_only) { diff --git a/modules/dasLLVM/daslib/llvm_jit_run.das b/modules/dasLLVM/daslib/llvm_jit_run.das index 6f472f2547..09318175ac 100644 --- a/modules/dasLLVM/daslib/llvm_jit_run.das +++ b/modules/dasLLVM/daslib/llvm_jit_run.das @@ -7,6 +7,8 @@ options no_global_variables = false require llvm/daslib/llvm_boost require llvm/daslib/llvm_dll_utils require llvm/daslib/llvm_exe +require llvm/daslib/jit_standalone +require daslib/c_api_header require llvm/daslib/llvm_aot require llvm/daslib/llvm_jit require llvm/daslib/llvm_jit_common @@ -42,7 +44,7 @@ let LLVM_JIT_CODEGEN_VERSION : uint64 = 0x78ul // 0x78: a standalone exe adopt // Read by tests-cpp/small/test_jit_emitter_pin.cpp: FNV-1a64 of the emitter sources // (normalized to LF; file list in the test) -let LLVM_JIT_EMITTER_HASH : uint64 = 0xa85abd6c7514b352ul +let LLVM_JIT_EMITTER_HASH : uint64 = 0x8e60d45ec7ed2378ul let JIT_FNV_PRIME : uint64 = 1099511628211ul @@ -581,8 +583,8 @@ def private run_split_codegen(prog : Program?; ctx : Context?; funcs : array push(obj) // link order = partition order, hit or miss if (obj_cache && stat(obj).is_valid) { cached++ @@ -733,7 +735,8 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL to_log(LOG_WARNING, "LLVM JIT: dll mode requested but this daslang build is static - no DLL cache, " + "every run pays full in-memory codegen\n") } - let gen_exe = prog.policies.jit_exe_mode + let gen_lib = cli_opts.lib |> unwrap_or((prog._options |> find_arg("jit_lib")) ?as tBool ?? false) + let gen_exe = prog.policies.jit_exe_mode || gen_lib // Drives the $::is_standalone_exe JIT intrinsic in llvm_jit_intrin so daslang // code can constant-fold to true when this codegen run produces a standalone exe. llvm_jit_intrin::g_jit_exe_mode = gen_exe @@ -797,6 +800,13 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL if (!empty(target_triple) && get_target_triple() != target_triple) { panic("LLVM JIT: the target triple `{target_triple}` is not on the command line - pass --jit-target={target_triple} after the `--` separator (the compile-time target folds read argv, so `options jit_target` alone folds the host's tiers into a cross artifact)") } + let lib_static = cli_opts.lib_static |> unwrap_or(false) + if ((cli_opts.lib_static |> is_some()) && !gen_lib) { + to_log(LOG_WARNING, "LLVM JIT: --jit-lib-static applies to -lib only - ignored for this run\n") + } + if (gen_lib && !(target_triple |> empty())) { + panic("LLVM LIB: -lib does not cross-compile (--jit-target) - build the library on its target host") + } let gen_wasm = gen_exe && (target_triple |> starts_with("wasm")) // Optional CLI override for the wasm runtime archive location. Falls // back to `options jit_runtime_lib = "..."` then write_wasm's @@ -826,7 +836,6 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL // nolint:STYLE014 // the disable pass consults target-gated intrinsic tables (has_intrinsic) — the // g_target_* truth must be set BEFORE it runs, not first at init_jit time below. - // Standalone exes target a GENERIC CPU (write_exe use_host_cpu=false default), so // host cpuid features must not leak into their emission (host_features=false) — // EXCEPT a native exe carrying [llvm_code] kernels: a local-use artifact by // definition, it targets the current box in BOTH the flags and the machine (the ONE @@ -888,10 +897,7 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL unsafe { delete disableJitVisitor } - // emit_aot_object always writes an object even with no functions to emit (a fully-no_aot - // module): an empty .o with just the fileinfo ctor, so the build always has the file it - // expects and those functions interpret at load (as Program::linkCppAot skips them). - if (!empty(funcs) || emit_aot_object) { + if (!empty(funcs) || emit_aot_object || gen_lib) { let totalTime = ref_time_ticks() // the front end's log_compile_time option extends into the backend: per-phase wall below let log_jit_time = prog._options |> find_arg("log_compile_time") ?as tBool ?? prog.policies.log_compile_time @@ -988,7 +994,17 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL reopen_and_gc_dll(g_dynamic_lib_handle, output_path, funcs, disabled, uids, lto_probe ? "" : dll_hash_basename, sres.objs) } elif (recompile_prog) { jit_has_externals = irgen_functions(ctx, uids, attrs, funcs, jit_flags, jit_mode, prog) - if (gen_exe) { + if (gen_lib) { + let export_marked = cli_opts.lib_export_marked |> unwrap_or((prog._options |> find_arg("jit_lib_export_marked")) ?as tBool ?? false) + let res = inject_lib(ctx, g_ctx, prog, g_mod, g_prim_t, uids, exe_strict, + prog.policies.export_public_functions || export_marked, output_path, + cli_opts.register_all_modules |> unwrap_or(false)) + LINK_WHOLE_LIB = res.link_whole_lib + if (res.fn == null) { + finalize_jit(use_dll, gen_exe, g_dynamic_lib_handle) + panic("LLVM LIB: no library written for {output_path}") + } + } elif (gen_exe) { let res = inject_main(ctx, g_ctx, prog, exe_main, g_mod, g_prim_t, uids, exe_strict, cli_opts.register_all_modules |> unwrap_or(false)) LINK_WHOLE_LIB = res.link_whole_lib @@ -1030,6 +1046,16 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL to_log(LOG_INFO, "LLVM JIT: compile-only - module built, optimized and verified; no artifact written, nothing installed\n") } elif (emit_aot_object) { emit_object_only(g_mod, output_path, use_host_cpu) + } elif (gen_lib) { + mkdir_rec(dir_name(output_path)) + write_artifact(g_mod, output_path, lib_static ? JitArtifact.static_lib : JitArtifact.shared_lib, + path_to_shared_lib, path_to_linker, linker_string, LINK_WHOLE_LIB, exe_host_cpu, + debug_info, log_jit_time) + let lib_names = CNames(prefix = lib_prefix_from_path(output_path), this_module = prog.getThisModule) + if (!emit_c_header(g_lib_exports, lib_names, string(prog.getThisModule.fileName), + output_path, lib_link_note(lib_static, LINK_WHOLE_LIB))) { + panic("LLVM LIB: the library is written but its C header is not - {output_path}.h") + } } elif (gen_wasm) { mkdir_rec(dir_name(output_path)) // path_to_linker is reused as the optional emcc override @@ -1038,10 +1064,12 @@ def public run_jit(prog : Program?; var ctx : Context?) : bool { // nolint:STYL write_wasm(g_mod, output_path, target_triple, path_to_linker, jit_has_externals, runtime_lib_override, emit_object) } elif (gen_exe) { mkdir_rec(dir_name(output_path)) - write_exe(g_mod, output_path, path_to_shared_lib, path_to_linker, linker_string, LINK_WHOLE_LIB, exe_host_cpu, debug_info) + write_artifact(g_mod, output_path, JitArtifact.exe, path_to_shared_lib, path_to_linker, + linker_string, LINK_WHOLE_LIB, exe_host_cpu, debug_info) } elif (use_dll) { mkdir_rec(dir_name(output_path)) - write_dll(g_mod, output_path, path_to_shared_lib, path_to_linker, linker_string, LINK_WHOLE_LIB, debug_info, log_jit_time, opt_level |> uint) + write_artifact(g_mod, output_path, JitArtifact.jit_dll, path_to_shared_lib, path_to_linker, + linker_string, LINK_WHOLE_LIB, true, debug_info, log_jit_time, opt_level |> uint) let no_keep_objs : array // monolith artifacts all carry the dll-hash prefix reopen_and_gc_dll(g_dynamic_lib_handle, output_path, funcs, disabled, uids, dll_hash_basename, no_keep_objs) } diff --git a/skills/daslang/references/cli-and-config.md b/skills/daslang/references/cli-and-config.md index 56e6ed1502..5b4b9d2253 100644 --- a/skills/daslang/references/cli-and-config.md +++ b/skills/daslang/references/cli-and-config.md @@ -83,6 +83,9 @@ interpreter wires it to `-?` instead: help : bool ``` +A `-lib` library never owns argv at all: the host's process arguments are whatever the host was +started with, so a library reads its configuration from its C parameters, not from `clargs`. + Only an `-exe` binary owns argv, so `-h` / `--help` - and `parse_args_with_help`'s automatic help flag - are reachable only there. diff --git a/skills/daslang/references/modules-and-stdlib.md b/skills/daslang/references/modules-and-stdlib.md index 7da35ea4fb..77470a534b 100644 --- a/skills/daslang/references/modules-and-stdlib.md +++ b/skills/daslang/references/modules-and-stdlib.md @@ -23,9 +23,9 @@ The only enforced ordering is `module` before any type declaration; `options` / `require` otherwise interleave. A file with no `module` line is a program, named by its file stem. `[export]` makes a function callable from the host by name, and `[export_c]` is an `[export]` that -also crosses to C - a standalone context declares it in the C half of the header it generates, so a -C host needs no daslang API (`require daslib/export_c`; `[export_c(name = "...")]` picks the C -symbol, which is how two overloads both reach C); `[init]` / `[finalize]` run at context +`daslang -lib` also surfaces as a C function in the header it generates (`-lib`/`-jit`/`-exe` carry +the annotation; a non-JIT compile of that source needs `require daslib/export_c`) (`[export_c(name = "...")]` +picks the C symbol, which is how two overloads both reach C); `[init]` / `[finalize]` run at context init / shutdown (no arguments, no return). `main` is a convention, not a keyword: it returns `void` unless declared `def main() : int`, whose return value is the process exit code (do not `panic` to force one). diff --git a/src/ast/ast_export.cpp b/src/ast/ast_export.cpp index 4fefc5f3d4..2eb572cb4a 100644 --- a/src/ast/ast_export.cpp +++ b/src/ast/ast_export.cpp @@ -131,6 +131,14 @@ namespace das { return true; }, "*"); } + void exportPublicFunctions( Module * thisModule ) { + for ( auto & fn : thisModule->functions.each() ) { + if ( fn->privateFunction || fn->builtIn || fn->generated || fn->isTemplate ) continue; + if ( fn->macroInit || fn->macroFunction || fn->init || fn->shutdown ) continue; + if ( fn->isClassMethod || fn->lambda || fn->generator || fn->fromGeneric ) continue; + fn->exports = true; + } + } void markModuleVarsUsed( ModuleLibrary &, Module * inWhichModule ) { for ( auto & var : inWhichModule->globals.each() ) { var->used = false; @@ -351,6 +359,7 @@ namespace das { MarkSymbolUse vis(false); vis.tw = logs; visit(vis); + if ( policies.export_public_functions ) vis.exportPublicFunctions(thisModule.get()); vis.markUsedFunctions(library, false, false, nullptr); vis.markVarsUsed(library, false); } diff --git a/src/builtin/module_builtin_rtti.cpp b/src/builtin/module_builtin_rtti.cpp index faf85fa361..1fd2b290e7 100644 --- a/src/builtin/module_builtin_rtti.cpp +++ b/src/builtin/module_builtin_rtti.cpp @@ -974,6 +974,7 @@ namespace das { addField("no_lint"); addField("no_init_check"); addField("export_all"); + addField("export_public_functions"); addField("serialize_main_module"); addField("keep_alive"); addField("very_safe_context"); diff --git a/src/builtin/module_jit.cpp b/src/builtin/module_jit.cpp index 44cf63fbae..1a3f5011f2 100644 --- a/src/builtin/module_jit.cpp +++ b/src/builtin/module_jit.cpp @@ -1317,8 +1317,31 @@ extern "C" { #endif return run_link_cmd(cmd.c_str(), libraryName, "Library", context); } + + bool create_static_library ( const char * objFilePath, const char * libraryName, const char * customTool, Context * context ) { + if ( !check_file_present(objFilePath) ) { + LOG(LogLevel::error) << "File '" << objFilePath << "' , containing compiled definitions, does not exist\n"; + return false; + } + remove(libraryName); + std::string cmd; + #if defined(_WIN32) || defined(_WIN64) + #if defined(_MSC_VER) + const auto tool = find_linker(customTool, "llvm-lib.exe", "lib"); + cmd = fmt::format(FMT_STRING("\"\"{}\" /nologo /OUT:\"{}\" \"{}\" 2>&1\""), tool.c_str(), libraryName, objFilePath); + #else + const auto tool = find_linker(customTool, "llvm-ar.exe", "ar"); + cmd = fmt::format(FMT_STRING("\"\"{}\" rcs \"{}\" \"{}\" 2>&1\""), tool.c_str(), libraryName, objFilePath); + #endif + #else + const auto tool = find_linker(customTool, "llvm-ar", "ar"); + cmd = fmt::format(FMT_STRING("\"{}\" rcs \"{}\" \"{}\" 2>&1"), tool.c_str(), libraryName, objFilePath); + #endif + return run_link_cmd(cmd.c_str(), libraryName, "Archive", context); + } #else bool create_shared_library ( const char * objFilePath, const char * libraryName, [[maybe_unused]] const char * dasLib, const char * customLinker, const char * extraLinkerArgs, bool isShared, bool linkWholeLib, bool debugInfo, Context *context ) { return true; } + bool create_static_library ( const char * objFilePath, const char * libraryName, const char * customTool, Context * context ) { return true; } #endif // ===== --jit-split-modules parallel optimize+emit ===== @@ -1689,6 +1712,9 @@ extern "C" { addExternInline(*this, lib, "create_shared_library", SideEffects::worstDefault, "create_shared_library") ->args({"objFilePath","libraryName","dasLib","customLinker","extraLinkerArgs","isShared","linkWholeLib","debugInfo","context"}); + addExternInline(*this, lib, "create_static_library", + SideEffects::worstDefault, "create_static_library") + ->args({"objFilePath","libraryName","customTool","context"}); addExternInline(*this, lib, "jit_par_emit_begin", SideEffects::worstDefault, "jit_par_emit_begin"); addExternInline(*this, lib, "jit_par_emit_add", @@ -1929,12 +1955,100 @@ DAS_API void das_ensure_environment () { das::daScriptEnvironment::ensure(); } +static das::atomic g_jitLibShutdown(0); +static das::string g_jitLibCreateError; +static das::daScriptEnvironment * g_jitLibEnv = nullptr; +static bool g_jitLibGuest = false; +static bool g_jitLibOwner = false; + +DAS_API void jit_lib_shutdown () { + if ( !g_jitLibOwner || !g_jitLibEnv ) return; + int expected = 0; + if ( !g_jitLibShutdown.compare_exchange_strong(expected, 1) ) return; + if ( das::daScriptEnvironment::getBound()!=g_jitLibEnv ) { + das::daScriptEnvironment::setBound(g_jitLibEnv); + } + if ( !das::daScriptEnvironment::getOwned() ) { + das::daScriptEnvironment::setOwned(g_jitLibEnv); + } + das::Module::ShutdownStandalone(); + g_jitLibEnv = nullptr; +} + +static void jit_lib_atexit_shutdown () { + jit_lib_shutdown(); +} + +DAS_API void * jit_register_module_once ( const char * dasName, das::Module * (*reg)() ) { + das::daScriptEnvironment::ensure(); + if ( das::Module * have = das::Module::require(dasName ? dasName : "") ) return have; + return reg(); +} + +DAS_API void jit_lib_arm_shutdown () { + if ( g_jitLibGuest ) return; + g_jitLibOwner = true; + static bool armed = (atexit(&jit_lib_atexit_shutdown), true); + (void)armed; +} + +DAS_API int32_t jit_lib_run_once ( int32_t * guard, void (*fn)() ) { + static das::mutex once_mutex; + das::lock_guard lock(once_mutex); + if ( *guard ) { + if ( g_jitLibEnv && das::daScriptEnvironment::getBound()!=g_jitLibEnv ) { + das::daScriptEnvironment::setBound(g_jitLibEnv); + } + return 1; + } + das::daScriptEnvironment::ensure(); + g_jitLibGuest = das::daScriptEnvironment::getBound()->modules != nullptr; + g_jitLibEnv = das::daScriptEnvironment::getBound(); + *guard = 1; + fn(); + return 1; +} + +DAS_API int32_t jit_lib_invoke_guarded ( das::Context * ctx, void (*tramp)(das::Context *, void *), void * frame ) { + if ( !ctx ) return 0; + ctx->clearException(); + if ( ctx->contextMutex ) { + das::lock_guard guard(*ctx->contextMutex); + return ctx->runWithCatch([&]() { tramp(ctx, frame); }) ? 1 : 0; + } + return ctx->runWithCatch([&]() { tramp(ctx, frame); }) ? 1 : 0; +} + +DAS_API das::Context * jit_lib_create_finish ( das::Context * ctx, int32_t ok ) { + if ( ok ) { + g_jitLibCreateError.clear(); + return ctx; + } + if ( ctx ) { + g_jitLibCreateError = ctx->getException() ? ctx->getException() : "unknown exception"; + delete ctx; + } else { + g_jitLibCreateError = "out of memory"; + } + return nullptr; +} + +DAS_API const char * jit_lib_last_error ( das::Context * ctx ) { + if ( !ctx ) return g_jitLibCreateError.empty() ? nullptr : g_jitLibCreateError.c_str(); + return ctx->getException(); +} + +DAS_API void jit_destroy_standalone_ctx ( das::Context * ctx ) { + delete ctx; +} + DAS_API void jit_initialize_modules () { // No need to initialize modules. JIT will generate required calls. das::daScriptEnvironment::ensure(); } DAS_API void jit_initialize_modules_done () { + if ( g_jitLibGuest ) return; das::Module::Initialize(); } @@ -2003,6 +2117,11 @@ DAS_API void jit_finalize_dynamic_modules () { } } +DAS_API void jit_lib_finalize_dynamic_modules () { + das::retry_pending_dynamic_modules(); + das::report_pending_dynamic_modules(); +} + // ABI shim: -exe binaries emitted before the resolving form link this runtime dynamically // and still import the 3-argument name. DAS_API void jit_register_native_path ( const char * mod_name, const char * src_path, const char * dst_path ) { diff --git a/tests-cpp/small/test_jit_lib_guard.cpp b/tests-cpp/small/test_jit_lib_guard.cpp new file mode 100644 index 0000000000..7be7ad35c4 --- /dev/null +++ b/tests-cpp/small/test_jit_lib_guard.cpp @@ -0,0 +1,110 @@ +#include + +#include "daScript/daScript.h" + +#include +#include + +extern "C" { + DAS_API das::Context * jit_create_standalone_ctx ( uint64_t totalVariables, + uint64_t totalFunctions, + uint64_t globalStringHeapSize, + uint64_t globalsSize, + uint64_t sharedSize, + bool pinvoke, + uint64_t stackSize ); + DAS_API int32_t jit_lib_invoke_guarded ( das::Context * ctx, + void (*tramp)(das::Context *, void *), + void * frame ); + DAS_API das::Context * jit_lib_create_finish ( das::Context * ctx, int32_t ok ); + DAS_API const char * jit_lib_last_error ( das::Context * ctx ); + DAS_API void jit_destroy_standalone_ctx ( das::Context * ctx ); + DAS_API int32_t jit_lib_run_once ( int32_t * guard, void (*fn)() ); + DAS_API void * jit_register_module_once ( const char * dasName, das::Module * (*reg)() ); +} + +namespace { + +static int g_ran = 0; + +static void tramp_quiet ( das::Context *, void * frame ) { + if ( frame ) *(int *) frame = 7; + g_ran ++; +} + +static void tramp_raises ( das::Context * ctx, void * ) { + ctx->throw_error("boom in the body"); +} + +static int g_once_calls = 0; +static void bump_once () { g_once_calls ++; } + +static das::Context * make_ctx () { + return jit_create_standalone_ctx(0, 1, 0, 0, 0, false, 16 * 1024); +} + +} + +TEST_CASE("jit_lib_invoke_guarded runs a body and reports success") { + das::Context * ctx = make_ctx(); + REQUIRE(ctx != nullptr); + g_ran = 0; + int slot = 0; + CHECK(jit_lib_invoke_guarded(ctx, &tramp_quiet, &slot) == 1); + CHECK(g_ran == 1); + CHECK(slot == 7); + CHECK(jit_lib_last_error(ctx) == nullptr); + jit_destroy_standalone_ctx(ctx); +} + +TEST_CASE("jit_lib_invoke_guarded turns a das panic into a return code plus a message") { + das::Context * ctx = make_ctx(); + REQUIRE(ctx != nullptr); + CHECK(jit_lib_invoke_guarded(ctx, &tramp_raises, nullptr) == 0); + const char * err = jit_lib_last_error(ctx); + REQUIRE(err != nullptr); + CHECK(std::strstr(err, "boom in the body") != nullptr); + + int slot = 0; + CHECK(jit_lib_invoke_guarded(ctx, &tramp_quiet, &slot) == 1); + CHECK(slot == 7); + CHECK(jit_lib_last_error(ctx) == nullptr); + jit_destroy_standalone_ctx(ctx); +} + +TEST_CASE("jit_lib_create_finish drops a context whose init raised, and keeps the message") { + das::Context * ctx = make_ctx(); + REQUIRE(ctx != nullptr); + jit_lib_invoke_guarded(ctx, &tramp_raises, nullptr); + CHECK(jit_lib_create_finish(ctx, 0) == nullptr); + const char * err = jit_lib_last_error(nullptr); + REQUIRE(err != nullptr); + CHECK(std::strstr(err, "boom in the body") != nullptr); + + das::Context * good = make_ctx(); + REQUIRE(good != nullptr); + CHECK(jit_lib_create_finish(good, 1) == good); + CHECK(jit_lib_last_error(nullptr) == nullptr); + jit_destroy_standalone_ctx(good); +} + +TEST_CASE("jit_lib_run_once runs a library's init in a process that already carries a runtime") { + das::daScriptEnvironment::ensure(); + REQUIRE(das::daScriptEnvironment::getBound() != nullptr); + REQUIRE(das::daScriptEnvironment::getBound()->modules != nullptr); + g_once_calls = 0; + int32_t guard = 0; + CHECK(jit_lib_run_once(&guard, &bump_once) == 1); + CHECK(g_once_calls == 1); + CHECK(guard == 1); + CHECK(jit_lib_run_once(&guard, &bump_once) == 1); + CHECK(g_once_calls == 1); + CHECK(das::daScriptEnvironment::getBound()->modules != nullptr); +} + +TEST_CASE("jit_register_module_once hands back a module the process already registered") { + das::daScriptEnvironment::ensure(); + das::Module * have = das::Module::require("math"); + REQUIRE(have != nullptr); + CHECK(jit_register_module_once("math", nullptr) == (void *) have); +} diff --git a/tests/jit_tests/jit_lib.das b/tests/jit_tests/jit_lib.das new file mode 100644 index 0000000000..f05619b83b --- /dev/null +++ b/tests/jit_tests/jit_lib.das @@ -0,0 +1,446 @@ +options gen2 +options no_aot + +require dastest/testing_boost +require daslib/fio +require daslib/strings_boost +require strings + + +let OUTPUT_DIR = "{get_das_root()}/build/tests" +let LIB_SCRIPT = "{OUTPUT_DIR}/jit_lib_probe.das" +let LIB_OUT = "{OUTPUT_DIR}/jit_lib_probe" +let BAD_SCRIPT = "{OUTPUT_DIR}/jit_lib_bad.das" +let BAD_OUT = "{OUTPUT_DIR}/jit_lib_bad" +let ALIGN_SCRIPT = "{OUTPUT_DIR}/jit_lib_align.das" +let ALIGN_OUT = "{OUTPUT_DIR}/jit_lib_align" +let BIND_SCRIPT = "{OUTPUT_DIR}/bindlib.das" +let BIND_OUT = "{OUTPUT_DIR}/bindlib" +let BINDER_SCRIPT = "{OUTPUT_DIR}/jit_lib_binder.das" +let SHARED_FIXTURE = "{get_das_root()}/tests-cpp/big/standalone_ctx/standalone_init_fixture.das" +let SHARED_OUT = "{OUTPUT_DIR}/standalone_init_fixture" +let SHARED_HOST = "{OUTPUT_DIR}/jit_lib_shared_host.das" +let GUEST_SCRIPT = "{OUTPUT_DIR}/jit_lib_guest.das" +let GUEST_OUT = "{OUTPUT_DIR}/jit_lib_guest" +let GUEST_HOST = "{OUTPUT_DIR}/jit_lib_guest_host.das" +let EMPTY_SCRIPT = "{OUTPUT_DIR}/jit_lib_empty.das" +let EMPTY_OUT = "{OUTPUT_DIR}/jit_lib_empty" + + +def private write_script(path, body : string) : bool { + var ok = false + mkdir_rec(OUTPUT_DIR) + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, body) + ok = true + } + } + return ok +} + + +def private write_probe_script(path : string) : bool { + var ok = false + mkdir_rec(OUTPUT_DIR) + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, "options gen2\n") + fprint(f, "struct Pair \{\n lo : int\n hi : int\n\}\n") + fprint(f, "var g_seen = 0\n") + fprint(f, "[export_c]\ndef widen(p : Pair) : Pair \{ g_seen++; return Pair(lo = p.lo - 1, hi = p.hi + 1) \}\n") + fprint(f, "[export_c(name = \"tally\")]\ndef count_seen() : int \{ return g_seen \}\n") + fprint(f, "[export_c]\ndef mapped(n : int) : int \{ var fx <- @@(x : int) => x * 3; return invoke(fx, n) \}\n") + fprint(f, "def helper_public(x : int) : int \{ return x + 1 \}\n") + ok = true + } + } + return ok +} + + +def private spawn(cmd : string; var lines : array) : int { + var rc : int + unsafe { + rc = popen_timeout("{cmd} 2>&1", 600.0) $(f) { + if (f == null) { + return + } + while (!feof(f)) { + let ln = strip(fgets(f)) + if (!empty(ln)) { + lines |> push("{ln}") + } + } + } + } + return rc +} + + +def private has(lines : array; needle : string) : bool { + for (ln in lines) { + if (find(ln, needle) >= 0) { + return true + } + } + return false +} + + +def private read_all(path : string) : string { + var text = "" + fopen(path, "rb") $(f) { + if (f != null) { + text = fread(f) + } + } + return text +} + + +def private shared_artifact(stem : string) : string { + let plat = get_platform_name() + if (plat == "windows") { + return "{stem}.dll" + } + return plat == "darwin" ? "{stem}.dylib" : "{stem}.so" +} + + +def private static_artifact(stem : string) : string { + return get_platform_name() == "windows" ? "{stem}.lib" : "{stem}.a" +} + + +[test] +def test_jit_lib_shared(t : T?) { + if (!jit_enabled()) { + t |> success(true, "-lib lives in the LLVM backend") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + t |> success(write_probe_script(LIB_SCRIPT), "cannot write the probe script") + var lines : array + let rc = spawn("\"{bin}\" -lib \"{LIB_SCRIPT}\" -output \"{LIB_OUT}\"", lines) + t |> success(rc == 0, "daslang -lib exits clean") + t |> success(lines |> has("LLVM LIB: C entry points generated"), "the C entry points are announced") + t |> success(fexist("{LIB_OUT}.h"), "the C header is written beside the library") + if (das_is_dll_build()) { + t |> success(fexist(shared_artifact(LIB_OUT)), "the shared library is written") + } + + let header = read_all("{LIB_OUT}.h") + t |> success(find(header, "jit_lib_probe_ctx * jit_lib_probe_create(void);") >= 0, "create is declared") + t |> success(find(header, "void jit_lib_probe_widen(jit_lib_probe_ctx * ctx, const jit_lib_probe_Pair * p, jit_lib_probe_Pair * out);") >= 0, + "a struct in and out crosses by pointer") + t |> success(find(header, "int32_t jit_lib_probe_tally(jit_lib_probe_ctx * ctx);") >= 0, "the [export_c] name is used") + t |> success(find(header, "int32_t jit_lib_probe_mapped(jit_lib_probe_ctx * ctx, int32_t n);") >= 0, + "a function using a lambda still exports") + t |> success(find(header, "helper_public") < 0, "a bare public function is not exported by default") +} + + +[test] +def test_jit_lib_static_and_export_all(t : T?) { + if (!jit_enabled()) { + t |> success(true, "-lib lives in the LLVM backend") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + t |> success(write_probe_script(LIB_SCRIPT), "cannot write the probe script") + var lines : array + let rc = spawn("\"{bin}\" -lib \"{LIB_SCRIPT}\" -output \"{LIB_OUT}_s\" -lib-export-all -- --jit-lib-static", lines) + t |> success(rc == 0, "a static export-all build exits clean") + t |> success(fexist(static_artifact("{LIB_OUT}_s")), "the archive is written") + let header = read_all("{LIB_OUT}_s.h") + t |> success(find(header, "int32_t jit_lib_probe_s_helper_public(") >= 0, "export-all reaches a bare public function") + t |> success(find(header, "jit_lib_probe_s_ctx") >= 0, "the prefix follows the output name, not the script name") + t |> success(find(header, "Link this archive") >= 0, "the header says what a static host links") +} + + +def private write_align_script(path : string) : bool { + var ok = false + mkdir_rec(OUTPUT_DIR) + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, "options gen2\n") + fprint(f, "struct Wide \{\n a : int8\n b : int64\n c : float\n\}\n") + fprint(f, "[export_c]\ndef make(n : int) : Wide \{ return Wide(a = int8(n), b = int64(n), c = float(n)) \}\n") + ok = true + } + } + return ok +} + + +[test] +def test_jit_lib_result_slot_is_aligned(t : T?) { + if (!jit_enabled()) { + t |> success(true, "-lib lives in the LLVM backend") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + t |> success(write_align_script(ALIGN_SCRIPT), "cannot write the alignment script") + var lines : array + spawn("\"{bin}\" -lib \"{ALIGN_SCRIPT}\" -output \"{ALIGN_OUT}\" -- --jit-dump", lines) + t |> success(lines |> has("%result = alloca [24 x i8], align 8"), + "the returned structure gets its own alloca at its own alignment") + t |> success(!(lines |> has("alloca \{ i32, [24 x i8] }")), + "the returned structure is not a field of the argument frame") +} + + +def private write_bind_library(path : string) : bool { + var body = "options gen2\n" + body += "struct Pair \{\n lo : int\n hi : int\n\}\n" + body += "var g_calls = 0\n" + body += "[export_c]\ndef bump() : int \{ g_calls++; return g_calls \}\n" + body += "[export_c]\ndef widen(p : Pair) : Pair \{ return Pair(lo = p.lo - 1, hi = p.hi + 1) \}\n" + body += "[export_c]\ndef greet(who : string) : string \{ return \"hola \{who}\" \}\n" + body += "[export_c(name = \"double_or_raise\")]\ndef boom(n : int) : int \{ if (n < 0) \{ panic(\"negative: \{n}\") \}; return n * 2 \}\n" + return write_script(path, body) +} + + +def private write_binder(path, lib : string) : bool { + var ok = false + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, "options gen2\nrequire dasbind\nrequire daslib/safe_addr\n") + fprint(f, "struct Pair \{\n lo : int\n hi : int\n\}\n") + let externs <- [ + ("bindlib_create", "bind_create() : void? \{ return null \}"), + ("bindlib_destroy", "bind_destroy(ctx : void?) : void \{\}"), + ("bindlib_last_error", "bind_last_error(ctx : void?) : string \{ return \"\" \}"), + ("bindlib_bump", "bind_bump(ctx : void?) : int \{ return 0 \}"), + ("bindlib_widen", "bind_widen(ctx, p, out : void?) : void \{\}"), + ("bindlib_greet", "bind_greet(ctx : void?; who : string) : string \{ return \"\" \}"), + ("bindlib_double_or_raise", "bind_double(ctx : void?; n : int) : int \{ return 0 \}") + ] + for (ext in externs) { + fprint(f, "[extern(cdecl, late, name=\"{ext._0}\", linux_library=\"{lib}\", macos_library=\"{lib}\", windows_library=\"{lib}\")]\ndef {ext._1}\n") + } + fprint(f, "[export]\ndef main() \{\n") + fprint(f, " var ctx = bind_create()\n") + fprint(f, " if (ctx == null) \{ print(\"create failed: \{bind_last_error(null)}\\n\"); return \}\n") + fprint(f, " var p = Pair(lo = 10, hi = 20)\n var out = Pair(lo = 0, hi = 0)\n") + fprint(f, " unsafe \{ bind_widen(ctx, reinterpret(safe_addr(p)), reinterpret(safe_addr(out))) \}\n") + fprint(f, " let raised = bind_double(ctx, -5)\n let err = bind_last_error(ctx)\n") + fprint(f, " print(\"BUMP=\{bind_bump(ctx)}\{bind_bump(ctx)} WIDEN=\{out.lo},\{out.hi} GREET=\{bind_greet(ctx, \"mundo\")} DOUBLE=\{bind_double(ctx, 21)} RAISED=\{raised} ERR=\{err} AFTER=\{bind_double(ctx, 3)}\\n\")\n") + fprint(f, " bind_destroy(ctx)\n print(\"BIND OK\\n\")\n\}\n") + ok = true + } + } + return ok +} + + +[test] +def test_jit_lib_binds_into_a_daslang_host(t : T?) { + if (!jit_enabled() || !das_is_dll_build()) { + to_log(LOG_WARNING, "jit_lib: SKIPPED - -lib needs the LLVM backend and a shared runtime\n") + t |> success(true, "-lib needs the LLVM backend and a shared runtime to link against") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + t |> success(write_bind_library(BIND_SCRIPT), "cannot write the bind library source") + var build_lines : array + let build_rc = spawn("\"{bin}\" -lib \"{BIND_SCRIPT}\" -output \"{BIND_OUT}\"", build_lines) + t |> success(build_rc == 0, "the bind library builds") + let artifact = shared_artifact(BIND_OUT) + t |> success(fexist(artifact), "the shared library exists at {artifact}") + t |> success(write_binder(BINDER_SCRIPT, artifact), "cannot write the binder script") + var lines : array + let rc = spawn("\"{bin}\" \"{BINDER_SCRIPT}\"", lines) + t |> success(rc == 0, "the binder script exits clean") + t |> success(lines |> has("BIND OK"), "the binder reached the end") + t |> success(lines |> has("BUMP=12"), "a global persists across two calls") + t |> success(lines |> has("WIDEN=9,21"), "a structure crosses in and out") + t |> success(lines |> has("GREET=hola mundo"), "a string crosses in and out") + t |> success(lines |> has("DOUBLE=42"), "a scalar crosses in and out") + t |> success(lines |> has("RAISED=0"), "a raising call answers zero") + t |> success(lines |> has("ERR=negative: -5"), "the panic text reaches the caller") + t |> success(lines |> has("AFTER=6"), "the instance still works after a panic") +} + + +def private write_guest_library(path : string) : bool { + var body = "options gen2\nrequire daslib/ast\nrequire UnitTest\n" + body += "[export_c]\ndef probe(n : int) : int \{ return n + 7 \}\n" + body += "[export_c]\ndef touch_ast : bool \{ return compiling_module() == null \}\n" + return write_script(path, body) +} + + +def private write_guest_host(path, lib : string) : bool { + var ok = false + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, "options gen2\nrequire dasbind\n") + let externs <- [ + ("jit_lib_guest_create", "guest_create() : void? \{ return null \}"), + ("jit_lib_guest_destroy", "guest_destroy(ctx : void?) : void \{\}"), + ("jit_lib_guest_last_error", "guest_last_error(ctx : void?) : string \{ return \"\" \}"), + ("jit_lib_guest_probe", "guest_probe(ctx : void?; n : int) : int \{ return 0 \}") + ] + for (ext in externs) { + fprint(f, "[extern(cdecl, late, name=\"{ext._0}\", linux_library=\"{lib}\", macos_library=\"{lib}\", windows_library=\"{lib}\")]\ndef {ext._1}\n") + } + fprint(f, "[export]\ndef main() \{\n") + fprint(f, " var ctx = guest_create()\n") + fprint(f, " if (ctx == null) \{ print(\"create failed: \{guest_last_error(null)}\\n\"); return \}\n") + fprint(f, " print(\"GUEST PROBE=\{guest_probe(ctx, 35)}\\n\")\n") + fprint(f, " guest_destroy(ctx)\n print(\"GUEST OK\\n\")\n\}\n") + ok = true + } + } + return ok +} + + +def private write_shared_host(path, lib : string) : bool { + var ok = false + fopen(path, "w") $(f) { + if (f != null) { + fprint(f, "options gen2\nrequire dasbind\nrequire daslib/safe_addr\n") + fprint(f, "struct Pair \{\n a : int\n b : int\n\}\n") + fprint(f, "struct Outer \{\n weight : int\n tag : int\n\}\n") + let externs <- [ + ("create", "sf_create() : void? \{ return null \}"), + ("destroy", "sf_destroy(ctx : void?) : void \{\}"), + ("last_error", "sf_last_error(ctx : void?) : string \{ return \"\" \}"), + ("get_first", "sf_get_first(ctx : void?) : int \{ return 0 \}"), + ("get_second", "sf_get_second(ctx : void?) : int \{ return 0 \}"), + ("get_init_fn_stamp", "sf_get_init_fn_stamp(ctx : void?) : int \{ return 0 \}"), + ("get_shared_total", "sf_get_shared_total(ctx : void?) : int \{ return 0 \}"), + ("pair_sum", "sf_pair_sum(ctx, p : void?) : int \{ return 0 \}"), + ("outer_weight", "sf_outer_weight(ctx, o : void?) : int \{ return 0 \}"), + ("head_value", "sf_head_value(ctx : void?) : int \{ return 0 \}"), + ("renamed_sum", "sf_renamed_sum(ctx : void?; a, b : int) : int \{ return 0 \}") + ] + for (ext in externs) { + fprint(f, "[extern(cdecl, late, name=\"standalone_init_fixture_{ext._0}\", linux_library=\"{lib}\", macos_library=\"{lib}\", windows_library=\"{lib}\")]\ndef {ext._1}\n") + } + fprint(f, "[export]\ndef main() \{\n") + fprint(f, " var ctx = sf_create()\n") + fprint(f, " if (ctx == null) \{ print(\"create failed: \{sf_last_error(null)}\\n\"); return \}\n") + fprint(f, " var p = Pair(a = 40, b = 2)\n var o = Outer(weight = 9, tag = 5)\n") + fprint(f, " unsafe \{\n") + fprint(f, " print(\"SF first=\{sf_get_first(ctx)} second=\{sf_get_second(ctx)} stamp=\{sf_get_init_fn_stamp(ctx)} shared=\{sf_get_shared_total(ctx)}\\n\")\n") + fprint(f, " print(\"SF pair_sum=\{sf_pair_sum(ctx, reinterpret(safe_addr(p)))} outer=\{sf_outer_weight(ctx, reinterpret(safe_addr(o)))} head=\{sf_head_value(ctx)} renamed=\{sf_renamed_sum(ctx, 20, 22)}\\n\")\n") + fprint(f, " \}\n") + fprint(f, " sf_destroy(ctx)\n print(\"SF OK\\n\")\n\}\n") + ok = true + } + } + return ok +} + + +[test] +def test_jit_lib_runs_the_standalone_fixture(t : T?) { + if (!jit_enabled() || !das_is_dll_build()) { + to_log(LOG_WARNING, "jit_lib: SKIPPED - -lib needs the LLVM backend and a shared runtime\n") + t |> success(true, "-lib needs the LLVM backend and a shared runtime to link against") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + var build_lines : array + let build_rc = spawn("\"{bin}\" -lib \"{SHARED_FIXTURE}\" -output \"{SHARED_OUT}\" -- --jit-lib-export-marked", + build_lines) + t |> success(build_rc == 0, "the AOT standalone fixture also builds as a JIT library") + let artifact = shared_artifact(SHARED_OUT) + t |> success(fexist(artifact), "the shared library exists at {artifact}") + t |> success(write_shared_host(SHARED_HOST, artifact), "cannot write the shared host") + var lines : array + let rc = spawn("\"{bin}\" \"{SHARED_HOST}\"", lines) + t |> success(rc == 0, "the host exits clean") + t |> success(lines |> has("SF OK"), "the host reached the end") + t |> success(lines |> has("SF first=31 second=2 stamp=3 shared=6"), + "globals, their order and the [init] functions answer what the AOT twin asserts") + t |> success(lines |> has("SF pair_sum=42 outer=14 head=41 renamed=42"), + "a struct in, a nested struct, a safe-navigation read and the renamed symbol all agree with the AOT twin") +} + + +[test] +def test_jit_lib_is_a_guest_of_a_populated_runtime(t : T?) { + if (!jit_enabled() || !das_is_dll_build()) { + to_log(LOG_WARNING, "jit_lib: SKIPPED - -lib needs the LLVM backend and a shared runtime\n") + t |> success(true, "-lib needs the LLVM backend and a shared runtime to link against") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + t |> success(write_guest_library(GUEST_SCRIPT), "cannot write the guest library source") + var build_lines : array + let build_rc = spawn("\"{bin}\" -lib \"{GUEST_SCRIPT}\" -output \"{GUEST_OUT}\"", build_lines) + t |> success(build_rc == 0, "the guest library builds") + let artifact = shared_artifact(GUEST_OUT) + t |> success(fexist(artifact), "the shared library exists at {artifact}") + t |> success(write_guest_host(GUEST_HOST, artifact), "cannot write the guest host script") + var lines : array + let rc = spawn("\"{bin}\" \"{GUEST_HOST}\"", lines) + t |> success(rc == 0, "the host survives a library that reaches its own modules") + t |> success(lines |> has("GUEST OK"), "the guest library reached the end") + t |> success(lines |> has("GUEST PROBE=42"), "and answered a call") + for (line in lines) { + t |> success(find(line, "already created") < 0, "no module is created a second time: {line}") + } +} + + +[test] +def test_jit_lib_example_binds_three_libraries(t : T?) { + if (!jit_enabled() || !das_is_dll_build()) { + to_log(LOG_WARNING, "jit_lib: SKIPPED - -lib needs the LLVM backend and a shared runtime\n") + t |> success(true, "-lib needs the LLVM backend and a shared runtime to link against") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + var lines : array + let rc = spawn("cd \"{get_das_root()}\" && \"{bin}\" examples/c_api_library/main.das", lines) + t |> success(rc == 0, "the c_api_library example exits clean") + t |> success(lines |> has("[export_c] only"), "form 1 builds") + t |> success(lines |> has("-lib-export-all"), "form 2 builds") + t |> success(lines |> has("--jit-lib-export-marked"), "form 3 builds") + t |> success(lines |> has("shapes.dot((3,4),(1,2)) = 11"), "a struct crosses through form 1") + t |> success(lines |> has("units.celsius_to_fahrenheit(100) = 212"), "a scalar crosses through form 2") + t |> success(lines |> has("greetings.greet(\"mundo\") = hola mundo"), "a string crosses through form 3") + t |> success(lines |> has("three daslang libraries, one process - ok"), "all three coexist") +} + + +[test] +def test_jit_lib_refuses_unrepresentable(t : T?) { + if (!jit_enabled()) { + t |> success(true, "-lib lives in the LLVM backend") + return + } + let args <- get_command_line_arguments() + let bin = args[0] + let body = "options gen2\n[export_c]\ndef takes_array(a : array) : int \{ return length(a) \}\n" + t |> success(write_script(BAD_SCRIPT, body), "cannot write the refusal script") + var err : string + remove(shared_artifact(BAD_OUT), err) + var lines : array + let rc = spawn("\"{bin}\" -lib \"{BAD_SCRIPT}\" -output \"{BAD_OUT}\"", lines) + t |> success(lines |> has("has no C representation"), "the refusal names the type C cannot spell") + t |> success(!fexist(shared_artifact(BAD_OUT)), "no library is written when an export is refused") + t |> success(rc != 0, "a refused export fails the build") + + let empty_body = "options gen2\ndef nothing() \{\}\n" + t |> success(write_script(EMPTY_SCRIPT, empty_body), "cannot write the empty script") + var empty_lines : array + let empty_rc = spawn("\"{bin}\" -lib \"{EMPTY_SCRIPT}\" -output \"{EMPTY_OUT}\"", empty_lines) + t |> success(empty_lines |> has("nothing to export"), "a program with no exports says so") + t |> success(empty_rc != 0, "a program with no exports fails the build") +} diff --git a/utils/daslang/main.cpp b/utils/daslang/main.cpp index d68bf4d440..b74f1009ea 100644 --- a/utils/daslang/main.cpp +++ b/utils/daslang/main.cpp @@ -48,11 +48,14 @@ enum class JitMode { Direct, Dll, Executable, + Library, }; static JitMode jitEnabled = JitMode::None; // Disabled by default. static bool jitNoCache = false; // -jit-no-cache: bypass DLL-cache path, run in-memory. static bool jitStack = false; // -jit-stack: retain every generated call in the logical das stack. static string jitOutPath = ""; // Empty, JIT module will choose default. +static bool libExportAll = false; +static bool libNeedsOutput = false; static string serFile = ""; // -ser : write the AST module cache (env serializer rail) after compile static string deserFile = ""; // -deser : read the AST module cache during compile instead of parsing static string moduleCacheFile = ""; // -module-cache : both - read when present, refresh when the compile diverged @@ -459,6 +462,9 @@ int compile_and_run ( const string & fn, const string & mainFnName, bool outputP policies.jit_enabled = true; switch (jitEnabled) { case JitMode::Executable: policies.jit_exe_mode = true; break; + case JitMode::Library: + policies.export_public_functions = libExportAll; + break; case JitMode::Dll: policies.jit_dll_mode = true; break; case JitMode::Direct: break; default: break; @@ -567,6 +573,9 @@ int compile_and_run ( const string & fn, const string & mainFnName, bool outputP if ( compileOnly ) return 0; + if ( jitEnabled==JitMode::Library ) { + program->options.push_back(AnnotationArgument("jit_lib", true)); + } auto simulate0 = ref_time_ticks(); auto pctx = SimulateWithErrReport(program, tout); startupSimulateUsec += get_time_usec(simulate0); @@ -705,6 +714,10 @@ void print_help() { << " Useful when the cached .jitted_scripts/ DLL is stale or unwanted.\n" << " -jit-stack with -jit: retain every generated call in the logical daslang stack.\n" << " -exe JIT compile to standalone executable (implies -dry-run)\n" + << " -lib JIT compile to a C-ABI native library: .so/.dylib/.dll plus .h\n" + << " (add -- --jit-lib-static for a .a/.lib archive instead; implies -dry-run)\n" + << " -lib-export-all with -lib: export every public entry-module function whose signature has a C\n" + << " representation, instead of only the [export_c] ones\n" << " -output set JIT output path\n" << " --list-shared-modules with -exe: write JSON describing the program's shared modules and daspkg-package .das module sources to \n" << " --force-shared-module with -exe: force-include a shared module by daslang or package name (repeatable)\n" @@ -911,6 +924,12 @@ int MAIN_FUNC_NAME ( int argc, char * argv[] ) { } else if ( cmd=="exe") { jitEnabled = JitMode::Executable; dryRun = true; + } else if ( cmd=="lib") { + jitEnabled = JitMode::Library; + dryRun = true; + libNeedsOutput = true; + } else if ( cmd=="lib-export-all") { + libExportAll = true; } else if ( cmd=="ser" ) { if ( i+1 >= argc ) { printf("-ser requires path argument\n"); @@ -1095,6 +1114,11 @@ int MAIN_FUNC_NAME ( int argc, char * argv[] ) { printf("-no-module-cache disables the cache; do not combine it with -ser/-deser\n"); return -1; } + if ( libNeedsOutput && jitOutPath.empty() ) { + printf("-lib needs -output : a host includes the generated header by name, and the\n" + "default JIT cache path is hash-named and swept\n"); + return -1; + } startupPreScanUsec = get_time_usec(startupMain0); auto builtin0 = ref_time_ticks(); // register modules