common: add json.h abstraction

[ngxson]

Overview

Add a light-weight common/json.h to avoid re-compiling json.hpp in multiple places. Also allow swapping another pimpl in the future if needed

Goals:

• pimpl nlohmann::json
• least changes to downstream code (almost drop-in replacement)

Changes to downstream code:

• Rename nlohmann::json to common_json
• obj.push_back({key, val}) is changed to obj[key] = val for clarify; push_back can be confused with std::vector in some places
• Some places having weird std::something<...> = obj["abc"], change to explicit = obj.get<std::something<...>>() to avoid excessive template instances

Results

End-to-end (compile common + server, exclude libllama/libmtmd, -j12):

• wall: 18.9s -> 15.5s (-17.9%)
• CPU (user): 98.3s -> 79.8s (-18.8%)

Single-TU dev loop (touch tools/server/server-context.cpp, rebuild + link):

• wall: 5.3s -> 4.0s (-23.9%)

Binary size:

• common/json.cpp.o adds 2.83 MB exactly once
• libserver-context.a: 41.5 MB -> 24.9 MB (-40.0%)
• linked libllama-server-impl.dylib: 6.86 MB -> 6.33 MB (-7.7%), __TEXT 4.95 MB -> 4.69 MB (-5.3%)

TODO:

• add json.cpp/.h
• migrate common/*
• migrate the rest of the code base
• test correctness --> did some smoke tests + fuzzing
• see how much we gain on compile time
• remove the deps jinja --> common (the json-to-internal code path) maybe a follow-up

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes, large part of code is AI-generated

[ServeurpersoCom]

LGTM. Read it on a checkout: the only silent change is a braced pair in value position, array before and object now, and they are all converted to json::array() here, tests included. Nice side effect, operator[] const throws where master aborted on the assert.

Two nits, non blocking: tests/gguf-model-data.cpp still includes nlohmann/json.hpp directly, and the object iterator re-walks with std::next(begin(), idx) on every dereference.

Every copy now sits behind one translation unit, which is also where a depth bound would live. Is that the intent, so #27434 can be closed without waiting for nlohmann/json#5389?

[ggerganov]

Nice work! I see 10% build time reduction on my M2 Ultra with 8 threads for the total project and about ~20% reduction for user-space code (common, tool, examples, tests).

Btw, I think there is a low-hanging fruit for reducing compilation time by amalgamating the models/*.cpp files as part of the build process.

[ggerganov]

nit: I think this declaration can be safely removed

[ngxson]

@ServeurpersoCom thanks for testing, I added your points as code comments

Is that the intent, so #27434 can be closed without waiting for nlohmann/json#5389?

maintainers are actively working on the upstream PR, so that this point we can be sure that it will be fixed upstream. let's close the PR #27434 to avoid duplicated works

Btw, I think there is a low-hanging fruit for reducing compilation time by amalgamating the models/*.cpp files as part of the build process.

@ggerganov TIL that cmakelists has a feature called UNITY_BUILD, may worth giving it a try?

[ngxson]

rebased to make sure I'm not missing any code paths that were not migrated, will merge this PR once is CI is green

[eyalezer]

@ngxson: is it just me or this merge broke my build on windows using clang? links issues (undefined symbols)

[ServeurpersoCom]

Interesting; I haven't tested Windows in advance at all, but it's inevitable: when I rebase my working branch, the little dedicated Windows machine (exposed at https://www.serveurperso.com/ia/windows/ !) is going to run into trouble, I check this now!

[ServeurpersoCom]

It's simply the build folder that needs to be deleted on my side. Windows OK

[eyalezer]

@ServeurpersoCom - nope... still fails at my end even after using a clean build folder...
i have to be honest i haven't really tried to do any digging yet... i'll keep you posted.

BTW i do wonder now what was the potential fixed you proposed earlier?

[ServeurpersoCom]

@ServeurpersoCom - nope... still fails at my end even after using a clean build folder... i have to be honest i haven't really tried to do any digging yet... i'll keep you posted.

BTW i do wonder now what was the potential fixed you proposed earlier?

Are you building master itself, or a branch of your own on top of it? If you carry local changes under tools/server or common, that is where I would look first: every JSON type and call in that code has to move to the new common_json. The header include becomes common/json.h instead of nlohmann/json.hpp, any nlohmann::ordered_json in a signature or a using alias becomes common_json, and a braced array literal in value position no longer builds, so {a, b, c} has to be written as json::array({a, b, c}).

[eyalezer]

Are you building master itself, or a branch of your own on top of it? If you carry local changes under tools/server or common, that is where I would look first: every JSON type and call in that code has to move to the new common_json. The header include becomes common/json.h instead of nlohmann/json.hpp, any nlohmann::ordered_json in a signature or a using alias becomes common_json, and a braced array literal in value position no longer builds, so {a, b, c} has to be written as json::array({a, b, c}).

indeed on a fork.... which i keep on rebased from master here.
thanks for the brief mate... i would go look there first thing.
don't sweat it, it might be something in my local fork i missed 😅

i'll keep you posted either way... i just started looking into it and thanks for the quick response ❤️

[eyalezer]

Found it — it was the -flto=full in my build flags. Dropping LTO made it link clean.

For anyone hitting the same thing on Windows + clang + LTO: explicit template instantiations in a static/shared lib can come out as weak (W) symbols instead of exported (T), so the linker reports them as undefined even though they're in the object.

@ServeurpersoCom: sorry if made you worry man 😅 looks like it a it's a clang-on-Windows LTO bug.

[maddes8cht]

@ngxson @eyalezer

I can confirm this regression on Windows when building with clang-cl and LTO enabled. But in my opinion, simply dropping LTO is not a viable long-term solution for those of us relying on LTO to minimize CPU overhead during graph execution and prompt processing.

The root cause appears to be tied to the recent Autoparser refactoring, which introduced new JSON-template instantiations inside static libraries. Under Windows (COFF format), clang-cl + LTO incorrectly marks explicit template instantiations inside static libraries as weak (W) symbols instead of exported text (T) symbols. When the linker builds the final executable with LTO, it assumes these symbols will be provided elsewhere and optimizes them away, resulting in undefined reference errors.

Instead of forcing users to abandon Clang or LTO, could we fix this at the CMake or source level?

1. CMake Object Libraries: Changing the affected static libraries to OBJECT libraries (e.g., add_library(common OBJECT ...)) bypasses the static-lib linker step entirely, feeding the .obj files directly to the final executable link. This completely avoids the Clang/Windows weak-symbol LTO bug.
2. Export Attributes: Wrapping the explicit template instantiations in __declspec(dllexport) (or the project's equivalent API macros) forces Clang to emit them as strong exported symbols.

I'd love to keep using clang-cl with LTO for the CPU overhead reduction.

[ngxson]

could you try #27575 ?