Backend protocol for LLM inference in Common Lisp: one generate surface over local llama.cpp, OpenAI-compatible HTTP APIs, multi-provider cloud APIs (via cl-llm-provider), and test mocks. Single completion, batch (N different prompts), and parallel fan-out (one prompt → N diverse completions with shared-prefix optimization).
Active Development / Beta. API is unstable and will change without notice.
| System | Description | Dependencies |
|---|---|---|
cl-llm-backend | Protocol, conditions, mock backend | none |
cl-llm-backend/llama | llama.cpp backend | cl-llama-cpp, cl-llama-cpp-extras/json-schema |
cl-llm-backend/openai | OpenAI-compatible HTTP backend | dexador, yason, bordeaux-threads |
cl-llm-backend/cl-llm-provider | Multi-provider adapter (OpenAI, Anthropic, Ollama) | cl-llm-provider, yason |
cl-llm-backend/tests | Core tests (no model needed) | rove |
cl-llm-backend/llama/tests | Backend tests (integration model-gated) | rove |
cl-llm-backend/openai/tests | Backend tests (integration server-gated) | rove |
cl-llm-backend/cl-llm-provider/tests | Adapter tests (modelless) | rove |
Not yet on Quicklisp or Ultralisp. Clone into a directory ASDF/Quicklisp searches, e.g. Roswell’s local-projects:
git clone https://github.com/licjon/cl-llm-backend ~/.roswell/local-projects/cl-llm-backendThen (ql:quickload "cl-llm-backend") (or /openai, /llama,
/cl-llm-provider) loads it and its dependencies. cl-llm-backend/openai
only needs dexador, yason, and bordeaux-threads, all on Quicklisp.
cl-llm-backend/llama additionally needs cl-llama-cpp and
cl-llama-cpp-extras, cloned the same way.
cl-llm-backend/cl-llm-provider needs cl-llm-provider:
git clone https://github.com/quasi/cl-llm-provider ~/.roswell/local-projects/cl-llm-provider(ql:quickload "cl-llm-backend/llama")
(llm/llama:with-llama-backend (backend "/path/to/model.gguf" :n-ctx 4096)
;; string in, string out
(llm:backend-generate backend "The capital of France is"
:max-tokens 32 :temperature 0.0)
;; messages + streaming
(llm:backend-generate backend
'((:role "system" :content "Be terse.")
(:role "user" :content "Explain CLOS."))
:max-tokens 128
:token-callback (lambda (tok)
(write-string tok)
(force-output)
t)))Against any OpenAI-compatible endpoint (OpenAI, llama-server, vLLM, Ollama):
(ql:quickload "cl-llm-backend/openai")
(let ((backend (llm/openai:make-openai-backend
:base-url "http://localhost:8080/v1" ; default: https://api.openai.com/v1
:model "anything" ; :api-key defaults to $OPENAI_API_KEY
:structured-output :json-schema))) ; or :json-object, or NIL
(llm:backend-generate backend "The capital of France is"
:max-tokens 32 :temperature 0.0))HTTP-level failures signal llm/openai:openai-http-error (status, body,
and the API’s error message). Extra sampler keys pass through as
snake_case body parameters (:top-p → top_p).
Via cl-llm-provider (Anthropic, OpenAI, Gemini, Ollama, OpenRouter — structured output via tool calling, observability hooks):
(ql:quickload "cl-llm-backend/cl-llm-provider")
;; From a provider type keyword
(let ((backend (llm/provider:make-cl-llm-provider-backend
:provider-type :anthropic
:model "claude-sonnet-4-20250514")))
(llm:backend-generate backend "The capital of France is"
:max-tokens 32 :temperature 0.0))
;; Or wrap an existing cl-llm-provider instance
(let* ((provider (cl-llm-provider:make-provider :openai :model "gpt-4o"))
(backend (llm/provider:make-cl-llm-provider-backend :provider provider)))
(llm:backend-generate backend
'((:role "user" :content "Extract the name."))
:output-schema (yason:parse
"{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"}},\"required\":[\"name\"]}")))
;; With observability
(let* ((hooks (cl-llm-provider:make-logging-hooks))
(backend (llm/provider:make-cl-llm-provider-backend
:provider-type :openai :model "gpt-4o" :hooks hooks)))
(llm:backend-generate backend "Hello" :max-tokens 32)):output-schema maps to cl-llm-provider’s tool-calling mechanism: the
adapter creates a tool from the schema, forces the model to use it, and
extracts the structured arguments as JSON text. This works across all
providers that support tool calling.
backend-generate returns (values text response-info); the info struct
carries stop reason, token counts, and model name where the backend can
report them. Constrained output: pass :output-schema (a JSON Schema hash
table) — the llama backend compiles it to a GBNF grammar.
Backends that support it (llama, mock) can snapshot a conversation point
and branch from it. A handle is captured once and resumed many times
until released. :resume-from is a pure optimization: you still pass the
full conversation, and the backend reuses the longest shared token prefix
from the snapshot, decoding only what is new.
(llm/llama:with-llama-backend (backend "/path/to/model.gguf" :n-ctx 4096)
(let ((msgs '((:role "user" :content "Suggest a name for a cat cafe."))))
(llm:backend-generate backend msgs :max-tokens 64)
(let ((point (llm:backend-capture-state backend)))
(unwind-protect
;; two candidate continuations from one snapshot
(list (llm:backend-generate backend msgs :resume-from point
:max-tokens 64 :seed 1)
(llm:backend-generate backend msgs :resume-from point
:max-tokens 64 :seed 2))
(llm:backend-release-state backend point)))))A resumed generation shares conversation state, not a paused
generation: the KV cache is restored, while sampler state (mirostat,
penalties, grammar) is fresh per call, as in any other generate. Reuse
depends on the rendered prompt matching the captured tokens — a chat
template upgrade or system-prompt change can drop reuse to zero (see
:cache-prefix-tokens in the response info) but never affects
correctness. Generation and capture on one backend must be externally
serialized.
backend-generate-parallel generates N completions from a single prompt.
The llama backend renders the prompt once and decodes all N sequences
concurrently with shared KV cache; the OpenAI backend fires N concurrent
HTTP requests. Every backend supports the call — the default method
falls back to N serial backend-generate calls.
(llm/llama:with-llama-backend (backend "/path/to/model.gguf"
:n-ctx 4096 :n-seq-max 8)
(multiple-value-bind (texts infos)
(llm:backend-generate-parallel
backend
'((:role "user" :content "Name a mythical creature."))
4 ; four completions
:max-tokens 64 :seed 42) ; seeds 42, 43, 44, 45
(dolist (text texts) (format t "~A~%---~%" text))))Seed semantics match backend-generate-batch: an integer :seed gives
completion i the seed seed+i (deterministic, distinct); nil gives
independent nondeterministic per-completion seeds.
backend-supports-parallel-p tells callers whether N > 1 buys
wall-clock improvement over N serial calls.
Specialize llm:backend-generate on your class; input arrives as a
normalized message list (plists with string :role and :content).
Declare capabilities by specializing
backend-supports-structured-output-p / backend-supports-streaming-p
(defaults are nil, and the protocol signals backend-capability-error if
a caller passes :output-schema / :token-callback to a backend without
the capability). Optionally provide backend-model-info (plist with
:name and :context-length).
For batch/parallel: optionally specialize backend-generate-batch
and/or backend-generate-parallel; both have working serial defaults.
Declare backend-supports-batch-p / backend-supports-parallel-p
when your override is truly concurrent.
For state capture: specialize backend-capture-state /
backend-resume-state / backend-release-state and declare
backend-supports-state-capture-p (or inherit the forkable-backend
mixin).
Generic protocol:
| Symbol | Signature / kind | Notes |
|---|---|---|
backend-generate | (backend input &rest keys &key output-schema token-callback temperature max-tokens seed) | Returns (values text response-info). The :around method normalizes input and checks capabilities before dispatch. |
backend-generate-batch | (backend inputs &rest keys &key temperature max-tokens seed) | inputs is a list of message-lists/strings. Returns (values texts infos). Default maps backend-generate serially. |
backend-generate-parallel | (backend input n &rest keys &key temperature max-tokens seed) | One prompt, N completions. Returns (values texts infos). Default calls backend-generate N times with seed+i. See backend-supports-parallel-p. |
normalize-input | (input) | String → single user message; message list validated as-is. Signals invalid-input-error on a malformed message, empty list, or nil. |
Capabilities & introspection (all default to nil on t):
| Symbol | Signature | Notes |
|---|---|---|
backend-supports-structured-output-p | (backend) | True when the backend honors :output-schema. |
backend-supports-streaming-p | (backend) | True when the backend honors :token-callback. |
backend-supports-batch-p | (backend) | True when backend-generate-batch decodes concurrently. |
backend-supports-parallel-p | (backend) | True when backend-generate-parallel decodes N sequences concurrently. |
backend-supports-state-capture-p | (backend) | True for forkable-backend subclasses and backends implementing the protocol. |
backend-model-info | (backend) | Plist, or nil. Defined keys: :name (string), :context-length (integer). |
response-info (struct, any slot may be nil):
| Symbol | Notes |
|---|---|
make-response-info, response-info-p | Constructor and type predicate. |
response-info-stop-reason | Keyword, backend-defined (e.g. :eog, :length, :stop, :callback, :aborted). |
response-info-prompt-tokens | Integer or nil. |
response-info-completion-tokens | Integer or nil. |
response-info-model-name | String or nil. |
response-info-raw | Backend-specific escape hatch (e.g. the parsed JSON body); only populated where noted. |
State capture
| Symbol | Notes |
|---|---|
forkable-backend | Mixin class marking a backend as capable of state capture. |
backend-capture-state | (backend) → opaque handle. Errors with backend-capability-error unless the backend is a forkable-backend and implements it. |
backend-resume-state | (backend handle). A handle may be resumed multiple times until released. |
backend-release-state | (backend handle). Mandatory even where a no-op. |
Conditions:
| Symbol | Notes |
|---|---|
llm-backend-error | Base condition for all errors this library signals. |
llm-backend-error-message | Reader for the human-readable message (may be nil). |
backend-capability-error | Signaled when a caller passes :output-schema=/:token-callback= to a backend lacking the capability, or invokes state-capture op. Subclass of llm-backend-error. |
capability-error-backend | Reader: the offending backend instance. |
capability-error-operation | Reader: the operation attempted (e.g. :output-schema, :token-callback, or a state-capture generic-function symbol). |
invalid-input-error | Signaled by normalize-input on malformed input. Subclass of llm-backend-error. |
invalid-input-error-input | Reader: the offending input (or message). |
invalid-input-error-reason | Reader: a string explaining why it was rejected. |
Mock backend, for use in your own tests:
| Symbol | Notes |
|---|---|
make-mock-backend | (&key responses). responses is a list of strings (consumed in call order) or a function of the normalized message list. Signals llm-backend-error if neither. Supports structured output and streaming. |
mock-backend-p | Type predicate. |
mock-backend-calls | Number of backend-generate calls so far. |
mock-backend-requests | List of (:messages ... :keys ...) plists, newest first. |
mock-backend-prompts | Content of the last message of each recorded request, newest first. |
When a list of responses runs out, further calls signal llm-backend-error.
When :token-callback is supplied, the scripted text streams in
whitespace-delimited chunks; returning nil from the callback stops early
with :stop-reason :callback.
Backend over a borrowed cl-llama-cpp model + context.
| Symbol | Signature / kind | Notes |
|---|---|---|
with-llama-backend | (var model-path &key (n-gpu-layers 99) (n-ctx 8192)) &body body | Macro. Loads the GGUF model at model-path, owns the model and context for body’s extent, and binds var to a llama-backend. |
make-llama-backend | (&key model context) | Both keys required; wraps an already-loaded cl-llama-cpp model/context pair. |
llama-backend-p | (object) | Type predicate. |
llama-backend-model | (backend) | Accessor for the underlying cl-llama-cpp model. |
llama-backend-context | (backend) | Accessor for the underlying cl-llama-cpp context. |
Supports structured output and streaming. Renders messages via the model’s
chat template, falling back to joining message contents with blank lines
for template-less models. :output-schema compiles to a GBNF grammar via
cl-llama-cpp-extras/json-schema. Keys other than the protocol core
(:output-schema :token-callback :temperature :max-tokens :seed
:grammar-first) pass through to cl-llama-cpp:generate verbatim
(e.g. :top-k, :min-p, :repeat-penalty, :mirostat).
When :output-schema is active, passing :grammar-first t enables
grammar-first resampling: the grammar sampler runs on the full logit
vocabulary before top-k/top-p/min-p/temp narrow the candidates, then
the quality samplers pick from the grammar-valid set. Without it, the
grammar runs after quality sampling has already narrowed to ~30-50
tokens, which can degrade sampling quality when few of those survive the
grammar filter.
(llm:backend-generate backend
'((:role "user" :content "Extract the name: Alice went home."))
:output-schema schema
:grammar-first t
:max-tokens 64 :temperature 0.0):grammar-first is opt-in (not defaulted to t) because grammar-first
resampling has an unconstrained fallback: when the grammar rejects the
entire vocabulary, sampling falls back to unconstrained generation
rather than failing. Callers that depend on strict grammar conformance
(e.g. JSON extraction pipelines) should evaluate this tradeoff before
enabling it. Without :output-schema, :grammar-first is silently
ignored.
Note: Prefer cl-llm-backend/cl-llm-provider for new work — it
delegates to cl-llm-provider, which supports multiple APIs behind a
uniform interface. This standalone OpenAI backend is kept for
compatibility.
Backend over any OpenAI-compatible /v1/chat/completions endpoint.
| Symbol | Signature / kind | Notes |
|---|---|---|
make-openai-backend | (&key (base-url “https://api.openai.com/v1”) model (api-key (uiop:getenv “OPENAI_API_KEY”)) (structured-output :json-schema) extra-headers (timeout 120)) | model is required. structured-output must be :json-schema, :json-object, or nil. Signals llm:llm-backend-error otherwise. |
openai-backend-p | (object) | Type predicate. |
openai-backend-base-url | (backend) | Right-trimmed of trailing /. |
openai-backend-model | (backend) | |
openai-backend-api-key | (backend) | May be nil (no Authorization header sent). |
openai-backend-structured-output | (backend) | :json-schema, :json-object, or nil. |
openai-backend-extra-headers | (backend) | Alist of extra request headers, appended after Content-Type=/=Authorization. |
openai-backend-timeout | (backend) | Read timeout in seconds, passed to dexador. |
openai-http-error | Condition | Signaled on any non-2xx HTTP response. Subclass of llm:llm-backend-error; llm:llm-backend-error-message carries the API’s error.message when the body is JSON. |
openai-http-error-status | (condition) | Reader: HTTP status code. |
openai-http-error-body | (condition) | Reader: raw response body string. |
Supports streaming unconditionally; supports structured output whenever
:structured-output is non-nil. With :json-schema, the schema is sent as
a strict response_format; with :json-object, schema guidance is appended
as text to the last message instead. Keys other than the protocol core
pass through as snake_case body parameters (:top-p → top_p), except
model, messages, stream, and response_format, which the backend
always sets itself.
Adapter wrapping a cl-llm-provider provider instance. Supports any
provider cl-llm-provider supports (OpenAI, Anthropic, Gemini, Ollama,
OpenRouter, OpenAI-compatible). Structured output uses tool calling;
streaming uses complete-stream callbacks.
| Symbol | Signature / kind | Notes |
|---|---|---|
make-cl-llm-provider-backend | (&key provider provider-type model api-key base-url hooks) | Supply :provider (instance) or :provider-type (keyword) + config. Signals llm:llm-backend-error if neither given. |
cl-llm-provider-backend-p | (object) | Type predicate. |
cl-llm-provider-backend-provider | (backend) | The wrapped cl-llm-provider:llm-provider instance. |
cl-llm-provider-backend-model | (backend) | Model string passed to every complete call. |
cl-llm-provider-backend-hooks | (backend) | cl-llm-provider:hooks struct for observability, or nil. |
Capabilities are derived from the underlying provider: structured output
requires :tools, streaming requires :streaming (per
cl-llm-provider:provider-capabilities). State capture is not supported.
When :output-schema is given, the adapter creates a tool definition
named "extract" from the schema, passes it to cl-llm-provider:complete
with forced tool choice, and returns the tool-call arguments as a JSON
string. If the model returns text instead of a tool call, the text is
returned as-is (fallback).
Provider errors (cl-llm-provider:provider-api-error, etc.) propagate
unwrapped — they are not subclasses of llm-backend-error, so callers
should handle both hierarchies or catch error.
ros -e '(ql:quickload "cl-llm-backend/tests" :silent t)' \
-e '(asdf:test-system "cl-llm-backend")'
# llama backend; integration tests need a model:
LLAMA_TEST_MODEL=/path/to/model.gguf \
ros -e '(ql:quickload "cl-llm-backend/llama/tests" :silent t)' \
-e '(asdf:test-system "cl-llm-backend/llama")'
# openai backend; integration tests need any OpenAI-compatible server:
OPENAI_TEST_BASE_URL=http://localhost:8080/v1 OPENAI_TEST_MODEL=local \
ros -e '(ql:quickload "cl-llm-backend/openai/tests" :silent t)' \
-e '(asdf:test-system "cl-llm-backend/openai")'
# cl-llm-provider adapter (modelless — no API keys needed):
ros -e '(ql:quickload "cl-llm-backend/cl-llm-provider/tests" :silent t)' \
-e '(asdf:test-system "cl-llm-backend/cl-llm-provider")'Use llm:make-mock-backend in your own tests: scripted string responses
or a function of the message list; records calls, messages, and keys
(mock-backend-calls, mock-backend-requests, mock-backend-prompts).
MIT