The complete reference for every script-facing builtin and configuration
accessor exposed by the llm module. For an overview, installation, and a
quickstart, see the README.
The module exposes three top-level builtins via load("llm", …) — message,
chat, and draw — plus a set of configuration accessors (get_<key> /
set_<key>) generated from the module's options. Module options serve as
defaults: they are used when the corresponding argument is not passed to chat
/ draw.
Creates a message object for the chat function. It is data only — no API call
is made. chat's messages=[...] consumes exactly these objects.
Parameters:
role(string): The role of the message (default:"user")text(string): The text content of the messageimage(string/bytes): Raw image data to include in the messageimage_file(string): Path to an image file to include in the messageimage_url(string): URL of an image to include in the message
Returns: A dictionary representing the message, containing only the
non-empty subset of role / text / image / image_file / image_url.
Example:
load("llm", "message")
msg = message(role="user", text="Hello, who are you?")chat(text?, image?, image_file?, image_url?, messages?, model?, n?, max_tokens?, max_completion_tokens?, temperature?, top_p?, frequency_penalty?, presence_penalty?, stop?, response_format?, reasoning_effort?, retry?, full_response?, allow_error?, stream?, stream_callback?, kwargs?)
Sends a chat completion request to the OpenAI (or OpenAI-compatible) API. The
top-level text / image / image_file / image_url arguments are assembled
into an implicit user message that is prepended to any messages list.
Parameters:
text(string): Text content for a user messageimage(string/bytes): Raw image data to include in the user messageimage_file(string): Path to an image file to include in the user messageimage_url(string): URL of an image to include in the user messagemessages(list): List of message dictionaries (from themessagefunction)model(string): Model to use (defaults to theopenai_gpt_modelconfig option)n(int): Number of completions to generate (default:1)max_tokens(int): Maximum number of tokens to generate;0(the default) leaves it unset so the API applies its own default. Deprecated for o1-series models — usemax_completion_tokensinsteadmax_completion_tokens(int): Upper bound for generated completion tokens (for o1/o3/o4-series models);0(the default) leaves it unsettemperature(float): Sampling temperature (default:1.0)top_p(float): Nucleus sampling parameter (default:1.0)frequency_penalty(float): Frequency penalty (default:0.0)presence_penalty(float): Presence penalty (default:0.0)stop(string/list): One or more stop sequencesresponse_format(string): Format of the response,"text"or"json"(default:"text")reasoning_effort(string): Controls reasoning effort for reasoning-capable models —"low","medium", or"high"retry(int): Number of attempts (default:1). Retries stop immediately on an HTTP 400 (bad request), which can never succeed on retryfull_response(bool): Return the full converted API response object instead of the content (default:False)allow_error(bool): ReturnNoneinstead of raising on request failure (default:False)stream(bool): Enable streaming mode (default:False)stream_callback(callable): Function invoked with each chunk (converted to a Starlark value) as it arrives in streaming modekwargs(dict): Additional custom or provider-specific parameters; passed through to the API request'sChatTemplateKwargs
Returns: The generated content string when n == 1, or a list of content
strings when n > 1. With full_response=True, the converted full API response
object (its shape depends on legacy_mode — see
Configuration). In streaming mode the content is aggregated
from all chunks, and the full response (when requested) includes token usage
accumulated across chunks. Returns None if the response contains no choices.
Errors: Returns an error (or None under allow_error=True) when no model
is set, on an unsupported response_format, on invalid kwargs, or on any API
/ network failure after the configured retries.
draw(prompt, model?, n?, quality?, size?, style?, response_format?, background?, moderation?, output_format?, output_compression?, retry?, full_response?, allow_error?)
Generates an image using DALL-E 2, DALL-E 3, or GPT Image 1. The model is
selected by model (or the openai_dalle_model config option), and the valid
parameter set / defaults depend on which model is chosen.
Parameters:
prompt(string, required): Text prompt for the image generation- Max length: 32000 characters for
gpt-image-1, 4000 fordall-e-3, 1000 fordall-e-2
- Max length: 32000 characters for
model(string): Model to use (defaults to theopenai_dalle_modelconfig option). Supported:"dall-e-2","dall-e-3","gpt-image-1"n(int): Number of images to generate (default:1)dall-e-2/gpt-image-1: 1-10 images;dall-e-3: only1is supported
quality(string): Image quality (default:"auto"forgpt-image-1,"standard"for DALL-E)- GPT Image 1:
"auto","high","medium","low" - DALL-E 3:
"standard","hd" - DALL-E 2:
"standard"only
- GPT Image 1:
size(string): Image size (default:"auto"forgpt-image-1,"1024x1024"for DALL-E)- GPT Image 1:
"auto","1024x1024","1536x1024"(landscape),"1024x1536"(portrait) - DALL-E 3:
"1024x1024","1792x1024","1024x1792" - DALL-E 2:
"256x256","512x512","1024x1024"
- GPT Image 1:
style(string): Image style — DALL-E 3 only (default:"vivid")- DALL-E 3:
"vivid","natural"
- DALL-E 3:
response_format(string): Response format — DALL-E only (default:"url")- DALL-E 2/3:
"url","b64_json" - GPT Image 1: ignored; the API returns base64, which the module decodes and returns as raw image bytes
- DALL-E 2/3:
background(string): Background type — GPT Image 1 only (default:"auto")- GPT Image 1:
"auto","transparent","opaque"
- GPT Image 1:
moderation(string): Content moderation level — GPT Image 1 only (default:"auto")- GPT Image 1:
"auto","low"
- GPT Image 1:
output_format(string): Output image format — GPT Image 1 only (default:"png")- GPT Image 1:
"png","jpeg","webp"
- GPT Image 1:
output_compression(int): Compression level 0-100 — GPT Image 1 only (default:100)- GPT Image 1: only supported with
"jpeg"or"webp"output formats
- GPT Image 1: only supported with
retry(int): Number of attempts (default:1). Retries stop immediately on an HTTP 400 (bad request)full_response(bool): Return the full converted API response object instead of the content (default:False)allow_error(bool): ReturnNoneinstead of raising on request failure (default:False)
Returns: A URL string (DALL-E with response_format="url", the default) or
raw image bytes (DALL-E with response_format="b64_json", and always for GPT
Image 1 — the base64 returned by the API is decoded before it reaches the
script). When n > 1, returns a list of those values. With
full_response=True, returns the converted full API response instead.
Errors: Returns an error (or None under allow_error=True) when prompt
is empty, no model is set, a model-specific parameter is out of range (e.g.
dall-e-3 with n > 1, an invalid background / moderation / output_format
/ quality / size, output_compression outside 0-100), the response carries
no image data, or on any API / network failure after the configured retries.
load("llm", "chat")
# Simple text generation
response = chat(
text="What are the three laws of robotics?",
max_tokens=200,
)
print(response)
# Using JSON mode
json_resp = chat(
text="Generate a JSON object with the three laws of robotics. Include each law as a separate field.",
response_format="json",
max_tokens=200,
)
print(json_resp)load("llm", "draw")
# Generate with DALL-E 3
dalle_image = draw(
prompt="A futuristic city with flying cars and tall skyscrapers",
model="dall-e-3",
quality="hd",
size="1024x1024",
style="vivid",
)
print("DALL-E 3 image URL:", dalle_image)
# Generate with GPT Image 1 (returns raw image bytes)
gpt_image = draw(
prompt="A realistic portrait of a robot scientist in a laboratory",
model="gpt-image-1",
quality="high",
size="1024x1536", # Portrait orientation
background="transparent",
output_format="png",
)
print("GPT Image 1 image bytes length:", len(gpt_image))
# Content moderation with GPT Image 1
moderated_image = draw(
prompt="A family-friendly cartoon character playing in a park",
model="gpt-image-1",
quality="medium",
moderation="low",
output_format="webp",
output_compression=85,
)
print("Moderated image bytes length:", len(moderated_image))
# Generate multiple images with GPT Image 1
multiple_images = draw(
prompt="Abstract geometric patterns in bright colors",
model="gpt-image-1",
n=3,
quality="medium",
size="1024x1024",
output_format="jpeg",
output_compression=90,
)
print("Generated {} images".format(len(multiple_images)))
# Get the full response with token usage information
full_resp = draw(
prompt="An artistic illustration of AI concepts and neural networks",
model="gpt-image-1",
quality="high",
full_response=True,
)
print("Image data:", full_resp.data[0].b64_json[:100] + "...")
if hasattr(full_resp, "usage"):
usage = full_resp.usage
print("Total tokens: {}".format(usage.total_tokens))
print("Input tokens: {}".format(usage.input_tokens))
print("Output tokens: {}".format(usage.output_tokens))load("llm", "chat")
# Handle each chunk of the response
def handle_chunk(chunk):
# Access the delta content from the first choice
if len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
# Print each chunk as it arrives (Starlark's print always adds a newline)
print(delta.content)
# Stream the response with a callback
full_response = chat(
text="Write a short poem about coding in Python, one line at a time.",
max_tokens=200,
stream=True,
stream_callback=handle_chunk,
)
# The full_response contains the complete aggregated text
print("\n\nFull response:", full_response)load("llm", "chat")
# A simple progress tracker
tracker = {
"tokens": 0,
"started": False,
"done": False,
}
def process_chunk(chunk):
if not tracker["started"]:
print("Generating response...")
tracker["started"] = True
# Count chunks received
if len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
tracker["tokens"] += 1
# Show progress (Starlark's print always adds a newline)
if tracker["tokens"] % 10 == 0:
print(".")
# Generate a longer response with progress tracking
response = chat(
text="Explain how transformers work in machine learning, with detailed technical information.",
max_tokens=500,
stream=True,
stream_callback=process_chunk,
)
print("\nDone! Received", tracker["tokens"], "chunks.")
print("\nFinal response:\n", response)load("llm", "chat")
# Get the full response with token usage information
full_resp = chat(
text="Explain the concept of transfer learning in AI.",
max_tokens=300,
stream=True,
full_response=True,
)
# Access token usage information
if hasattr(full_resp, "usage"):
usage = full_resp.usage
print("Prompt tokens: {}".format(usage.prompt_tokens))
print("Completion tokens: {}".format(usage.completion_tokens))
print("Total tokens: {}".format(usage.total_tokens))
# Calculate an approximate cost (example rates for gpt-4)
# Note: token counts are accumulated from all stream responses for accuracy
prompt_cost = usage.prompt_tokens * 0.00003 # $0.03 per 1000 tokens
completion_cost = usage.completion_tokens * 0.00006 # $0.06 per 1000 tokens
total_cost = prompt_cost + completion_cost
print("Approximate cost: ${}".format(total_cost))load("llm", "chat")
# Ask about an image
response = chat(
text="What's in this image?",
image_url="https://example.com/image.jpg",
max_tokens=150,
)
print(response)load("llm", "message", "chat")
# Build a conversation history
messages = [
message(role="user", text="Hello, who are you?"),
message(role="assistant", text="I'm an AI assistant. How can I help you today?"),
message(role="user", text="Can you explain what an LLM is?"),
]
# Continue the conversation
response = chat(
messages=messages,
max_tokens=200,
)
print(response)load("llm", "chat", "set_openai_endpoint_url", "set_openai_api_key")
# Set the endpoint and API key for a reasoning-capable model provider
set_openai_endpoint_url("https://reasoning-model-api-endpoint.com")
set_openai_api_key("your-api-key-here")
# Call a reasoning-capable model with a specific reasoning effort
response = chat(
text="Solve this step by step: If 3x + 7 = 22, what is the value of x?",
model="reasoning-model-name",
reasoning_effort="high", # Can be "low", "medium", or "high"
max_tokens=300,
full_response=True,
)
# If the model provides reasoning content, it is available on the message
if hasattr(response.choices[0].message, "reasoning_content"):
print("Reasoning:")
print(response.choices[0].message.reasoning_content)
print("\nFinal answer:")
print(response.choices[0].message.content)load("llm", "chat")
# Use kwargs for custom, experimental, or provider-specific parameters
response = chat(
text="Generate a creative story about space exploration.",
max_tokens=200,
kwargs={
"custom_parameter": "value",
"experimental_feature": True,
"custom_config": {
"setting_a": "option1",
"setting_b": 42,
},
},
)
print(response)
# Combine standard parameters with custom kwargs
response = chat(
text="Explain machine learning in simple terms.",
model="gpt-4",
temperature=0.7,
max_tokens=200,
kwargs={
"safety_level": "strict",
"response_style": "educational",
"custom_instruction": "Use analogies when possible",
},
full_response=True,
)
# Access both the standard response and any custom fields
print("Content:", response.choices[0].message.content)
if hasattr(response, "custom_fields"):
print("Custom response fields:", response.custom_fields)The module is built on starpkg/base's configurable-module system. Each option
is exposed to scripts as a pair of generated accessor builtins (loaded from the
llm module alongside the functions above):
get_<key>()— returns the current value of the option.set_<key>(value)— sets the option (takes a single value).
An option's value resolves in priority order: an explicit set_<key> value, the
environment variable, then the default. These options serve as defaults used by
chat / draw when the corresponding argument is not provided.
The table below describes the historical self-configuring constructors. When
using NewModuleWithHostPolicy, the provider, endpoint, and API-key setters
are omitted as described in Host-controlled construction.
openai_api_key is a secret option (SetSecret(true) in the code), so it
exposes only set_openai_api_key — there is no get_openai_api_key
builtin. The key can be set from a script but never read back.
| Option | Getter | Setter | Type | Env var | Default | Description |
|---|---|---|---|---|---|---|
openai_provider |
get_openai_provider |
set_openai_provider |
string | LLM_OPENAI_PROVIDER |
openai |
Provider type: openai, azure, or anthropic |
openai_endpoint_url |
get_openai_endpoint_url |
set_openai_endpoint_url |
string | LLM_OPENAI_ENDPOINT_URL |
"" |
API endpoint base URL (required for Azure; overrides the default for the others) |
openai_api_key |
(secret — none) | set_openai_api_key |
string | LLM_OPENAI_API_KEY |
"" |
API key (required). Secret: no getter is exposed to scripts |
openai_gpt_model |
get_openai_gpt_model |
set_openai_gpt_model |
string | LLM_OPENAI_GPT_MODEL |
"" |
Default model for chat |
openai_dalle_model |
get_openai_dalle_model |
set_openai_dalle_model |
string | LLM_OPENAI_DALLE_MODEL |
"" |
Default model for draw |
api_version |
get_api_version |
set_api_version |
string | LLM_API_VERSION |
2024-02-01 |
API version (used by Azure and Anthropic) |
legacy_mode |
get_legacy_mode |
set_legacy_mode |
bool | LLM_LEGACY_MODE |
true |
Conversion mode for full_response objects (see below) |
request_timeout |
get_request_timeout |
set_request_timeout |
int | LLM_REQUEST_TIMEOUT |
120 |
Per-request timeout in seconds. A blocking request (chat without stream, draw) uses it as a total deadline; a streaming chat uses it only as a connect + time-to-first-response bound (so a long stream is not truncated). A hanging endpoint can't stall the caller either way. Non-positive falls back to the default (never disabled) |
Example:
load("llm", "set_openai_endpoint_url", "set_openai_api_key", "set_openai_gpt_model", "chat")
set_openai_endpoint_url("https://api.openai.com/v1")
set_openai_api_key("sk-...") # secret: no get_openai_api_key
set_openai_gpt_model("gpt-4o")
print(chat(text="Hello!"))These accessors are also reachable as module attributes — e.g.
llm.set_openai_api_key("sk-...") — when the module is loaded under its llm
name rather than via load(...) of individual symbols.
Go hosts that run untrusted scripts can use
NewModuleWithHostPolicy(HostPolicy{...}) instead of the historical
NewModule / NewModuleWithConfig constructors:
mod := llm.NewModuleWithHostPolicy(llm.HostPolicy{
Provider: "openai",
EndpointURL: "https://api.openai.com/v1",
APIKey: os.Getenv("OPENAI_API_KEY"),
FileRoot: "/srv/llm-images",
})In this mode Provider, EndpointURL, and APIKey are host-only: their
set_openai_* builtins are not exposed, and the API key has no getter. The
client rejects redirects to a different scheme, host, or port than the policy
endpoint. FileRoot is canonicalized at construction; relative image_file
paths are resolved beneath it and symlinks that leave it are rejected. An empty
FileRoot disables image_file. Invalid policy values are returned by the
module loader. Model selection, API version, legacy conversion, and request
timeout remain script-configurable so the existing request surface stays small.
The file check is a path-level guard; a host that must resist concurrent
filesystem replacement still needs an OS-level sandbox or an immutable fixture
directory.
The historical NewModule and NewModuleWithConfig constructors are designed
so a script can self-configure the remote service — it may
set_openai_provider / set_openai_endpoint_url / set_openai_api_key to point
at its own provider (OpenAI, Azure, an Anthropic or OpenAI-compatible gateway)
and call chat / draw. That flexibility has two consequences a host must
understand before running untrusted scripts:
-
A host-injected API key travels to the script-chosen endpoint.
openai_api_keyis secret (write-only; no getter), but a script can stillset_openai_endpoint_url(...)to any host and then make a request — and the configured key is sent there in theAuthorizationheader. So if the host injects a key (viaNewModuleWithConfigorLLM_OPENAI_API_KEY) and runs an untrusted script, that script can exfiltrate the key by pointing the endpoint at a server it controls (and can reach internal addresses — an SSRF vector). Only inject a host API key for scripts you trust when using these historical constructors; for untrusted scripts, useNewModuleWithHostPolicyor run the self-configuring mode behind an egress allowlist. The policy constructor fixes the endpoint and rejects cross-origin redirects before a request can follow them. -
image_filereads arbitrary host files in the historical mode.chat(..., image_file="/path")(andmessage(image_file=...)) reads a host file and sends its bytes to the endpoint. The read is bounded (≤ 64 MiB) so it can't exhaust memory, but the path is not jailed — an untrusted script can read any host-readable file and mail it to the endpoint. UseNewModuleWithHostPolicywith a non-emptyFileRootto jail paths, or leaveFileRootempty to disable file reads; scripts can also pass image bytes they already hold viaimage=. -
Requests are time-bounded. Each HTTP request uses
request_timeoutseconds (default120), so a slow or hanging endpoint cannot stall the caller indefinitely.
legacy_mode controls how full_response=True results are turned into Starlark
values. When true (the default) the response is exposed via direct struct
access (field names like choices[0].message.content). When false, the
response is converted through JSON. The default preserves the historical
behavior; flip it with set_legacy_mode(False) only if you need JSON-shaped
access.