Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
cd54548
fix: compact instead of failing when the pre-send context check trips
anandgupta42 Nov 21, 2026
3787e24
fix: compact in place on a pre-send refusal; bound the calibrated mea…
anandgupta42 Nov 21, 2026
6bda81f
fix: recoverable message on a compacting refusal; tighter usage floor
anandgupta42 Nov 21, 2026
622f1a7
fix: address review of the context guard calibration
anandgupta42 Nov 21, 2026
4bfda4d
fix: replay the prompt on a turn-start refusal; keep echo recovery
anandgupta42 Nov 21, 2026
2f94162
fix: size the unmeasured tail with the guard's conservative estimator
anandgupta42 Nov 21, 2026
a425fe9
fix: treat an input-plus-reservation provider error as an overflow
anandgupta42 Nov 21, 2026
761bd28
fix: keep media turns explicit; environment-independent guard tests
anandgupta42 Nov 21, 2026
ec8edeb
fix: media check covers the whole turn; echo refusals are not compact…
anandgupta42 Nov 21, 2026
5392b42
fix: protect unseen tool-result media; trust usage only from tool steps
anandgupta42 Nov 21, 2026
4454131
fix: block compaction only for media the model has not seen
anandgupta42 Nov 21, 2026
f108190
refactor: explicit directive init; if/else for the refusal error
anandgupta42 Nov 21, 2026
5a9fe5c
fix: distrust reported usage far below the estimate
anandgupta42 Nov 21, 2026
4ddb528
fix: no compaction when the new content alone fills the window
anandgupta42 Nov 21, 2026
9a84ce8
fix: the oversize-tail rule leaves room for the response floor
anandgupta42 Nov 21, 2026
89bf33c
fix: oversize-tail check uses the input limit as is and counts only n…
anandgupta42 Nov 21, 2026
1468a6d
fix: tag tool-less requests instead of guessing; drop the oversize-ta…
anandgupta42 Nov 21, 2026
e9ac304
fix: add attachment allowances to the calibrated measure instead of d…
anandgupta42 Nov 21, 2026
43f522d
fix: tag deadline-finalization requests as sent without tools; pin th…
anandgupta42 Nov 21, 2026
3329c5b
fix: inherited deadline tagging; narrow the input-plus-reservation ov…
anandgupta42 Nov 21, 2026
ca66c30
fix: the guard's calibration tail counts tool-call inputs of later steps
anandgupta42 Nov 21, 2026
5202805
refactor: drop the provider input-plus-reservation pattern; share the…
anandgupta42 Nov 21, 2026
c1f6985
fix: account for request overhead in context calibration
anandgupta42 Nov 21, 2026
3e84d94
fix: use the filtered tool set only for the overhead estimate
anandgupta42 Nov 21, 2026
5082b82
fix: calibrate context guards using finalized requests
anandgupta42 Nov 21, 2026
aa59ce3
fix: calibrate context guard from full request estimate growth
anandgupta42 Nov 21, 2026
e41271d
fix: keep guard estimates lazy and share the metadata key
anandgupta42 Nov 21, 2026
2974cdb
fix: remove context guard leftovers and cover recovery paths
anandgupta42 Nov 21, 2026
0dac29a
fix: clarify attachment refusals and count retried directives once
anandgupta42 Nov 21, 2026
838c267
fix: word the media refusal for tool-returned images too
anandgupta42 Nov 21, 2026
e628fb7
fix: name the input limit in the media refusal message
anandgupta42 Nov 21, 2026
d516552
refactor: drop a redundant id check in the unseen-media scan
anandgupta42 Nov 21, 2026
c5d2a7e
docs: correct the context-measure comments
anandgupta42 Nov 21, 2026
7daeeca
docs: describe media allowances accurately in the clamp comment
anandgupta42 Nov 21, 2026
dc047e5
fix: avoid overestimating recursive file listings
anandgupta42 Nov 22, 2026
99049b6
fix: verify measured request content before reuse
anandgupta42 Nov 22, 2026
52555e9
test: record all observed fixture attachments
anandgupta42 Nov 22, 2026
a8450d4
fix: protect unseen media after a model switch
anandgupta42 Nov 22, 2026
966b9a4
fix: size the prepared provider request
anandgupta42 Nov 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/docs/configure/context-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,11 @@ These ratios are tuned against the cl100k_base tokenizer used by Claude and GPT-

!!! note "Limitations"
The heuristic uses JavaScript string length (UTF-16 code units), which over-estimates tokens for emoji (2 code units but ~1-2 tokens) and CJK characters. For precise token counting, a future update will integrate a native tokenizer.

## Pre-Send Size Check

Before each request, altimate checks that the prompt plus the space reserved for the response fits the model's context window, so an oversized request is caught before it is sent instead of failing at the provider. When the prompt leaves less room than requested, the response reservation is reduced first; the request is refused only when too little room would remain.

For the same model and agent, the check can start with the last response's reported input usage, including cached input, and add the estimated growth from appended messages and attachments. Replacing already-measured history, system text, tools, or provider instructions causes the full request estimate to be used. The check also uses the full estimate when the calibrated count falls below 60% of it, after a model or agent change or compaction summary, or when the last response lacks reported input usage or a recorded request estimate.

If the check finds the request too large and the conversation can be compacted, altimate compacts it and retries. This also applies when context pressure reduces the response reservation so far that the reasoning budget cannot be reconciled with the visible output floor. If the original reservation also fails the reasoning budget check, the refusal is a configuration error. The size check fails with "Context budget exceeded before the request was sent" (or "Input budget exceeded" for models with a separate input limit) when the request still does not fit after compaction (the first request after a compaction is sized by the full estimate and is not compacted again), when there is nothing to compact yet (a first message that is already too large), or when auto-compaction is disabled. If an unseen image or PDF (attached, or returned by a tool) blocks auto-compaction, the error suggests running `/compact` and retrying.
113 changes: 108 additions & 5 deletions packages/opencode/src/provider/output-token-budget.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,29 @@ const log = Log.create({ service: "provider.output-token-budget" })
/** Smallest completion budget worth sending for an agent turn. */
export const OUTPUT_TOKEN_FLOOR = 1_024

export const REQUEST_ESTIMATE_METADATA_KEY = "request_estimate_tokens"
export const REQUEST_SHAPE_METADATA_KEY = "request_estimate_shape"
export const REQUEST_MEDIA_METADATA_KEY = "request_media_fingerprints"

export type RequestEstimateShape = {
readonly messageCount: number
readonly messageHash: string
readonly overheadHash: string
}

export function requestEstimateShape(value: unknown): RequestEstimateShape | undefined {
if (!isRecord(value)) return undefined
const { messageCount, messageHash, overheadHash } = value
if (typeof messageCount !== "number" || !Number.isSafeInteger(messageCount) || messageCount < 0) return undefined
if (typeof messageHash !== "string" || typeof overheadHash !== "string") return undefined
return { messageCount, messageHash, overheadHash }
}

export function requestEstimateHash(value: unknown, mediaMessages?: readonly unknown[]): string {
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
const media = mediaMessages && messageMediaAllowances(mediaMessages, false)
return Bun.hash(serializeForEstimate(value, media).text).toString(16)
}

const CLAMP_MARGIN_FRACTION = 0.02
const CLAMP_MARGIN_MIN = 512
const ESTIMATE_CHUNK_SIZE = 400
Expand All @@ -20,11 +43,21 @@ const VIDEO_TOKEN_ALLOWANCE = 8_192
const SEMANTIC_MEDIA_BYTES_PER_TOKEN = 64
const DATA_URL_HEADER_LIMIT = 1_024
const MIN_REASONING_BUDGET = 1_024
/**
* Lowest share of the character estimate that provider-reported usage is trusted at; below it the
* full estimate decides.
* The estimate runs ~15-20% high on SQL/JSON and higher on tool schemas; the floor bounds an
* undercount from usage that missed inputs changed since (per-step tool selection, refreshed rules).
*/
const MIN_CALIBRATED_ESTIMATE_FRACTION = 0.6
const EMOJI = /\p{Extended_Pictographic}/u
const DENSE_ASCII_CHARACTER = /[A-Za-z0-9+/_=-]/
const DENSE_ASCII_MIN_LENGTH = 32
const DENSE_ASCII_MIN_UNIQUE = 6
const DENSE_ASCII_EXTRA_FRACTION = 0.75
// The estimator scans JSON, so a listing line starts after a quote or an escaped newline.
// Require a dot on absolute paths to distinguish them from slash-prefixed encoded data.
const LISTING_PATH = /("|\\n)(\.\/(?:[\w.-]+\/)+[\w.-]+|\/?(?:[\w.-]+\/)+[\w.-]+\.[\w-]+)(?=\\?"|(?:\\r)?\\n)/g
const REASONING_BUDGET_KEYS = new Set(["budgetTokens", "thinkingBudget", "budget_tokens"])
const CONTEXT_WINDOW_BETAS = new Map([["context-1m-2025-08-07", 1_000_000]])
const MEDIA_PART_TYPES = new Set([
Expand Down Expand Up @@ -105,6 +138,7 @@ export class ReasoningTokenBudgetError extends Error {
readonly path: string
readonly configured: number
readonly maxOutputTokens: number
readonly contextInduced?: boolean
},
) {
super(
Expand Down Expand Up @@ -149,7 +183,7 @@ export function mergeRequestHeaders(...sources: readonly unknown[]): Record<stri
return result
}

/** Count opaque ASCII runs in one bounded-memory pass, including across estimator chunks. */
/** Count opaque ASCII runs, including those that cross estimator chunk boundaries. */
function denseAsciiCharacters(input: string): number {
let total = 0
let length = 0
Expand All @@ -159,7 +193,10 @@ function denseAsciiCharacters(input: string): number {
length = 0
unique.clear()
}
for (const character of input) {
// Directory separators in listing lines break dense runs. Long opaque filenames and
// directory names still get the same surcharge as other dense text.
const text = input.replace(LISTING_PATH, (_, prefix: string, path: string) => prefix + path.replaceAll("/", " "))
for (const character of text) {
if (!DENSE_ASCII_CHARACTER.test(character)) {
flush()
continue
Expand Down Expand Up @@ -277,7 +314,7 @@ function mediaTokenAllowance(part: JsonRecord): number {
}

/** Mark only actual ModelMessage content parts whose payload is provider media. */
function messageMediaAllowances(messages: readonly unknown[]): WeakMap<object, number> {
function messageMediaAllowances(messages: readonly unknown[], charge = true): WeakMap<object, number> {
const result = new WeakMap<object, number>()
const visited = new WeakSet<object>()

Expand All @@ -286,7 +323,7 @@ function messageMediaAllowances(messages: readonly unknown[]): WeakMap<object, n
visited.add(content)
for (const part of content) {
if (!isRecord(part)) continue
if (MEDIA_PART_TYPES.has(String(part.type))) result.set(part, mediaTokenAllowance(part))
if (MEDIA_PART_TYPES.has(String(part.type))) result.set(part, charge ? mediaTokenAllowance(part) : 0)

// Tool-result media is nested in the AI SDK's typed content output.
if (part.type !== "tool-result" || !isRecord(part.output)) continue
Expand Down Expand Up @@ -392,6 +429,57 @@ export function estimateInputTokens(input: {
return total
}

/** Provider-reported token usage of one finished assistant turn. */
type ReportedUsage = {
readonly total?: number
readonly input: number
readonly output: number
readonly cache: { readonly read: number; readonly write: number }
}

/**
* The context size the compaction trigger compares against its threshold: provider-reported usage
* plus an estimated uncounted tail inflated by 1/fraction.
*/
export function reportedContextTokens(input: {
readonly tokens: ReportedUsage
readonly estimatedTokens?: number
readonly fraction: number
}): number {
const count =
input.tokens.total || input.tokens.input + input.tokens.output + input.tokens.cache.read + input.tokens.cache.write
const estimated = input.estimatedTokens ?? 0
return estimated > 0 ? count + Math.ceil(estimated / input.fraction) : count
}

/**
* Prefer the provider-reported size of the same conversation when it is smaller than the character
* estimate, which runs high on SQL- and JSON-heavy context. Never raises the estimate, and distrusts
* usage below MIN_CALIBRATED_ESTIMATE_FRACTION of it; in those cases the estimate decides, so the
* guard can still refuse (and the session compact) below the compaction trigger.
*/
export function calibratedInputTokens(estimate: number, reported: number | undefined): number {
if (reported === undefined || !Number.isFinite(reported) || reported <= 0) return estimate
// Usage far below the estimate more likely missed inputs (partial reporting, a changed request)
// than measured them: the estimate decides, so the response reservation is clamped as before.
if (reported < estimate * MIN_CALIBRATED_ESTIMATE_FRACTION) return estimate
Comment thread
anandgupta42 marked this conversation as resolved.
Resolved
Comment thread
anandgupta42 marked this conversation as resolved.
Resolved
return Math.min(estimate, reported)
}

/** True for the pre-send refusals that compacting the conversation can resolve. */
export function isContextBudgetError(
error: unknown,
): error is
| OutputTokenBudgetError
| InputTokenBudgetError
| (ReasoningTokenBudgetError & { readonly info: { readonly contextInduced: true } }) {
return (
error instanceof OutputTokenBudgetError ||
error instanceof InputTokenBudgetError ||
(error instanceof ReasoningTokenBudgetError && error.info.contextInduced === true)
)
}

/** Resolve a direct or lazy input estimate after cheap no-op checks have passed. */
function resolveInputTokens(value: number | (() => number)): number {
return typeof value === "function" ? value() : value
Expand Down Expand Up @@ -523,10 +611,25 @@ function transformReasoningBudgets(
export function clampReasoningBudget(
options: Record<string, any>,
maxOutputTokens: number | undefined,
requested?: number,
): Record<string, any> {
if (maxOutputTokens === undefined) return options
const ceiling = Math.floor(maxOutputTokens - OUTPUT_TOKEN_FLOOR)
return transformReasoningBudgets(options, ceiling, maxOutputTokens)
try {
return transformReasoningBudgets(options, ceiling, maxOutputTokens)
} catch (error) {
if (
error instanceof ReasoningTokenBudgetError &&
requested !== undefined &&
requested > maxOutputTokens
) {
// A small explicit budget may fit below the usual minimum. Check the original options
// against the original reservation so only refusals compaction can fix are flagged.
clampReasoningBudget(options, requested)
throw new ReasoningTokenBudgetError({ ...error.info, contextInduced: true })
}
throw error
}
}

export * as OutputTokenBudget from "./output-token-budget"
Expand Down
15 changes: 9 additions & 6 deletions packages/opencode/src/session/compaction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ import { Agent } from "@/agent/agent"
import { Plugin } from "@/plugin"
import { Config } from "@/config/config"
import { ProviderTransform } from "@/provider/transform"
// altimate_change start — reported usage plus the inflated uncounted tail
import { reportedContextTokens } from "@/provider/output-token-budget"
// altimate_change end
import { Telemetry } from "@/telemetry" // altimate_change — telemetry for compaction events
import { ModelID, ProviderID } from "@/provider/schema"
// altimate_change start — summarizer-integrity error
Expand Down Expand Up @@ -223,17 +226,17 @@ export namespace SessionCompaction {
const context = input.model.limit.context
if (context === 0) return false

const count =
input.tokens.total ||
input.tokens.input + input.tokens.output + input.tokens.cache.read + input.tokens.cache.write

const maxOutput = ProviderTransform.maxOutputTokens(input.model)
const reserved = config.compaction?.reserved ?? COMPACTION_BUFFER
const headroom = Math.max(reserved, maxOutput)
const base = input.model.limit.input ?? context
if (base <= headroom) return false
const estimated = input.estimatedTokens ?? 0
const adjusted = estimated > 0 ? count + Math.ceil(estimated / contextSafetyFraction(config)) : count
const adjusted = reportedContextTokens({
tokens: input.tokens,
estimatedTokens: input.estimatedTokens,
// Resolve the fraction only when an estimate needs it (an invalid override warns on each read).
fraction: (input.estimatedTokens ?? 0) > 0 ? contextSafetyFraction(config) : 1,
})
const threshold = overflowThreshold({ base, headroom, fraction: 1 })
return adjusted >= threshold
}
Expand Down
67 changes: 44 additions & 23 deletions packages/opencode/src/session/llm.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,14 @@ import { mergeDeep, pipe } from "remeda"
import { ProviderTransform } from "@/provider/transform"
// altimate_change start — size and clamp the finalized provider request
import {
calibratedInputTokens,
clampOutputTokens,
clampReasoningBudget,
effectiveContextWindow,
estimateInputTokens,
mergeRequestHeaders,
requestEstimateHash,
type RequestEstimateShape,
} from "@/provider/output-token-budget"
// altimate_change end
// altimate_change start — tool retrieval
Expand Down Expand Up @@ -81,6 +84,13 @@ export namespace LLM {
tools: Record<string, Tool>
retries?: number
toolChoice?: "auto" | "required" | "none"
// altimate_change start — calibrate from a measured step and the finalized request
/** Provider-reported input usage for the measured request, including cache reads and writes. */
measuredInputTokens?: number
measuredEstimateTokens?: number
measuredEstimateShape?: RequestEstimateShape
onRequestEstimate?: (tokens: number, shape: RequestEstimateShape) => void
// altimate_change end
}

export type StreamOutput = StreamTextResult<ToolSet, never>
Expand Down Expand Up @@ -346,10 +356,13 @@ export namespace LLM {
// altimate_change end

// altimate_change start — clamp after every context-affecting request field is finalized.
// Tool schemas and provider instructions consume the shared context window, while encoded
// media bytes do not count as literal text. The estimator runs lazily so providers that omit
// maxOutputTokens pay no serialization cost. Known context beta headers widen the catalog
// limit before the clamp. Fixed reasoning budgets are reconciled with the final reservation.
// Tool schemas and provider instructions share the window; encoded media is charged by the
// estimator's media allowances, not as literal text. The estimate runs only when the clamp needs it.
const effortOptions = runBudget?.effort ? applyBudgetEffort(input.model, params.options) : params.options
// Numeric reasoning clamps leave the reasoning mode unchanged.
const budgetedMessages = budgetMessages(input.messages, input.model, effortOptions)
const declaresNoTools = Object.keys(tools).filter((x) => x !== "invalid").length === 0
const requestMessages = declaresNoTools ? ProviderTransform.flattenToolParts(budgetedMessages) : budgetedMessages
const maxOutputTokens = clampOutputTokens({
model: input.model,
requested: params.maxOutputTokens,
Expand All @@ -358,25 +371,33 @@ export namespace LLM {
// Provider defaults are lower precedence than the exact case-normalized outgoing record.
headerSources: [provider.options, requestHeaders],
}),
inputTokens: () =>
estimateInputTokens({
system,
messages: ProviderTransform.messagesForInputEstimate(input.messages, input.model),
tools,
instructions: params.options.instructions,
}),
inputTokens: () => {
const messages = ProviderTransform.messagesForInputEstimate(requestMessages, input.model)
const overhead = { system, tools, instructions: params.options.instructions }
const estimate = estimateInputTokens({ ...overhead, messages })
const { measuredInputTokens, measuredEstimateTokens, measuredEstimateShape } = input
const overheadHash = requestEstimateHash(overhead)
const messageHash = requestEstimateHash(messages, messages)
input.onRequestEstimate?.(estimate, { messageCount: messages.length, messageHash, overheadHash })
// Removed text must not cancel the estimate of new, unmeasured content.
const prefix = measuredEstimateShape && messages.slice(0, measuredEstimateShape.messageCount)
const unchanged =
!measuredEstimateShape ||
(overheadHash === measuredEstimateShape.overheadHash &&
prefix?.length === measuredEstimateShape.messageCount &&
requestEstimateHash(prefix, prefix) === measuredEstimateShape.messageHash)
const calibrated =
unchanged &&
typeof measuredInputTokens === "number" &&
Number.isFinite(measuredInputTokens) &&
typeof measuredEstimateTokens === "number" &&
Number.isFinite(measuredEstimateTokens)
? measuredInputTokens + Math.max(0, estimate - measuredEstimateTokens)
: undefined
return calibratedInputTokens(estimate, calibrated)
Comment thread
anandgupta42 marked this conversation as resolved.
Resolved
},
})
// altimate_change start — deadline effort wins configured variants and plugin overrides
const requestOptions = clampReasoningBudget(
runBudget?.effort ? applyBudgetEffort(input.model, params.options) : params.options,
maxOutputTokens,
)
const requestMessages = budgetMessages(input.messages, input.model, requestOptions)
// altimate_change end
// altimate_change end

// altimate_change start — detect a toolless request once, for the message flattening below
const declaresNoTools = Object.keys(tools).filter((x) => x !== "invalid").length === 0
const requestOptions = clampReasoningBudget(effortOptions, maxOutputTokens, params.maxOutputTokens)
// altimate_change end

// altimate_change start — record final system text and resolved tools before provider middleware
Expand Down Expand Up @@ -484,7 +505,7 @@ export namespace LLM {
// never declares. The Altimate gateway fails every provider in its fallback chain on
// that shape and reports one generic error; see ProviderTransform.flattenToolParts.
// altimate_change start — keep interrupted reasoning in storage without sending empty disabled-thinking turns
...(declaresNoTools ? ProviderTransform.flattenToolParts(requestMessages) : requestMessages),
...requestMessages,
// altimate_change end
// altimate_change end
],
Expand Down
Loading
Loading