diff --git a/contracts/inline-image-inputs.md b/contracts/inline-image-inputs.md index edc5394..da72a8a 100644 --- a/contracts/inline-image-inputs.md +++ b/contracts/inline-image-inputs.md @@ -6,7 +6,7 @@ URL accepts only data:image/png|jpeg|webp|gif;base64,,data:}}`, selecting image/png, image/jpeg, image/webp and image/gif. Gemini maps to `{inlineData:{mimeType:,data:}}`, selecting PNG/JPEG/WebP. GIF is outside this conservative Gemini subset; its generic Blob reference lists more formats than its image guide, so this is not a universal claim that Gemini rejects GIF. Public HEIC/HEIF/AVIF inputs remain unimplemented. + +Remove only the data URL wrapper; preserve exact standard canonical base64 without transcoding, decoding pixels, checking MIME authenticity, resizing, fetching remote hosts or reading files. Preserve input part order, empty text, multiple/image-only arrays and image turns in function-result history. Native converters retain function IDs/names/results and existing adjacent-result grouping. Google still rejects non-image plain/marked arrays at its native boundary; public text-only arrays continue their existing normalization to strings. + +At most 20 image occurrences across the complete native history are accepted, including repeated references to the same part; 21 rejects before credentials. This is a local limit, not a provider maximum. Existing full-URL/history/part/HTTP-body limits apply independently. Unsupported role, remote/file URL, malformed encoding/fields and any request/block/tool cache mixture retain existing public early rejection. Direct converters also reject unsupported image/detail/MIME, non-user images and marked history rather than sending OpenAI blocks or dropping them. + +The adapter snapshots request history and constructs/serializes the native body before asynchronous secret resolution. Later mutation cannot change the wire payload, and an invalid first image field never retries a later value. Upstream/native model, pixel, animation and image-byte validity limits remain authoritative; encoding-valid data can still fail safely upstream. Provider model capabilities are not inferred from catalog metadata. + +Preserve authentication, complete model/final-provider IAM with explicit Deny, limits and registered fixed upstream hosts. No caller image URL becomes a proxy destination. Required usage/audit persistence, missing usage, streamed interruption, cancellation/backpressure, failed possibly-billed attempts and duplicate-accounting signals retain the shared behavior. Image data/MIME/detail/text and provider credentials stay outside operational events, ledger data and errors. Image inputs create no inferred token usage or cost. Secret failures before contact produce no billed usage record; post-contact failures/cancellation preserve possible billing. + +Actual installed OpenAI7.23.0/OpenRouter1.4.18 clients on local sockets cover both bases, native text/function nonstream/streams and correlated follow-ups with fresh provider Deny overriding Allow. OpenAI stream abort tests cancel both native bodies and preserve failed possibly-billed accounting. Tests use mock fixed providers and a tiny inline fixture; they do not certify live native models. All three structural pins remain byte-identical. + +OpenAI/delegated wire behavior, image source guard and Jev restrictions remain as documented in [inline images](inline-image-inputs.md) and [image schema](image-content-schema.md). Jev sendPrompt:true rejects images before selector credentials; metadata-only eligible selection remains unchanged. Remote inputs, supplied native detail/resolution equivalence, native image caching/outputs, richer parts and live per-model certification remain open under #116; #7 stays unresolved. See [plan](../docs/plans/468-native-inline-images.md), [Anthropic vision](https://platform.claude.com/docs/en/build-with-claude/vision), [Gemini image guide](https://ai.google.dev/gemini-api/docs/generate-content/image-understanding) and [Gemini Part/Blob reference](https://ai.google.dev/api/generate-content#Part). diff --git a/docs/PRD.md b/docs/PRD.md index d0d4743..1e87969 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -946,7 +946,7 @@ Offline checks validate all three pins without network; explicit live checks val Both chat bases accept bounded user image_url arrays with plain text, preserving exact order/empty text and omission of optional auto/low/high detail. Accept only canonical nonempty base64 data:image/png/jpeg/webp/gif URLs: at most524288 UTF-16 units per full URL and786432 across history, at most128 parts per image-bearing message; existing1MiB HTTP body cap remains. Validate encoding without certifying image pixels/MIME or live model capability. Reject other roles, remote/file URLs, original/unknown detail, unknown fields and any request/block/tool caching mixture before routes. Freeze captured primitives/parts/history before async work. -Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini reject before secrets. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. +Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini initially rejected images before secrets; #468 adds the omitted-detail native subset documented below. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets exercise wire shapes, streamed/nonstream completions and image/function continuation with fresh Deny. Existing structural pins stay byte-identical; transitive image definitions, native/cache/remote/richer mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/464-inline-image-inputs.md) and [contract](../contracts/inline-image-inputs.md). @@ -956,4 +956,14 @@ Version 31 adds complete ChatContentItems and ChatContentImage source definition Independent fixtures cover nested image and union drift, missing/non-object targets, stale version-30 and rehashed invalid exact maps. Controlled live CLI cases verify chat-only drift, one fixed-host credential-free retrieval, unchanged pins, no fetch for stale pins and no success output for malformed selected image sources. Full checks and a fresh official three-pin comparison apply. -This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). +This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, broader native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). + +## Bounded native inline images (#468) + +Direct Anthropic and Gemini map omitted-detail user images on both chat bases and nonstream/text/function streams. Anthropic selects PNG/JPEG/WebP/GIF base64 image sources; Gemini selects PNG/JPEG/WebP inlineData as a conservative subset. Generic Gemini Blob documentation also lists GIF, so its exclusion is a local restriction. Preserve exact MIME/base64, part order, empty text, image-only/multiple arrays and correlated function-result histories. All supplied detail values reject before credentials without invented resolution equivalence. + +Native histories accept at most20 image occurrences across all turns, including repeated parts; existing URL/history/part/body budgets also apply. Snapshot and serialize before secret resolution. No remote/file fetch, pixel validation, resizing, output images or marked-image caching is added. Google native plain/marked arrays without images remain rejected; public text-only normalization and Anthropic cache/function behavior stay intact. + +Authentication, complete model/final-provider IAM/Deny, limits, fixed registered hosts, private operational audit/usage/errors, required persistence and missing/possibly-billed failed attempts remain shared. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets cover both bases, streams/nonstream function follow-ups and fresh provider Deny; SDK image stream cancellation cancels native bodies and retains failed possible-billing records. Raw cases cover selected MIME, count boundaries, detail/GIF/cache/role denial, mutation, secret failure and usage/persistence errors. + +All three source pins remain byte-identical. Jev text disclosure still rejects image histories before selector credentials and metadata-only selection remains available. Native resolution/detail equivalence, broader MIME/caching/output/richer inputs, remote references, live per-model certification, full #116 and unresolved #7 remain open. See [plan](plans/468-native-inline-images.md) and [contract](../contracts/native-inline-images.md). diff --git a/docs/acceptance.md b/docs/acceptance.md index c0c3e5f..949218e 100644 --- a/docs/acceptance.md +++ b/docs/acceptance.md @@ -1345,7 +1345,7 @@ Offline checks validate all three pins without network; explicit live checks val Both chat bases accept bounded user image_url arrays with plain text, preserving exact order/empty text and omission of optional auto/low/high detail. Accept only canonical nonempty base64 data:image/png/jpeg/webp/gif URLs: at most524288 UTF-16 units per full URL and786432 across history, at most128 parts per image-bearing message; existing1MiB HTTP body cap remains. Validate encoding without certifying image pixels/MIME or live model capability. Reject other roles, remote/file URLs, original/unknown detail, unknown fields and any request/block/tool caching mixture before routes. Freeze captured primitives/parts/history before async work. -Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini reject before secrets. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. +Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini initially rejected images before secrets; #468 adds the omitted-detail native subset documented below. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets exercise wire shapes, streamed/nonstream completions and image/function continuation with fresh Deny. Existing structural pins stay byte-identical; transitive image definitions, native/cache/remote/richer mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/464-inline-image-inputs.md) and [contract](../contracts/inline-image-inputs.md). @@ -1355,4 +1355,14 @@ Version 31 adds complete ChatContentItems and ChatContentImage source definition Independent fixtures cover nested image and union drift, missing/non-object targets, stale version-30 and rehashed invalid exact maps. Controlled live CLI cases verify chat-only drift, one fixed-host credential-free retrieval, unchanged pins, no fetch for stale pins and no success output for malformed selected image sources. Full checks and a fresh official three-pin comparison apply. -This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). +This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, broader native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). + +## Bounded native inline images (#468) + +Direct Anthropic and Gemini map omitted-detail user images on both chat bases and nonstream/text/function streams. Anthropic selects PNG/JPEG/WebP/GIF base64 image sources; Gemini selects PNG/JPEG/WebP inlineData as a conservative subset. Generic Gemini Blob documentation also lists GIF, so its exclusion is a local restriction. Preserve exact MIME/base64, part order, empty text, image-only/multiple arrays and correlated function-result histories. All supplied detail values reject before credentials without invented resolution equivalence. + +Native histories accept at most20 image occurrences across all turns, including repeated parts; existing URL/history/part/body budgets also apply. Snapshot and serialize before secret resolution. No remote/file fetch, pixel validation, resizing, output images or marked-image caching is added. Google native plain/marked arrays without images remain rejected; public text-only normalization and Anthropic cache/function behavior stay intact. + +Authentication, complete model/final-provider IAM/Deny, limits, fixed registered hosts, private operational audit/usage/errors, required persistence and missing/possibly-billed failed attempts remain shared. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets cover both bases, streams/nonstream function follow-ups and fresh provider Deny; SDK image stream cancellation cancels native bodies and retains failed possible-billing records. Raw cases cover selected MIME, count boundaries, detail/GIF/cache/role denial, mutation, secret failure and usage/persistence errors. + +All three source pins remain byte-identical. Jev text disclosure still rejects image histories before selector credentials and metadata-only selection remains available. Native resolution/detail equivalence, broader MIME/caching/output/richer inputs, remote references, live per-model certification, full #116 and unresolved #7 remain open. See [plan](plans/468-native-inline-images.md) and [contract](../contracts/native-inline-images.md). diff --git a/docs/architecture.md b/docs/architecture.md index 6d42e88..468efa5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1153,7 +1153,7 @@ Offline checks validate all three pins without network; explicit live checks val Both chat bases accept bounded user image_url arrays with plain text, preserving exact order/empty text and omission of optional auto/low/high detail. Accept only canonical nonempty base64 data:image/png/jpeg/webp/gif URLs: at most524288 UTF-16 units per full URL and786432 across history, at most128 parts per image-bearing message; existing1MiB HTTP body cap remains. Validate encoding without certifying image pixels/MIME or live model capability. Reject other roles, remote/file URLs, original/unknown detail, unknown fields and any request/block/tool caching mixture before routes. Freeze captured primitives/parts/history before async work. -Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini reject before secrets. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. +Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini initially rejected images before secrets; #468 adds the omitted-detail native subset documented below. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets exercise wire shapes, streamed/nonstream completions and image/function continuation with fresh Deny. Existing structural pins stay byte-identical; transitive image definitions, native/cache/remote/richer mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/464-inline-image-inputs.md) and [contract](../contracts/inline-image-inputs.md). @@ -1163,4 +1163,14 @@ Version 31 adds complete ChatContentItems and ChatContentImage source definition Independent fixtures cover nested image and union drift, missing/non-object targets, stale version-30 and rehashed invalid exact maps. Controlled live CLI cases verify chat-only drift, one fixed-host credential-free retrieval, unchanged pins, no fetch for stale pins and no success output for malformed selected image sources. Full checks and a fresh official three-pin comparison apply. -This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). +This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, broader native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). + +## Bounded native inline images (#468) + +Direct Anthropic and Gemini map omitted-detail user images on both chat bases and nonstream/text/function streams. Anthropic selects PNG/JPEG/WebP/GIF base64 image sources; Gemini selects PNG/JPEG/WebP inlineData as a conservative subset. Generic Gemini Blob documentation also lists GIF, so its exclusion is a local restriction. Preserve exact MIME/base64, part order, empty text, image-only/multiple arrays and correlated function-result histories. All supplied detail values reject before credentials without invented resolution equivalence. + +Native histories accept at most20 image occurrences across all turns, including repeated parts; existing URL/history/part/body budgets also apply. Snapshot and serialize before secret resolution. No remote/file fetch, pixel validation, resizing, output images or marked-image caching is added. Google native plain/marked arrays without images remain rejected; public text-only normalization and Anthropic cache/function behavior stay intact. + +Authentication, complete model/final-provider IAM/Deny, limits, fixed registered hosts, private operational audit/usage/errors, required persistence and missing/possibly-billed failed attempts remain shared. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets cover both bases, streams/nonstream function follow-ups and fresh provider Deny; SDK image stream cancellation cancels native bodies and retains failed possible-billing records. Raw cases cover selected MIME, count boundaries, detail/GIF/cache/role denial, mutation, secret failure and usage/persistence errors. + +All three source pins remain byte-identical. Jev text disclosure still rejects image histories before selector credentials and metadata-only selection remains available. Native resolution/detail equivalence, broader MIME/caching/output/richer inputs, remote references, live per-model certification, full #116 and unresolved #7 remain open. See [plan](plans/468-native-inline-images.md) and [contract](../contracts/native-inline-images.md). diff --git a/docs/openrouter-compatibility.md b/docs/openrouter-compatibility.md index 793b355..5c58c6a 100644 --- a/docs/openrouter-compatibility.md +++ b/docs/openrouter-compatibility.md @@ -29,7 +29,7 @@ Tools with a hardcoded openrouter.ai host need a configurable endpoint or an int | Non-streaming text chat | One normalized choice with portable output/sampling controls, provider-bounded verbosity/effort, delegated min_p/top_a/repetition_penalty, bounded OpenAI/OpenRouter logprobs/top_logprobs with content/refusal token alternatives and bytes, and Gemini chosen/alternative probabilities with unavailable bytes | Anthropic probability mapping, remaining request/response schema, sampling and capability metadata | | Streaming | Managed OpenAI/delegated OpenRouter nullable probability controls and bounded choice-level content/refusal token alternatives/bytes, including private final usage-choice probabilities after required handoffs; managed Gemini HTTP text/function streams with exact initial version identity, complete objects with bounded same-part signatures through raw HTTP/OpenAI SDK and explicitly configured OpenCode, dense call indices, clean framed EOF, final reported totals and actual SDK/persisted direct/dual checks on both bases; managed Anthropic HTTP text/function streams with bounded JSON arguments, generated persisted direct/dual wiring, cumulative aggregate usage and actual OpenAI/OpenRouter SDK checks on both bases; managed OpenAI HTTP text/refusal/indexed function streams with per-attempt usage/audit, generated persisted direct/dual wiring and actual SDK checks on both bases; delegated HTTP text/refusal/scalar/detail reasoning and indexed function streams with bounded validation, awaited delivery, cancellation, final usage metadata and interruption audit; both installed SDKs exercise function workflows on both bases; OpenCode 1.18.5 reads a fixture through a streamed function and completes correlated results on delegated and managed OpenAI/Anthropic/Gemini fixture routes | Gemini partial-argument streams and Anthropic/Gemini thinking/server-tool streams, native custom/multimodal and other tool variants, unselected schema targets, additional stream option fields and full named external-client conformance | | Tool calling | Validated function-tool requests and complete text-only result histories; delegated OpenRouter nonstream/stream assistant calls and direct OpenAI nonstream/stream calls and bounded managed Anthropic nonstream/stream declarations/choices/calls/correlated results; bounded managed Gemini nonstream/stream declarations/choices/calls/results with thinking disabled, plus official nonstream/stream same-part tool-call signature replay through raw HTTP/OpenAI SDK and explicitly configured OpenCode; SDK socket tests cover two-function continuations and fresh IAM | Server/custom tools, rich content, other native mappings and broader named external-tool workflows and application configurations | -| Rich inputs and outputs | Text-only parts normalize to strings; validated refusal, scalar reasoning (including bounded reasoning-only stop/length) and summary/text/encrypted detail responses; service tier/fingerprint/native finish metadata on supported routes | Multimodal/cached content, native thinking/block semantics, broader structured reasoning request controls and native detail history, local JSON-schema output enforcement and server-tool details | +| Rich inputs and outputs | Bounded canonical inline user images on direct OpenAI/delegated OpenRouter; omitted-detail native images on Anthropic (PNG/JPEG/WebP/GIF) and Gemini (PNG/JPEG/WebP), with native history maximum20; text-only parts normalize to strings; validated refusal, scalar reasoning (including bounded reasoning-only stop/length) and summary/text/encrypted detail responses; service tier/fingerprint/native finish metadata on supported routes | Remote/file images, supplied native detail/resolution controls, broader MIME/image caching/output/audio/video/file content, native thinking/block semantics, broader structured reasoning request controls and native detail history, local JSON-schema output enforcement and server-tool details | | Client routing controls | Rejected today | Client preferences narrow approved model/provider scope; no arbitrary destinations or authority widening | | Errors | /api/v1 numeric status codes, fixed messages, safe local reason/typed metadata, request ID and compatible midstream error chunks; legacy /v1 symbolic codes | Precise upstream error_type propagation, retry hints and full provider streaming errors | | Other model-use endpoints | Not implemented | Inventory completions, responses, embeddings and generation lookup against external-tool requirements and authorization | @@ -958,7 +958,7 @@ Offline checks validate all three pins without network; explicit live checks val Both chat bases accept bounded user image_url arrays with plain text, preserving exact order/empty text and omission of optional auto/low/high detail. Accept only canonical nonempty base64 data:image/png/jpeg/webp/gif URLs: at most524288 UTF-16 units per full URL and786432 across history, at most128 parts per image-bearing message; existing1MiB HTTP body cap remains. Validate encoding without certifying image pixels/MIME or live model capability. Reject other roles, remote/file URLs, original/unknown detail, unknown fields and any request/block/tool caching mixture before routes. Freeze captured primitives/parts/history before async work. -Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini reject before secrets. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. +Managed OpenAI/delegated OpenRouter forward images on nonstream/text/refusal/function streams through shared approved model/final-provider IAM/Deny, limits, fixed hosts, required audit/usage and safe missing/possibly-billed failure/cancellation handling. Native Anthropic/Gemini initially rejected images before secrets; #468 adds the omitted-detail native subset documented below. Jev text disclosure rejects image histories before selector credentials rather than dropping or disclosing images; metadata-only selection remains available. Images never enter operational records/errors or generate inferred tokens/cost; no gateway image fetch or output-image support is added. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets exercise wire shapes, streamed/nonstream completions and image/function continuation with fresh Deny. Existing structural pins stay byte-identical; transitive image definitions, native/cache/remote/richer mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/464-inline-image-inputs.md) and [contract](../contracts/inline-image-inputs.md). @@ -968,4 +968,14 @@ Version 31 adds complete ChatContentItems and ChatContentImage source definition Independent fixtures cover nested image and union drift, missing/non-object targets, stale version-30 and rehashed invalid exact maps. Controlled live CLI cases verify chat-only drift, one fixed-host credential-free retrieval, unchanged pins, no fetch for stale pins and no success output for malformed selected image sources. Full checks and a fresh official three-pin comparison apply. -This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). +This extends source coverage following #464; the bounded inline-image runtime contract, authorization, secrets, audit, usage and provider restrictions stay unchanged. Remote URLs, original/unknown detail, richer content, broader native image mappings, live model certification, full #116 and unresolved #7 remain open. See [plan](plans/466-image-content-schema.md) and [contract](../contracts/image-content-schema.md). + +## Bounded native inline images (#468) + +Direct Anthropic and Gemini map omitted-detail user images on both chat bases and nonstream/text/function streams. Anthropic selects PNG/JPEG/WebP/GIF base64 image sources; Gemini selects PNG/JPEG/WebP inlineData as a conservative subset. Generic Gemini Blob documentation also lists GIF, so its exclusion is a local restriction. Preserve exact MIME/base64, part order, empty text, image-only/multiple arrays and correlated function-result histories. All supplied detail values reject before credentials without invented resolution equivalence. + +Native histories accept at most20 image occurrences across all turns, including repeated parts; existing URL/history/part/body budgets also apply. Snapshot and serialize before secret resolution. No remote/file fetch, pixel validation, resizing, output images or marked-image caching is added. Google native plain/marked arrays without images remain rejected; public text-only normalization and Anthropic cache/function behavior stay intact. + +Authentication, complete model/final-provider IAM/Deny, limits, fixed registered hosts, private operational audit/usage/errors, required persistence and missing/possibly-billed failed attempts remain shared. Actual installed OpenAI7.23.0/OpenRouter1.4.18 sockets cover both bases, streams/nonstream function follow-ups and fresh provider Deny; SDK image stream cancellation cancels native bodies and retains failed possible-billing records. Raw cases cover selected MIME, count boundaries, detail/GIF/cache/role denial, mutation, secret failure and usage/persistence errors. + +All three source pins remain byte-identical. Jev text disclosure still rejects image histories before selector credentials and metadata-only selection remains available. Native resolution/detail equivalence, broader MIME/caching/output/richer inputs, remote references, live per-model certification, full #116 and unresolved #7 remain open. See [plan](plans/468-native-inline-images.md) and [contract](../contracts/native-inline-images.md). diff --git a/docs/plans/468-native-inline-images.md b/docs/plans/468-native-inline-images.md new file mode 100644 index 0000000..fe7a3c2 --- /dev/null +++ b/docs/plans/468-native-inline-images.md @@ -0,0 +1,52 @@ +# Native Inline Image Inputs + +## Issue and problem + +- Issue: #468, following #464/#466 and tracking #116. +- Bounded inline user images reach direct OpenAI/delegated OpenRouter, but direct Anthropic/Gemini reject images before credentials even when native inline formats preserve them. +- Official Anthropic Messages base64 sources and Gemini generateContent inlineData can represent a reviewed subset without retrieving remote media. + +## Scope and expected behavior + +- Map user image parts with omitted detail to native ordered image/text arrays. Anthropic selects PNG/JPEG/WebP/GIF; Gemini selects PNG/JPEG/WebP as a conservative subset (generic Blob documentation also mentions GIF, so no universal GIF incompatibility is claimed). +- Preserve exact base64 payload/MIME, order, empty text and image-only messages across both chat bases, nonstream/text/function streams and correlated function-result histories. +- Native histories admit at most 20 images as an explicit local bound, plus the existing URL/history/part/body limits. All supplied detail values reject before credentials; no resolution-control equivalence is invented. +- Authentication, complete model/final-provider IAM/Deny, limits, registered fixed hosts, private audit/usage/errors, persistence and missing/possibly-billed usage remain shared. Images create no estimated tokens or billed cost. +- Remote/file input, output images, cache-marked images, richer parts and Jev text disclosure remain outside this change. Preserve three schema pins and #7's unresolved policy. + +## Design + +- Translate validated immutable image parts into provider-specific blocks before secret resolution. Share bounded inline image extraction and native-history count/role/detail checks; retain native function/result correlation and text/cache behavior. +- Permit Google arrays only for user image-bearing content; plain/marked text arrays unsupported by that converter remain rejected. Anthropic retains existing text/cache mapping. +- Split canonical captured data URLs without decoding pixels, resizing or transcoding. Native API/model/pixel limits can still reject an encoding-valid image. +- Alternatives: passing OpenAI blocks to native APIs loses semantics; dropping detail silently invents equivalence; remote media retrieval changes the trust boundary. All remain excluded. +- No costly new domain decision or ADR. Refer to updated PRD, architecture, acceptance, compatibility and native-inline-images/inline-image-inputs contracts. +- Sources: [Anthropic vision](https://platform.claude.com/docs/en/build-with-claude/vision), [Gemini image guide](https://ai.google.dev/gemini-api/docs/generate-content/image-understanding), [Gemini Part/Blob](https://ai.google.dev/api/generate-content#Part), [media resolution](https://ai.google.dev/gemini-api/docs/media-resolution). + +## TDD plan + +- First reproduce an omitted-detail Anthropic image request through createChatHandler returning 502 instead of 200. Record the expected failure before production changes. +- Add exact native shape/order/MIME/image-only/count boundary tests, detail/Gemini GIF denial before secrets, auth/IAM/limits/persistence failure cases, missing usage, upstream failure, captured mutation and function continuation. +- Verify actual installed OpenAI/OpenRouter clients against local sockets on both bases and nonstream/text/function streams, with fresh current Deny on follow-up. +- Implement only shared native extraction and the two existing history converter changes; run focused tests, npm run format, npm run check and live schema comparison. + +## Delivery + +- Issue/new branch/plan, red test, minimal mapping, green/security/client tests, English contracts/docs, full checks, exact-head PR review/CI and authorized merge. +- Risks: upstream model/image-byte validity and pixel/native limits are not certified; supplied detail and richer formats remain partial compatibility gaps. Rollback reverts native conversion and restores blanket denial together. +- Include red/green commands, unchanged pin evidence, mock-only client scope and material limitations in the PR. + +## Verification evidence + +Bounded inline images were accepted at the gateway but direct Anthropic/Gemini rejected every image history. This adds omitted-detail user images to both native routes on both chat bases, nonstream completions, text streams and custom-function streams. Anthropic receives ordered base64 image sources for PNG/JPEG/WebP/GIF; Gemini receives ordered inlineData for PNG/JPEG/WebP. Preserve exact payload/MIME, empty text, image-only/multiple arrays and correlated tool-result continuations. + +The selected native subset permits at most 20 image occurrences across the complete history. Every supplied detail remains rejected before credentials; no resolution equivalence is invented. Gemini GIF is outside this conservative local subset, not a universal upstream incompatibility claim. Shared snapshots/body serialization precede asynchronous secret resolution. Preserve public role/URL/encoding/cache/body restrictions, registered fixed hosts, complete model/final-provider IAM/Deny, limits, private audit/usage/errors, required persistence and missing/possibly-billed failed-attempt accounting. OpenAI/delegated and Jev behavior remain unchanged; all three structural pins are byte-identical. + +Red: node --test test/native-inline-images.test.ts reproduced the public omitted-detail Anthropic request returning 502 instead of 200. Minimal native mapping made this test pass. Green: node --test test/native-inline-images.test.ts test/sdk-native-inline-images.test.ts test/inline-image-inputs.test.ts test/sdk-inline-image-inputs.test.ts passes 152 tests. New raw cases cover exact native order/MIME/count bounds, image-only/multiple parts, detail/GIF/cache/role denial, first-read and async capture, secret/upstream failures, missing usage and audit/ledger failure. Actual OpenAI7.23.0/OpenRouter1.4.18 sockets cover 48 successful native requests and eight fresh explicit provider Deny requests across both bases, text/function streams and function-result continuations; two additional OpenAI image-stream aborts cancel native bodies and retain failed possibly-billed accounting. No live inference is used. + +npm run format and git diff --check pass. Full npm run check passes strict types, lint, 5482 tests with one existing PostgreSQL skip, planning/contracts/fixture checks and all three offline pins. npm run compatibility:drift reports every selected subset unchanged against the fresh fixed-host bounded official retrieval. + +Remaining limits: pixel/MIME authenticity, native model capability and live/provider certification are not validated; encoding-valid images can still fail upstream. Remote/file inputs, supplied native detail/resolution equivalence, broader MIME/image caching/output/richer inputs and complete external-client certification remain open under #116. Jev sendPrompt:true still rejects image histories before selector credentials; unresolved #7 stays open. + + +Full npm run check passes strict types, lint, 5482 tests with one existing PostgreSQL skip, planning/contracts/fixture checks and offline schema integrity. diff --git a/src/providers/anthropic-client-functions.ts b/src/providers/anthropic-client-functions.ts index a51f7a7..0650f1b 100644 --- a/src/providers/anthropic-client-functions.ts +++ b/src/providers/anthropic-client-functions.ts @@ -4,13 +4,13 @@ import { snapshotBoundedJsonObject, type ToolChoice, } from '../gateway/chat-tools.ts'; -import { hasInlineImages } from '../gateway/inline-image-parts.ts'; import { finalToolResultCacheControl, hasPromptCacheBreakpoints, } from '../gateway/prompt-cache-parts.ts'; import { DirectProviderFailure } from '../routing/invoke-jev-managed-route.ts'; import { type AssistantResponse, normalizeAssistantResponse } from './assistant-response.ts'; +import { prepareNativeImages } from './native-image-parts.ts'; function invalid(): never { throw new DirectProviderFailure('other', false, false); @@ -74,7 +74,7 @@ export function prepareAnthropicFunctions( /** Complete adjacent tool results form one native user turn, following the matching assistant. */ export function prepareAnthropicMessages(messages: readonly ChatMessage[]): readonly object[] { - if (hasInlineImages(messages)) invalid(); + const images = prepareNativeImages(messages, 'anthropic'); const native: { role: string; content: string | null | readonly object[] }[] = []; let results: object[] | undefined; for (const message of messages) { @@ -122,7 +122,22 @@ export function prepareAnthropicMessages(messages: readonly ChatMessage[]): read }), ], }); - } else native.push({ role: message.role, content: message.content }); + } else + native.push({ + role: message.role, + content: + typeof message.content === 'string' || message.content === null + ? message.content + : message.content.map((part) => { + if (part.type !== 'image_url') return part; + const image = images.get(part); + if (!image) invalid(); + return { + type: 'image', + source: { type: 'base64', media_type: image.mimeType, data: image.data }, + }; + }), + }); } return native; } diff --git a/src/providers/direct-chat.ts b/src/providers/direct-chat.ts index cd82403..10b82fa 100644 --- a/src/providers/direct-chat.ts +++ b/src/providers/direct-chat.ts @@ -33,7 +33,6 @@ import { snapshotParallelToolCalls, snapshotToolChoice, } from '../gateway/chat-tools.ts'; -import { hasInlineImages } from '../gateway/inline-image-parts.ts'; import { hasBlockCacheControls, hasPromptCacheBreakpoints, @@ -729,7 +728,6 @@ export function createDirectChatTransport( let messages: readonly ChatMessage[]; try { messages = snapshotChatMessages(request.messages); - if (registration.kind !== 'openai' && hasInlineImages(messages)) fail('other'); validatePromptCacheHistory(cacheControl, messages, cacheOptions, tools); if ( registration.kind !== 'anthropic' && diff --git a/src/providers/google-client-functions.ts b/src/providers/google-client-functions.ts index 0681f0c..d8b77e2 100644 --- a/src/providers/google-client-functions.ts +++ b/src/providers/google-client-functions.ts @@ -14,6 +14,7 @@ import { MAX_GOOGLE_SIGNATURE_UNITS, snapshotGoogleThoughtSignature, } from './google-thought-signature.ts'; +import { prepareNativeImages } from './native-image-parts.ts'; const LOCAL_PREFIX = 'og_google_missing_id_'; const LOCAL_ID = @@ -86,11 +87,29 @@ export function prepareGoogleFunctions( /** Preserve exact result strings, correlation and original call order without inventing native IDs. */ export function prepareGoogleMessages(messages: readonly ChatMessage[]): readonly object[] { + const images = prepareNativeImages(messages, 'google'); const native: object[] = []; let calls: readonly { id: string; function: { name: string } }[] = [], results = new Map(); for (const message of messages) { - if (typeof message.content !== 'string' && message.content !== null) invalid(); + if (typeof message.content !== 'string' && message.content !== null) { + if ( + message.role !== 'user' || + !message.content.some((part) => part.type === 'image_url') || + calls.length + ) + invalid(); + native.push({ + role: 'user', + parts: message.content.map((part) => { + if (part.type === 'text') return { text: part.text }; + const image = images.get(part); + if (!image) invalid(); + return { inlineData: image }; + }), + }); + continue; + } if (message.role === 'tool') { if ( !calls.some((call) => call.id === message.tool_call_id) || diff --git a/src/providers/native-image-parts.ts b/src/providers/native-image-parts.ts new file mode 100644 index 0000000..f81d495 --- /dev/null +++ b/src/providers/native-image-parts.ts @@ -0,0 +1,44 @@ +import type { ChatMessage } from '../gateway/chat-messages.ts'; +import { type InlineImagePart, snapshotInlineImagePart } from '../gateway/inline-image-parts.ts'; +import { validatePromptCacheHistory } from '../gateway/prompt-cache-parts.ts'; +import { DirectProviderFailure } from '../routing/invoke-jev-managed-route.ts'; + +interface NativeImage { + readonly mimeType: string; + readonly data: string; +} +function invalid(): never { + throw new DirectProviderFailure('other', false, false); +} + +/** Validate the selected native subset before credentials; never fetch or transcode media. */ +export function prepareNativeImages( + messages: readonly ChatMessage[], + kind: 'anthropic' | 'google', +): ReadonlyMap { + const images = new Map(); + let count = 0; + try { + for (const message of messages) { + if (typeof message.content === 'string' || message.content === null) continue; + for (const part of message.content) { + const type = part.type; + if (type !== 'image_url') continue; + if (message.role !== 'user' || ++count > 20) invalid(); + const captured = snapshotInlineImagePart(part, type); + if (captured.image_url.detail !== undefined) invalid(); + const match = /^data:(image\/(?:png|jpeg|webp|gif));base64,(.+)$/u.exec( + captured.image_url.url, + ); + const mimeType = match?.[1]; + const data = match?.[2]; + if (!mimeType || !data || (kind === 'google' && mimeType === 'image/gif')) invalid(); + images.set(part, Object.freeze({ mimeType, data })); + } + } + if (count) validatePromptCacheHistory(undefined, messages); + } catch { + invalid(); + } + return images; +} diff --git a/test/native-inline-images-fixture.ts b/test/native-inline-images-fixture.ts new file mode 100644 index 0000000..c56ac8e --- /dev/null +++ b/test/native-inline-images-fixture.ts @@ -0,0 +1,84 @@ +import { chunk, frames } from './google-function-stream-fixture.ts'; +import { imageFixture, inlineUrl } from './inline-image-inputs-fixture.ts'; +import { probabilityFixture } from './nonstream-logprobs-fixture.ts'; + +export const nativeImageHistory = [ + { + role: 'user' as const, + content: [ + { type: 'text' as const, text: 'private image question Ω' }, + { type: 'image_url' as const, image_url: { url: inlineUrl } }, + { type: 'text' as const, text: '' }, + ], + }, +]; +export const nativeSdkHistory = [ + { + role: 'user' as const, + content: [ + { type: 'text' as const, text: 'private image question Ω' }, + { type: 'image_url' as const, imageUrl: { url: inlineUrl } }, + { type: 'text' as const, text: '' }, + ], + }, +]; +export const nativeBase64 = inlineUrl.split(',')[1]; +export function nativeExpected(kind: 'anthropic' | 'google', mimeType = 'image/png') { + return [ + { + role: 'user', + ...(kind === 'anthropic' + ? { + content: [ + { type: 'text', text: 'private image question Ω' }, + { + type: 'image', + source: { type: 'base64', media_type: mimeType, data: nativeBase64 }, + }, + { type: 'text', text: '' }, + ], + } + : { + parts: [ + { text: 'private image question Ω' }, + { inlineData: { mimeType, data: nativeBase64 } }, + { text: '' }, + ], + }), + }, + ]; +} +export function nativeImageFixture( + kind: 'anthropic' | 'google', + options: NonNullable[1]> = {}, +) { + if (kind === 'anthropic') return imageFixture(kind, options); + const usage = options.missingUsage + ? undefined + : { promptTokenCount: 2, candidatesTokenCount: 1, totalTokenCount: 3 }; + const parts = + options.mode === 'function' + ? [{ functionCall: { id: 'call', name: 'lookup', args: {} } }] + : options.mode === 'refusal' + ? [] + : [{ text: 'reply' }]; + const finish = options.mode === 'refusal' ? 'SAFETY' : 'STOP'; + return probabilityFixture(kind, undefined, { + ...options, + reply: () => + options.stream + ? new Response( + frames([ + chunk(parts, undefined, undefined, 'upstream-model'), + chunk([], finish, usage, 'upstream-model'), + ]), + { headers: { 'content-type': 'text/event-stream' } }, + ) + : Response.json({ + responseId: 'completion', + modelVersion: 'upstream-model', + candidates: [{ content: { parts }, finishReason: finish }], + ...(usage === undefined ? {} : { usageMetadata: usage }), + }), + }); +} diff --git a/test/native-inline-images.test.ts b/test/native-inline-images.test.ts new file mode 100644 index 0000000..b94b7b2 --- /dev/null +++ b/test/native-inline-images.test.ts @@ -0,0 +1,313 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { snapshotChatMessages } from '../src/gateway/chat-messages.ts'; +import { prepareAnthropicMessages } from '../src/providers/anthropic-client-functions.ts'; +import { prepareGoogleMessages } from '../src/providers/google-client-functions.ts'; +import { imageFixture, imagePrivacy, inlineUrl } from './inline-image-inputs-fixture.ts'; +import { + nativeBase64, + nativeExpected, + nativeImageFixture, + nativeImageHistory, +} from './native-inline-images-fixture.ts'; +import { probabilityTools } from './nonstream-logprobs-fixture.ts'; + +const nativeMessage = nativeImageHistory[0]; +assert.ok(nativeMessage); + +const nativeHistory = [ + { + role: 'user' as const, + content: [ + { type: 'text' as const, text: 'private image question Ω' }, + { type: 'image_url' as const, image_url: { url: inlineUrl } }, + { type: 'text' as const, text: '' }, + ], + }, +]; +test('Anthropic omitted-detail images cross the authorized public boundary in native order', async () => { + const f = imageFixture('anthropic'); + const response = await f.handler(f.request('/api/v1', { messages: nativeHistory })); + assert.equal(response.status, 200); + await response.text(); + assert.deepEqual(f.sent[0]?.messages, [ + { + role: 'user', + content: [ + { type: 'text', text: 'private image question Ω' }, + { + type: 'image', + source: { type: 'base64', media_type: 'image/png', data: inlineUrl.slice(22) }, + }, + { type: 'text', text: '' }, + ], + }, + ]); + imagePrivacy(f); +}); + +for (const kind of ['anthropic', 'google'] as const) + for (const stream of [false, true]) + for (const mode of ['text', 'function'] as const) { + const fields = { + messages: nativeImageHistory, + stream, + ...(mode === 'function' ? { tools: probabilityTools } : {}), + }; + test(`${kind} ${mode} stream=${stream}: ordered native inline images survive both bases`, async () => { + for (const base of ['/v1', '/api/v1']) { + const f = nativeImageFixture(kind, { stream, mode }); + const response = await f.handler(f.request(base, fields)); + assert.equal(response.status, 200); + const output = await response.text(); + if (stream) assert.ok(output.endsWith('data: [DONE]\n\n')); + assert.doesNotMatch(output, /base64|iVBOR|private image/u); + assert.deepEqual( + f.sent[0]?.[kind === 'anthropic' ? 'messages' : 'contents'], + nativeExpected(kind), + ); + assert.equal(f.records[0]?.usage.totalTokens, 3); + assert.equal(f.counts().secrets, 1); + imagePrivacy(f); + } + }); + test(`${kind} ${mode} stream=${stream}: image auth, model/provider Deny, limits and persistence stay enforced`, async () => { + for (const [gate, status] of [ + ['auth', 401], + ['implicit', 403], + ['model', 403], + ['provider', 403], + ['limit', 429], + ['selection', 503], + ['audit', 503], + ['usage', 503], + ] as const) + for (const base of ['/v1', '/api/v1']) { + const f = nativeImageFixture(kind, { stream, mode, gate }); + const response = await f.handler(f.request(base, fields)); + const persisted = gate === 'audit' || gate === 'usage'; + assert.equal(response.status, stream && persisted ? 200 : status); + assert.doesNotMatch(await response.text(), /iVBOR|base64|private image|\[DONE\]/u); + assert.equal(f.counts().secrets, persisted ? 1 : 0); + assert.equal(f.sent.length, persisted ? 1 : 0); + imagePrivacy(f); + } + }); + test(`${kind} ${mode} stream=${stream}: missing usage and failed attempts never infer image cost`, async () => { + const missing = nativeImageFixture(kind, { stream, mode, missingUsage: true }); + const response = await missing.handler(missing.request('/api/v1', fields)); + assert.equal(response.status, 200); + await response.text(); + assert.equal(missing.records[0]?.usage.status, 'missing'); + assert.equal(missing.records[0]?.usage.totalTokens, null); + imagePrivacy(missing); + const failed = nativeImageFixture(kind, { stream, mode, transportFails: true }); + const error = await failed.handler(failed.request('/api/v1', fields)); + assert.equal(error.status, 502); + assert.doesNotMatch(await error.text(), /base64|iVBOR|private|\[DONE\]/u); + assert.equal(failed.records.length, 1); + assert.equal(failed.records[0]?.possiblyBilled, true); + assert.equal(failed.sent.length, 1); + imagePrivacy(failed); + }); + test(`${kind} ${mode} stream=${stream}: supplied detail and unselected native MIME reject before credentials`, async () => { + for (const image_url of [ + ...['auto', 'low', 'high'].map((detail) => ({ url: inlineUrl, detail })), + ...(kind === 'google' ? [{ url: inlineUrl.replace('/png', '/gif') }] : []), + ]) { + const f = nativeImageFixture(kind, { stream, mode }); + const response = await f.handler( + f.request('/api/v1', { + ...fields, + messages: [{ role: 'user', content: [{ type: 'image_url', image_url }] }], + }), + ); + assert.equal(response.status, 502); + assert.doesNotMatch(await response.text(), /base64|iVBOR|private|\[DONE\]/u); + assert.equal(f.counts().secrets, 0); + assert.equal(f.sent.length, 0); + imagePrivacy(f); + } + }); + } + +for (const kind of ['anthropic', 'google'] as const) { + test(`${kind}: secret failure and an invalid first image getter remain private without upstream contact`, async () => { + for (const stream of [false, true]) + for (const mode of ['text', 'function'] as const) { + const f = nativeImageFixture(kind, { + stream, + mode, + mutate: () => { + throw new Error(`private image ${inlineUrl} fixture-provider-key`); + }, + }); + const response = await f.handler( + f.request('/api/v1', { + messages: nativeImageHistory, + stream, + ...(mode === 'function' ? { tools: probabilityTools } : {}), + }), + ); + assert.equal(response.status, 502); + assert.doesNotMatch(await response.text(), /base64|iVBOR|private|fixture-provider-key/u); + assert.equal(f.counts().secrets, 1); + assert.equal(f.sent.length, 0); + assert.equal(f.records.length, 0); + imagePrivacy(f); + } + let reads = 0; + const image_url = Object.defineProperty({}, 'url', { + enumerable: true, + get: () => (++reads === 1 ? 'https://private.invalid/image' : inlineUrl), + }); + const invalid = nativeImageFixture(kind); + await assert.rejects(() => + invalid.call({ messages: [{ role: 'user', content: [{ type: 'image_url', image_url }] }] }), + ); + assert.equal(reads, 1); + assert.equal(invalid.counts().secrets, 0); + assert.equal(invalid.sent.length, 0); + }); + test(`${kind}: selected MIME types, multiple and image-only arrays preserve exact payloads`, async () => { + for (const mime of kind === 'anthropic' + ? ['png', 'jpeg', 'webp', 'gif'] + : ['png', 'jpeg', 'webp']) { + const url = inlineUrl.replace('/png', `/${mime}`); + const messages = [ + { + role: 'user', + content: Array.from({ length: 2 }, () => ({ type: 'image_url', image_url: { url } })), + }, + ]; + const f = nativeImageFixture(kind); + const response = await f.handler(f.request('/api/v1', { messages })); + assert.equal(response.status, 200); + await response.text(); + const image = + kind === 'anthropic' + ? { + type: 'image', + source: { type: 'base64', media_type: `image/${mime}`, data: nativeBase64 }, + } + : { inlineData: { mimeType: `image/${mime}`, data: nativeBase64 } }; + assert.deepEqual(f.sent[0]?.[kind === 'anthropic' ? 'messages' : 'contents'], [ + { role: 'user', [kind === 'anthropic' ? 'content' : 'parts']: [image, image] }, + ]); + imagePrivacy(f); + } + }); + test(`${kind}: twenty native images pass and twenty-one across history fail before credentials`, async () => { + for (const count of [20, 21]) { + const messages = Array.from({ length: count }, () => ({ + role: 'user', + content: [{ type: 'image_url', image_url: { url: inlineUrl } }], + })); + const f = nativeImageFixture(kind); + const response = await f.handler(f.request('/api/v1', { messages })); + assert.equal(response.status, count === 20 ? 200 : 502); + await response.text(); + assert.equal(f.counts().secrets, count === 20 ? 1 : 0); + assert.equal(f.sent.length, count === 20 ? 1 : 0); + imagePrivacy(f); + } + const converter = kind === 'anthropic' ? prepareAnthropicMessages : prepareGoogleMessages; + const repeated = { type: 'image_url' as const, image_url: { url: inlineUrl } }; + assert.throws(() => converter([{ role: 'user', content: Array(21).fill(repeated) }])); + }); + test(`${kind}: invalid images, non-user roles and cache mixtures still reject before routing`, async () => { + for (const fields of [ + { + messages: [ + { + role: 'user', + content: [ + { type: 'image_url', image_url: { url: 'https://example.invalid/image.png' } }, + ], + }, + ], + }, + { + messages: [ + { + role: 'user', + content: [{ type: 'image_url', image_url: { url: 'data:image/png;base64,YR==' } }], + }, + ], + }, + { messages: [{ role: 'assistant', content: nativeMessage.content }] }, + { cache_control: { type: 'ephemeral' } }, + { prompt_cache_options: { mode: 'explicit' } }, + { tools: [{ ...probabilityTools[0], cache_control: { type: 'ephemeral' } }] }, + { + messages: [ + { + role: 'user', + content: [ + { type: 'text', text: 'private', cache_control: { type: 'ephemeral' } }, + ...nativeMessage.content, + ], + }, + ], + }, + ]) { + const f = nativeImageFixture(kind); + const response = await f.handler( + f.request('/api/v1', { messages: nativeImageHistory, ...fields }), + ); + assert.equal(response.status, 400); + assert.deepEqual(f.counts(), { routes: 0, secrets: 0 }); + assert.doesNotMatch(await response.text(), /base64|iVBOR|private/u); + } + }); + test(`${kind}: captured image/text cannot change at secret resolution`, async () => { + const messages = structuredClone(nativeImageHistory); + const first = messages[0]; + assert.ok(first); + const f = nativeImageFixture(kind, { + mutate: () => { + first.content.splice(0, first.content.length); + }, + }); + await f.call({ messages }); + assert.deepEqual( + f.sent[0]?.[kind === 'anthropic' ? 'messages' : 'contents'], + nativeExpected(kind), + ); + let reads = 0; + const image_url = Object.defineProperty({}, 'url', { + enumerable: true, + get: () => (++reads === 1 ? inlineUrl : 'private invalid second value'), + }); + const one = nativeImageFixture(kind); + await one.call({ messages: [{ role: 'user', content: [{ type: 'image_url', image_url }] }] }); + assert.equal(reads, 1); + assert.equal(one.sent.length, 1); + }); + test(`${kind}: exported converters reject non-user images, malformed payloads, cache mixtures and unsupported Google plain arrays`, () => { + const converter = kind === 'anthropic' ? prepareAnthropicMessages : prepareGoogleMessages; + assert.deepEqual(converter(snapshotChatMessages(nativeImageHistory)), nativeExpected(kind)); + assert.throws(() => converter([{ role: 'assistant', content: nativeMessage.content }])); + assert.throws(() => + converter([ + { + role: 'user', + content: [{ type: 'image_url', image_url: { url: 'https://private.invalid/image' } }], + }, + ]), + ); + assert.throws(() => + converter([ + ...nativeImageHistory, + { + role: 'user', + content: [{ type: 'text', text: 'private', cache_control: { type: 'ephemeral' } }], + }, + ]), + ); + if (kind === 'google') + assert.throws(() => + converter([{ role: 'user', content: [{ type: 'text', text: 'plain' }] }]), + ); + }); +} diff --git a/test/nonstream-logprobs-fixture.ts b/test/nonstream-logprobs-fixture.ts index bddf75e..38b7fc7 100644 --- a/test/nonstream-logprobs-fixture.ts +++ b/test/nonstream-logprobs-fixture.ts @@ -40,6 +40,7 @@ export function probabilityFixture( missingUsage?: boolean; absent?: boolean; raw?: boolean; + stream?: boolean; transportFails?: boolean; mutate?: () => void; gate?: 'auth' | 'implicit' | 'model' | 'provider' | 'limit' | 'selection' | 'audit' | 'usage'; @@ -72,7 +73,7 @@ export function probabilityFixture( ? 'https://api.openai.com/v1/chat/completions' : kind === 'anthropic' ? 'https://api.anthropic.com/v1/messages' - : 'https://generativelanguage.googleapis.com/v1beta/models/upstream-model:generateContent', + : `https://generativelanguage.googleapis.com/v1beta/models/upstream-model:${options.stream ? 'streamGenerateContent?alt=sse' : 'generateContent'}`, ); sent.push(JSON.parse(String(init?.body)) as Record); if (options.transportFails) return new Response('private upstream error', { status: 500 }); diff --git a/test/sdk-native-inline-images.test.ts b/test/sdk-native-inline-images.test.ts new file mode 100644 index 0000000..8032a9c --- /dev/null +++ b/test/sdk-native-inline-images.test.ts @@ -0,0 +1,324 @@ +import assert from 'node:assert/strict'; +import type { AddressInfo } from 'node:net'; +import test from 'node:test'; +import { OpenRouter } from '@openrouter/sdk'; +import OpenAI from 'openai'; +import { createNodeChatServer } from '../src/gateway/node-chat-server.ts'; +import { createRegisteredDirectTextStreamInvoker } from '../src/providers/direct-text-stream.ts'; +import { + frames as anthropicFrames, + start as anthropicStart, +} from './anthropic-function-stream-fixture.ts'; +import { chunk as googleChunk, frames as googleFrames } from './google-function-stream-fixture.ts'; +import { imagePrivacy } from './inline-image-inputs-fixture.ts'; +import { + nativeImageFixture as imageFixture, + nativeImageHistory as imageHistory, + nativeExpected, + nativeSdkHistory as sdkHistory, +} from './native-inline-images-fixture.ts'; +import { probabilityTools } from './nonstream-logprobs-fixture.ts'; + +for (const kind of ['anthropic', 'google'] as const) + for (const base of ['/v1', '/api/v1']) + for (const mode of ['text', 'function'] as const) + for (const stream of [false, true]) + test(`${kind} ${mode} stream=${stream}: both actual SDKs send inline image parts on ${base}`, async () => { + const f = imageFixture(kind, { mode, stream }); + const server = createNodeChatServer(f.ports); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + try { + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}${base}`; + const openai = new OpenAI({ + apiKey: 'fixture-proxy-key', + baseURL: url, + maxRetries: 0, + timeout: 3000, + }); + const result = await openai.chat.completions.create({ + model: 'chat', + messages: imageHistory, + stream, + ...(mode === 'function' ? { tools: probabilityTools } : {}), + }); + if (Symbol.asyncIterator in result) { + const chunks = []; + for await (const chunk of result) chunks.push(chunk); + assert.ok(chunks.length >= 2); + assert.doesNotMatch(JSON.stringify(chunks), /base64|iVBOR|private image/u); + } else { + assert.equal(result.choices[0]?.message.role, 'assistant'); + assert.doesNotMatch(JSON.stringify(result), /base64|iVBOR|private image/u); + } + const compatible = new OpenRouter({ + apiKey: 'fixture-proxy-key', + serverURL: url, + retryConfig: { strategy: 'none' }, + timeoutMs: 3000, + }); + const other = await compatible.chat.send({ + chatRequest: { + model: 'chat', + messages: sdkHistory, + stream, + ...(mode === 'function' ? { tools: probabilityTools } : {}), + }, + }); + if (Symbol.asyncIterator in other) { + const chunks = []; + for await (const chunk of other) chunks.push(chunk); + assert.ok(chunks.length >= 2); + assert.doesNotMatch(JSON.stringify(chunks), /base64|iVBOR|private image/u); + } else { + assert.ok('choices' in other); + assert.doesNotMatch(JSON.stringify(other), /base64|iVBOR|private image/u); + } + assert.equal(f.sent.length, 2); + for (const body of f.sent) + assert.deepEqual( + body[kind === 'anthropic' ? 'messages' : 'contents'], + nativeExpected(kind), + ); + assert.equal(f.records.length, 2); + imagePrivacy(f); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())), + ); + } + }); +for (const kind of ['anthropic', 'google'] as const) + for (const base of ['/v1', '/api/v1']) + test(`${kind}: actual SDK image/function follow-ups reevaluate current Deny on ${base}`, async () => { + const f = imageFixture(kind, { mode: 'function' }); + let deny = false; + const server = createNodeChatServer({ + ...f.ports, + authenticate: async (...args) => { + const p = await f.ports.authenticate(...args); + return ( + p && { + ...p, + statements: deny + ? [ + ...p.statements, + { effect: 'Deny', actions: ['llm:*'], resources: ['provider:provider'] }, + ] + : p.statements, + } + ); + }, + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + try { + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}${base}`; + const openai = new OpenAI({ + apiKey: 'fixture-proxy-key', + baseURL: url, + maxRetries: 0, + timeout: 3000, + }); + const first = await openai.chat.completions.create({ + model: 'chat', + messages: imageHistory, + tools: probabilityTools, + }); + const call = first.choices[0]?.message.tool_calls?.[0]; + assert.ok(call); + assert.equal(call.id, 'call'); + const next = [ + ...imageHistory, + { role: 'assistant' as const, content: null, tool_calls: [call] }, + { role: 'tool' as const, tool_call_id: call.id, content: 'private result' }, + ]; + await openai.chat.completions.create({ + model: 'chat', + messages: next, + tools: probabilityTools, + }); + assert.deepEqual( + f.sent[1]?.[kind === 'anthropic' ? 'messages' : 'contents'], + nativeFollowup(kind), + ); + deny = true; + await assert.rejects( + () => + openai.chat.completions.create({ + model: 'chat', + messages: next, + tools: probabilityTools, + }), + (error) => error instanceof OpenAI.APIError && error.status === 403, + ); + assert.equal(f.sent.length, 2); + deny = false; + const compatible = new OpenRouter({ + apiKey: 'fixture-proxy-key', + serverURL: url, + retryConfig: { strategy: 'none' }, + timeoutMs: 3000, + }); + const response = await compatible.chat.send({ + chatRequest: { + model: 'chat', + messages: sdkHistory, + tools: probabilityTools, + stream: false, + }, + }); + assert.ok('choices' in response); + const sdkCall = response.choices[0]?.message.toolCalls?.[0]; + assert.ok(sdkCall); + const follow = [ + ...sdkHistory, + { role: 'assistant' as const, content: null, toolCalls: [sdkCall] }, + { role: 'tool' as const, toolCallId: sdkCall.id, content: 'private result' }, + ]; + await compatible.chat.send({ + chatRequest: { model: 'chat', messages: follow, tools: probabilityTools, stream: false }, + }); + assert.equal(f.sent.length, 4); + assert.deepEqual( + f.sent[3]?.[kind === 'anthropic' ? 'messages' : 'contents'], + nativeFollowup(kind), + ); + deny = true; + await assert.rejects(() => + compatible.chat.send({ + chatRequest: { + model: 'chat', + messages: follow, + tools: probabilityTools, + stream: false, + }, + }), + ); + assert.equal(f.sent.length, 4); + assert.equal(f.records.length, 4); + imagePrivacy(f); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())), + ); + } + }); + +function nativeFollowup(kind: 'anthropic' | 'google') { + return [ + ...nativeExpected(kind), + ...(kind === 'anthropic' + ? [ + { + role: 'assistant', + content: [{ type: 'tool_use', id: 'call', name: 'lookup', input: {} }], + }, + { + role: 'user', + content: [{ type: 'tool_result', tool_use_id: 'call', content: 'private result' }], + }, + ] + : [ + { role: 'model', parts: [{ functionCall: { id: 'call', name: 'lookup', args: {} } }] }, + { + role: 'user', + parts: [ + { + functionResponse: { + id: 'call', + name: 'lookup', + response: { output: 'private result' }, + }, + }, + ], + }, + ]), + ]; +} + +for (const kind of ['anthropic', 'google'] as const) + test(`${kind}: actual SDK image stream abort cancels native body and preserves possibly-billed accounting`, { + timeout: 5000, + }, async () => { + const f = imageFixture(kind, { stream: true }); + let interrupted: (() => void) | undefined; + const done = new Promise((resolve) => { + interrupted = resolve; + }); + let cancelled = false; + const invoker = createRegisteredDirectTextStreamInvoker({ + registrations: [ + { providerId: 'provider', kind, credentialRef: 'secret/reference', maxOutputTokens: 128 }, + ], + resolveSecret: async () => 'fixture-provider-key', + fetcher: async (url, init) => { + assert.equal( + String(url), + kind === 'anthropic' + ? 'https://api.anthropic.com/v1/messages' + : 'https://generativelanguage.googleapis.com/v1beta/models/upstream-model:streamGenerateContent?alt=sse', + ); + const body = JSON.parse(String(init?.body)); + assert.deepEqual( + body[kind === 'anthropic' ? 'messages' : 'contents'], + nativeExpected(kind), + ); + f.sent.push(body); + return new Response( + new ReadableStream({ + start(controller) { + controller.enqueue( + new TextEncoder().encode( + kind === 'anthropic' + ? anthropicFrames([anthropicStart('upstream-model')]) + : googleFrames([ + googleChunk([{ text: 'reply' }], undefined, undefined, 'upstream-model'), + ]), + ), + ); + }, + cancel() { + cancelled = true; + }, + }), + { headers: { 'content-type': 'text/event-stream' } }, + ); + }, + }); + const server = createNodeChatServer({ + ...f.ports, + invokeDirectTextStream: invoker, + writeAudit: async (event) => { + f.audits.push(event); + if (event.kind === 'stream-interrupted') interrupted?.(); + }, + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + try { + const client = new OpenAI({ + apiKey: 'fixture-proxy-key', + baseURL: `http://127.0.0.1:${(server.address() as AddressInfo).port}/api/v1`, + maxRetries: 0, + }); + const stream = await client.chat.completions.create({ + model: 'chat', + messages: imageHistory, + stream: true, + }); + const iterator = stream[Symbol.asyncIterator](); + assert.equal((await iterator.next()).value?.choices[0]?.delta.role, 'assistant'); + stream.controller.abort(); + await done; + assert.equal(cancelled, true); + assert.equal(f.records.length, 1); + assert.equal(f.records[0]?.outcome, 'failed'); + assert.equal(f.records[0]?.possiblyBilled, true); + imagePrivacy(f); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())), + ); + } + });