Repository navigation
feat: map bounded inline images natively - #469
Conversation
Refs: #468 Assisted-by: Codex Signed-off-by: sjungwon03 <sjungwon03@gmail.com>
sjungwon03-ai
left a comment
There was a problem hiding this comment.
Reviewed exact head 69f9621, the published diff, issue #468, plan/contracts, native history converters and shared transport dependencies. No blocking findings.
The shared helper enforces user role, omitted detail, selected MIME and a 20-occurrence bound before credentials; repeated references do not bypass the count. It reuses canonical inline validation and rejects marked image histories with fixed private failures. Anthropic emits base64 image sources and Google emits inlineData while preserving empty text/order and function-result correlation. Google arrays remain image-bearing user-only, and pending function results cannot be bypassed by a new image turn. Direct transport still snapshots, validates cache/control/history and serializes the body before resolving registered credentials against fixed hosts. OpenAI/delegated, Jev, IAM, routing and accounting flows are unchanged.
The public failing omitted-detail request was reproduced before the change. New tests cover native wire shapes, all selected MIME literals, 20/21 boundaries including repeated parts, unsupported detail/local Gemini GIF, roles/caches/remote URLs, mutation and first-value capture, authentication/Deny/limits, secrets/upstream failures, missing usage and required persistence failures. Both installed SDKs cover native text/function nonstream and streaming on both bases plus fresh explicit provider Deny on correlated follow-ups. Native image stream cancellation cancels the body and records failed possible billing. The changed test URL fixture remains limited to the two fixed Gemini endpoints.
Full local checks pass strict types, lint, 5482 tests with the existing PostgreSQL skip, planning/contracts/fixture checks and all offline pins; focused image/client checks pass 152 tests. All three pins are unchanged and fresh official comparison passes. Both required check jobs pass at this exact head. No new dependencies, database migrations, secret logging, remote retrieval or write-capable PR-head automation were added.
Contracts accurately keep this as a bounded omitted-detail subset. Gemini GIF exclusion is local, and no universal provider format or resolution-equivalence claim is made. Pixel/model/live-client certification, broader image formats/caching/output and full #116 remain open; Jev #7 stays unresolved. Approval is an AI-assisted review through the user-authorized sjungwon03-ai account, not independent human certification.
Assisted-by: Codex
Reviewed by Codex operating as sjungwon03-ai, as explicitly authorized by the contributor. Assisted-by: Codex.
Closes #468
Bounded inline images were accepted at the gateway but direct Anthropic/Gemini rejected every image history. This adds omitted-detail user images to both native routes on both chat bases, nonstream completions, text streams and custom-function streams. Anthropic receives ordered base64 image sources for PNG/JPEG/WebP/GIF; Gemini receives ordered inlineData for PNG/JPEG/WebP. Preserve exact payload/MIME, empty text, image-only/multiple arrays and correlated tool-result continuations.
The selected native subset permits at most 20 image occurrences across the complete history. Every supplied detail remains rejected before credentials; no resolution equivalence is invented. Gemini GIF is outside this conservative local subset, not a universal upstream incompatibility claim. Shared snapshots/body serialization precede asynchronous secret resolution. Preserve public role/URL/encoding/cache/body restrictions, registered fixed hosts, complete model/final-provider IAM/Deny, limits, private audit/usage/errors, required persistence and missing/possibly-billed failed-attempt accounting. OpenAI/delegated and Jev behavior remain unchanged; all three structural pins are byte-identical.
Red: node --test test/native-inline-images.test.ts reproduced the public omitted-detail Anthropic request returning 502 instead of 200. Minimal native mapping made this test pass. Green: node --test test/native-inline-images.test.ts test/sdk-native-inline-images.test.ts test/inline-image-inputs.test.ts test/sdk-inline-image-inputs.test.ts passes 152 tests. New raw cases cover exact native order/MIME/count bounds, image-only/multiple parts, detail/GIF/cache/role denial, first-read and async capture, secret/upstream failures, missing usage and audit/ledger failure. Actual OpenAI7.23.0/OpenRouter1.4.18 sockets cover 48 successful native requests and eight fresh explicit provider Deny requests across both bases, text/function streams and function-result continuations; two additional OpenAI image-stream aborts cancel native bodies and retain failed possibly-billed accounting. No live inference is used.
npm run format and git diff --check pass. Full npm run check passes strict types, lint, 5482 tests with one existing PostgreSQL skip, planning/contracts/fixture checks and all three offline pins. npm run compatibility:drift reports every selected subset unchanged against the fresh fixed-host bounded official retrieval.
Remaining limits: pixel/MIME authenticity, native model capability and live/provider certification are not validated; encoding-valid images can still fail upstream. Remote/file inputs, supplied native detail/resolution equivalence, broader MIME/image caching/output/richer inputs and complete external-client certification remain open under #116. Jev sendPrompt:true still rejects image histories before selector credentials; unresolved #7 stays open.
Plan: 468-native-inline-images. Contract: native-inline-images.
Validation: full npm run check passes strict typing, lint, 5482 tests with one existing PostgreSQL skip, planning/contracts and offline schema integrity. Full #116 remains open.
Assisted-by: Codex