fix: stop overwriting multi-message INPUT_VALUE in OpenAI instrumentor#184
fix: stop overwriting multi-message INPUT_VALUE in OpenAI instrumentor#184nigamgauri wants to merge 2 commits into
Conversation
|
Comparing this with #152 they fix the same bug but land in very different places. #152 removes the JSON write and keeps eval_input joined as a flat string. This PR does the opposite drops eval_input entirely and keeps the structured JSON. For an observability tool the JSON is clearly right: role metadata survives, the trace can actually be replayed or filtered by role downstream. The flat string in #152 throws that away permanently. One edge case I don't see tested: a message where content is a list containing only image_url items with no text. That hits continue at line 41 and the message is silently absent from input_content so INPUT_VALUE gets set but that turn just isn't in it. The URL does end up in INPUT_IMAGES but there's no indication in the span that a message was dropped. Worth either documenting that this is intentional, or adding the message dict (without the image bytes) to input_content as a placeholder so the conversation structure stays intact. Otherwise the fix is solid and the InMemorySpanExporter tests are the right way to catch this class of bug. |
What does this PR do?
Fixes a bug in the OpenAI instrumentor where multi-turn conversations lost every message after the first one in the trace.
Why?
_process_input_datain_span_io_handler.pysetINPUT_VALUEon the span three times. The first write was correct, a JSON array of every message in the conversation. The next two writes overwrote it: first with a joined string of all message text, then with justeval_input[0], the text of the first message only. Any conversation with more than one message lost everything past turn one in the trace. There's noEVAL_INPUTattribute defined infi_types.py, so theeval_inputaccumulation had no other consumer, it was just clobbering good data. Fixes #151.How was it tested?
pytest)ruff checkpasses on the files I changedmypywas not run cleanly on this branch (pre-existing errors in unrelated files, unrelated to this fix)Added
tests/test_span_io_handler.pywith two cases: a plain multi-turn conversation, and a multimodal message list with text and an image. Both fail onmain(the multi-turn case raises aJSONDecodeErrorsinceINPUT_VALUEends up as raw text instead of JSON) and pass with the fix.Checklist
mainNotes for reviewers
I didn't touch the multimodal branch's image handling, only removed the three-way overwrite on
INPUT_VALUE. Happy to add a changelog entry if the project wants one for this package.