Skip to content

export() raises UnicodeEncodeError and interpolate_file() returns a corrupted prompt when the locale is not UTF-8 #437

Description

@ebarkhordar

On a machine whose locale is not UTF-8 (LC_ALL=en_US.ISO-8859-1 here), two things break as soon as the text has non-ASCII in it.

chat.export() raises instead of writing the transcript, for both .md and .html.

interpolate_file() is the quieter one. It returns a corrupted prompt rather than raising, so a UTF-8 prompt file reaches the model as mojibake and nothing says so.

Repro, no API key needed:

import os, tempfile
from chatlas import ChatOpenAI, UserTurn, AssistantTurn, interpolate_file

TEXT = "Wie heißt die Hauptstadt? 中文测试 café"

chat = ChatOpenAI(api_key="not-used", model="gpt-4o")
chat.set_turns([UserTurn(TEXT), AssistantTurn(TEXT)])
try:
    chat.export(os.path.join(tempfile.mkdtemp(), "transcript.md"))
    print("export: ok")
except Exception as e:
    print("export:", type(e).__name__ + ":", e)

p = os.path.join(tempfile.mkdtemp(), "prompt.md")
with open(p, "w", encoding="utf-8") as f:
    f.write(TEXT + " {{ name }}")
print("interpolate_file gives back what the file says:",
      TEXT in interpolate_file(p, variables={"name": "x"}))

Under LC_ALL=en_US.ISO-8859-1:

export: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 35-38: ordinal not in range(256)
interpolate_file gives back what the file says: False

Same script under LC_ALL=en_US.UTF-8:

export: ok
interpolate_file gives back what the file says: True

One note on the second check: do not print the interpolated string to compare it by eye. Under a latin-1 stdout it re-encodes back to the original bytes and looks correct in the terminal, which is how I missed it the first time.

python:3.12-slim, Python 3.12.14, chatlas 0.23.0 from PyPI and 7b3aa56 on main, identical on both.

I only hit this in a container. The case that probably matters is Windows, where cp1252 is the default and a prompt file with an accent or a curly quote in it is completely ordinary. I have not tried it there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions