Skip to content

DocumentInserter drops whole task batches when a payload key contains a dot #352

Description

@JaimeCernuda

Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:

pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
           Consider using $getField or $setField."

Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions