Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:
pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
Consider using $getField or $setField."
Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.
Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named
yr.doy. Mongo rejects.(and leading$) in field paths, and sincemongodb_dao.insert_and_update_many_taskswrites tasks via$setupserts in abulk_write, the whole batch raisesBulkWriteError(code 16412) and every task in it is lost:Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping
./$in payload keys before emission.