add support to combine logger kafka and kafkaproducer to one line to …

[lstruman]

…reduce logs

[Copilot]

🟡 Changes recommended

Unresolved API compatibility and log-field preservation issues remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds configurable Kafka logging modes to reduce duplicate logs while bounding payloads and preserving useful fields.

Changes:

• Adds one-line/two-line logging configuration.
• Consolidates and filters Kafka logging output.
• Adds tests, documentation, and dependency updates.

File summaries

File	Summary
README.md	Documents logging configuration.
kafka/consumer.go	Implements consolidated Kafka logging and payload bounds.
kafka/consumer_group_test.go	Tests Kafka logging helpers.
http/webconfig_server.go	Adds logging configuration, filtering, and producer logging.
http/webconfig_server_test.go	Tests logging behavior and payload handling.
go.sum	Updates dependency checksums.
go.mod	Updates dependencies.
config/sample_webconfig.conf	Adds sample logging configuration.

Review details

Suppressed comments (3)

http/webconfig_server.go:1279

• Removing output_body here leaves every correction sent by this loop with the same output_key (the MAC), so messages for different namespaces cannot be distinguished in the logs. Include the current message namespace, or another bounded per-message identifier, before logging each send.

		tfields["output_key"] = mac
		log.WithFields(tfields).Info("send")

http/webconfig_server.go:1222

• In one_line mode this is the replacement for the consumer success record, but the consumer always populates kafka_lag and handleGetMessage populates status; neither field is allowlisted here. The consolidated log therefore silently loses Kafka lag and HTTP result status. Preserve these fields (and update the capacity hint) so one-line mode retains the useful consumer telemetry.

		"app_name", "audit_id", "body", "cpe_mac", "subdoc_id",
		"kafka_key", "topic", "cluster_name", "kafka_partition", "kafka_offset",
		"message_length", "duration", "event_name", "rpt", "kafka_operation",

kafka/consumer.go:326

• After output_body was removed from the producer log, these state-correction records no longer identify the subdocument being sent: fields still describes the original notification and the Kafka key is only the device key. When multiple updatedSubdocIds are corrected, their enqueue logs are indistinguishable; add the current subdocId to the fields before forwarding.

							fields["kafka_operation"] = "state_correction_send"
							c.ForwardKafkaMessage(message.Key, em, fields, logMessage)

• Files reviewed: 7/8 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.