NMS-6400: OpenNMS syslogd allow receiving messages on TCP (and via TLS) by marshallmassengill · Pull Request #8794 · OpenNMS/opennms

marshallmassengill · 2026-08-17T21:14:23Z

Adds syslog ingestion over TCP, with optional TLS, for both the core and Minions. Off unless configured, so existing installs keep listening on UDP alone.

syslog-tcp-port in syslogd-configuration.xml starts a TCP listener alongside the UDP one. Unset by default.
Both RFC 6587 framings are supported. The default auto reads the framing from the first message of a connection and latches it.
TLS follows RFC 5425, with none, optional or require client authentication. Unusable certificate material stops the listener rather than falling back to plaintext.
A Minion configures the same thing as syslog.tcp.* keys on the syslog listener feature it already installs.
Messages reach the existing sink, one connection's messages in order, with reads paused while a dispatch is outstanding so a slow sink becomes TCP backpressure.
Verified end to end against real rsyslog and syslog-ng: both framings against plaintext, TLS and mutual TLS, on both ingestion paths. New unit tests, integration tests and a Minion smoke test.

This pull request brought to you by the letter N for necromancy and assisted by Anthropic Claude Opus 5.

External References

Jira (Issue Tracker): https://opennms.atlassian.net/browse/NMS-6400

dino2gnt · 2026-08-17T21:18:58Z

👑

You trying to steal my "King Shit of Developing Features No One Asked For" crown ?

Unless there's a really good reason, this needs to target develop, IMO.

Copilot

Pull request overview

Adds optional syslog ingestion over TCP (including TLS/mTLS), extending the existing syslog pipeline (sink + parsers) so operators can enable TCP/TLS without changing downstream processing. This introduces a Netty-based TCP listener, new config surface (core XML + Minion .cfg), and accompanying unit/integration/smoke/e2e tooling/docs.

Changes:

Introduces a Netty TCP syslog listener with RFC 6587 framing support (auto, octet-counting, non-transparent incl. NUL/LF) and backpressure via sink dispatch gating.
Adds TLS support per RFC 5425 (server cert + optional/required client auth) and expands syslogd config model + XSD + docs.
Adds/updates integration tests, smoke tests, and a dev end-to-end matrix environment to validate plaintext/TLS/mTLS across core + Minion ingestion paths.

Reviewed changes

Copilot reviewed 51 out of 51 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
syslog-tcp-status.md	Adds a status/notes markdown file describing current branch state and findings.
smoke-test/src/test/java/org/opennms/smoketest/minion/SyslogTcpIT.java	New smoke test validating Minion TCP syslog ingestion in both framings.
smoke-test/src/main/resources/syslog-tcp/org.opennms.netmgt.syslog.cfg	Smoke-test Minion `.cfg` enabling TCP listener and batching tweaks.
smoke-test/src/main/java/org/opennms/smoketest/utils/SyslogUtils.java	Adds a TCP syslog sender helper for smoke tests (RFC 6587 framing).
smoke-test/src/main/java/org/opennms/smoketest/stacks/NetworkProtocol.java	Adds SYSLOG_TCP protocol enum entry.
smoke-test/src/main/java/org/opennms/smoketest/containers/MinionContainer.java	Exposes Minion TCP syslog port and accessor for tests.
opennms-container/minion/CONFD_README.md	Documents Minion syslog TCP/TLS configuration keys.
opennms-config/src/main/java/org/opennms/netmgt/config/SyslogdConfigFactory.java	Exposes TCP config from syslogd config factory.
opennms-config/src/main/java/org/opennms/netmgt/config/SyslogdConfig.java	Adds `getTcpConfig()` to the syslogd config interface.
opennms-config-model/src/test/java/org/opennms/netmgt/config/syslogd/SyslogTcpConfigTest.java	New unit tests for TCP config defaults/validation.
opennms-config-model/src/test/java/org/opennms/netmgt/config/syslogd/SyslogdConfigurationTest.java	Extends JAXB model serialization tests to include TCP attributes.
opennms-config-model/src/main/resources/xsds/syslog.xsd	Adds syslog TCP/TLS attributes to syslogd XSD.
opennms-config-model/src/main/java/org/opennms/netmgt/config/syslogd/SyslogTcpFraming.java	New enum defining RFC 6587 framing options + parsing.
opennms-config-model/src/main/java/org/opennms/netmgt/config/syslogd/SyslogTcpConfig.java	New POJO aggregating TCP listener settings with validation/defaults.
opennms-config-model/src/main/java/org/opennms/netmgt/config/syslogd/SyslogTcpClientAuth.java	New enum defining TLS client-auth modes + parsing.
opennms-config-model/src/main/java/org/opennms/netmgt/config/syslogd/Configuration.java	Extends JAXB syslogd configuration model with TCP/TLS attributes and `getTcpConfig()`.
opennms-base-assembly/src/main/filtered/etc/syslogd-configuration.xml	Documents TCP/TLS syslog configuration in the shipped config template (commented).
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogTcpSslContextFactoryTest.java	Unit tests for TLS context creation and validation failures.
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogTcpListenerDispatchIT.java	Integration tests pinning dispatcher threading/backpressure/timeout behavior.
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogTcpFrameDecoderTest.java	Unit tests for RFC 6587 framing (LF/NUL, octet counting, limits, errors).
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogdReceiverNettyTcpTlsIT.java	Integration tests exercising TLS + optional/required client auth over real socket.
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogdReceiverNettyTcpIT.java	Integration tests exercising TCP ingestion via real syslogd XML + socket.
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogdMultiListenerIT.java	Integration tests running UDP+TCP together and validating shared dispatcher behavior.
features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogClient.java	Extends test syslog client to support TCP/TLS and RFC 6587 framing.
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogTcpSslContextFactory.java	New TLS context builder with strict validation of cert/key/trust material.
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogTcpListener.java	New Netty TCP listener with framing decode, TLS, connection limits, idle handling, and ordered dispatch/backpressure.
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogTcpFrameDecoder.java	New Netty decoder splitting TCP stream into syslog messages (RFC 6587).
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogTcpExceptionHandler.java	New pipeline handler closing/logging on framing/decode failures.
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogReceiverJavaNetImpl.java	Adjusts stop lifecycle to delegate dispatcher/TCP teardown to base class.
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogConfigBean.java	Adds `SyslogTcpConfig` property support for blueprint wiring (Minion).
features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SinkDispatchingSyslogReceiver.java	Starts/stops the TCP listener alongside the dispatcher lifecycle.
features/events/syslog/pom.xml	Adds direct Netty dependencies required by the TCP listener implementation.
features/events/syslog/blueprint-syslog-listener-javanet.xml	Adds syslog.tcp.* config keys and wires a SyslogTcpConfig bean into SyslogConfigBean.
features/events/syslog/blueprint-syslog-listener-camel-netty.xml	Same as above for the Camel-netty listener blueprint.
docs/modules/reference/pages/daemons/daemon-config-files/syslogd.adoc	Updates daemon config reference to mention TCP listener config surface.
docs/modules/operation/pages/deep-dive/events/sources/syslog.adoc	Adds comprehensive operator documentation for TCP/TLS framing, defaults, and sender examples.
dev/syslog-tcp-test/stage.sh	Adds staging script for local e2e core+minion environment with TCP/TLS configs.
dev/syslog-tcp-test/senders/rsyslog/Dockerfile	Adds rsyslog sender image with TLS support for e2e matrix.
dev/syslog-tcp-test/scripts/verify.sh	Adds event verification script (exact counts + framing sanity checks).
dev/syslog-tcp-test/scripts/start-minion.sh	Adds script to start Minion and wait for TCP listener readiness.
dev/syslog-tcp-test/scripts/start-core.sh	Adds script to initialize/start core and wait for UI readiness.
dev/syslog-tcp-test/scripts/set-mode.sh	Adds script to switch between plaintext/TLS listener modes via reload.
dev/syslog-tcp-test/scripts/send.sh	Adds sender harness (rsyslog/syslog-ng/raw) for matrix cells (plain/tls/mtls).
dev/syslog-tcp-test/scripts/run-matrix.sh	Adds matrix runner covering sender/framing/transport across core+minion.
dev/syslog-tcp-test/scripts/extra-runs.sh	Adds additional long-run/churn/reload/failure-mode checks.
dev/syslog-tcp-test/README.md	Documents the e2e environment and what its assertions catch.
dev/syslog-tcp-test/gen-certs.sh	Adds certificate generation script for TLS/mTLS testing.
dev/syslog-tcp-test/env.sh	Adds shared env vars (ports/counts) for the e2e environment.
dev/syslog-tcp-test/Dockerfile	Adds base image for core/minion containers in the e2e environment.
dev/syslog-tcp-test/docker-compose.yml	Adds compose stack for core+minion+postgres e2e environment.
dev/syslog-tcp-test/.gitignore	Ignores staged assemblies, generated certs, and results output.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

marshallmassengill · 2026-08-17T21:21:27Z

👑

You trying to steal my "King Shit of Developing Features No One Asked For" crown ?

Unless there's a really good reason, this needs to target develop, IMO.

LOL

I can target it at develop. We do have a semi-important request for this but I'm good to put it on develop and backport later. It does need testing.

Eleven optional attributes on the syslogd configuration, all of them off by default: the listen port and address, the RFC 6587 framing, message and connection limits, an idle timeout, and the RFC 5425 TLS material. Leaving syslog-tcp-port unset gives the UDP-only behaviour installs have today. SyslogTcpConfig is the shared shape both SyslogdConfig implementations expose, so the core reaches it through the JAXB model and a Minion through a bean the blueprint populates. Framing and client authentication are strings there, parsed on the way out, because the Minion has no schema to reject a typo for it.

TCP carries no message boundaries, and RFC 6587 defines two incompatible ways to supply them that senders disagree about. The decoder handles both: octet counting, where a length prefix precedes each message, and non-transparent, where a newline or a NUL separates them. Under the default auto it reads the framing from the first frame of a connection and then latches it, since a stream that changes framing cannot be decoded unambiguously and guessing per frame would turn a sender bug into corrupt events. A framing error leaves the position of every later message unknown, so SyslogTcpExceptionHandler drops the connection and lets the sender resynchronise rather than skipping bytes. Netty is used directly instead of through Camel because the Camel netty component has no codec for octet counting.

cgorantla

Would prefer to model this as

<configuration
      listen-address="0.0.0.0"
      syslog-port="514"
      parser="...">

      <tcp port="601"
           listen-address="0.0.0.0"
           framing="auto"
           max-message-size="65536"
           max-connections="1024"
           idle-timeout="0">

          <tls enabled="true"
               cert-filepath="..."
               private-key-filepath="..."
               trust-cert-filepath="..."
               client-auth="require"/>
      </tcp>
  </configuration>

This way tcp remains completely optional and tls is embedded within tcp and can be extended if we need to add more settings.

.cfg can retain flat but should be able to populate the tcp config object.

marshallmassengill · 2026-08-20T21:29:42Z

Would prefer to model this as
<configuration
      listen-address="0.0.0.0"
      syslog-port="514"
      parser="...">

      <tcp port="601"
           listen-address="0.0.0.0"
           framing="auto"
           max-message-size="65536"
           max-connections="1024"
           idle-timeout="0">

          <tls enabled="true"
               cert-filepath="..."
               private-key-filepath="..."
               trust-cert-filepath="..."
               client-auth="require"/>
      </tcp>
  </configuration>
This way tcp remains completely optional and tls is embedded within tcp and can be extended if we need to add more settings.

.cfg can retain flat but should be able to populate the tcp config object.

I attempted to address in latest. Check and let me know if that is what you had in mind.

indigo423

Approach LGTM here a few topics I would take a closer look and this one is especially one I would investigate a bit deeper:

A bad TCP value takes UDP down with it on a Minion.** The Core is protected: SinkDispatchingSyslogReceiver.run() wraps TCP setup in catch (Throwable) so UDP starts regardless. The Minion builds SyslogTcpConfig as a Blueprint bean whose setters throw, and a throwing Blueprint bean fails the whole container which is the one also owns the UDP listener.

With framing set to a value outside the accepted set I got ComponentDefinitionException on setFraming and both listeners bound zero sockets.
When restored it to auto it brought both back. This is the only asymmetry between the two ingestion paths. Seems like it fails in the unsafe direction, and the property placeholder reloads, so it can strike long after a healthy start.

SyslogTcpListener owns the socket and hands decoded messages to a dispatcher that belongs to someone else. It is deliberately not a second SyslogReceiver: a receiver creates its own sink dispatcher, the Sink API names its metrics after the module id, and a second dispatcher for the same module throws and takes its listener down, sometimes the UDP one. SinkDispatchingSyslogReceiver therefore owns one dispatcher and starts a TCP socket alongside its UDP one, which gives both existing receiver implementations TCP without touching Syslogd. Messages from one connection are dispatched one at a time, in arrival order, from a pool thread rather than an event loop: AsyncDispatcher.send() blocks while the sink queue is full and SyslogSinkModule asks for blockWhenFull, so dispatching on an event loop stalls every connection on that worker. Reads are paused until the dispatch returns, which turns a slow sink into TCP backpressure instead of unbounded buffering, and the connection is not treated as idle while that is outstanding. That leaves one message per connection riding on the sink confirming it, so the wait is bounded by dispatch-timeout. Any sink slow enough to leave a future uncompleted for that long would otherwise stall the connection outright, since reads stay off until the dispatch returns. A Minion whose configuration was reloaded is the case that never recovers, because the sink then delivers messages while completing the wrong futures, but a merely busy sink reaches the same bound. Past it the connection keeps delivering and gives up ordering and backpressure for the rest of its life, which it logs, leaving the sink queue's own blockWhenFull as the only thing underneath. TLS follows RFC 5425 and refuses to start on unusable certificate material rather than falling back to plaintext on a port an operator believes is encrypted.

The TCP settings hang off the syslog listener feature a Minion already installs, as syslog.tcp.* keys in org.opennms.netmgt.syslog.cfg, rather than a second feature. syslog.tcp.listen.port defaults to 0, meaning off, because the .cfg always carries the key. The placeholder already reloads, so a change takes effect through config admin without restarting the container.

Commented out, so an install that is not touched keeps listening on UDP alone. The example uses unprivileged ports, since the registered 601 and 6514 need a redirect or added capabilities for a process that is not root.

Covers both framings and what a mismatch looks like from the outside, since that is the failure operators will actually hit, along with the TLS attributes and the Minion keys.

Sends both framings to a Minion and counts the events that reach the core. The count is exact rather than a lower bound, because over-delivery is what a framing bug produces and a greater-than assertion would pass straight over one message becoming several. The .cfg that switches the port on is placed relative to the Minion's etc, since the overlay is rsynced into it rather than into the Minion home.

Review feedback: eleven attributes on the configuration element become an optional tcp element carrying a nested tls one. The presence of the element is now what asks for a listener, so port is required inside it and the "unset means off" convention is gone from the XML. Grouping the TLS settings also means a further one is an attribute on tls rather than another tcp-tls-* attribute on the parent, which is already wide. The .cfg keys a Minion uses are unchanged. Its blueprint builds the nested bean and injects it, which is the same object graph the XML unmarshals into. The tcp element may only appear once, matching what the runtime supports. Raising that later is a schema change plus the work to own more than one listener. Two details worth keeping: The listen-address of the parent is still the fallback for the tcp element, passed to the listener rather than copied into the element, so it does not reappear when the configuration is marshalled. Binding TCP to every address while UDP was restricted to one interface is a poor surprise. The tls enabled attribute is a primitive. A nullable Boolean setter alongside a boolean getter makes the property read-only under bean introspection, which the blueprint rejects, and absence means false either way.

Review feedback. The Core was already protected, since the receiver wraps its TCP setup in a catch. A Minion was not: there SyslogTcpConfig is a Blueprint bean, its setters validated, and a bean that throws fails the container that also owns the UDP listener. A typo in any one of six properties bound zero sockets. The setters now accept anything and SyslogTcpConfig.validate() reports the same problems, called by the listener before it binds. So a bad value leaves TCP down with an error in the log and UDP running, matching the Core. The XML path still fails on load, because the schema rejects those values. Blueprint's own String to int conversion still fails the container on a non-numeric value, which is outside this code. getTcpConfig() is defaulted rather than abstract so an implementation outside this repository keeps compiling, and its javadoc no longer promises non-null while the factory could return null. An oversized frame under non-transparent framing now loses the message rather than the connection: the trailer is already in hand, so the next message starts after it. The octet-counted path still closes, because a length prefix that cannot be trusted leaves the next message nowhere in particular. The wait for a dispatch is configurable as dispatch-timeout rather than a constant only tests could reach, and zero waits indefinitely. Reaching it costs ordering and read backpressure for the life of that connection, so the warning now says so and names the knob. Connections refused at max-connections warn once a minute instead of only at debug, where an operator at the cap saw nothing.

Holding one message per connection at the sink and waiting for it to be confirmed was measured at roughly 53 messages a second per connection against the shipped batch-size, where not waiting measured roughly 21500. A batch that does not fill waits out batch-interval, and every message pays that. What the wait buys is real. Aggregator.aggregate() appends under a non-fair striped lock keyed on the source address, so two of the dispatcher's drain threads can invert a pair whatever order they took them off the queue, and syslog has no sequence number to repair it with. One message in flight keeps a connection out of that race. But that is the same dispatcher, drain pool and aggregator the UDP listener uses, and both transports share one AsyncDispatcher, so syslog over UDP has always had the same exposure and nothing compensates for it. TCP was the only path paying two orders of magnitude for a guarantee nothing else offers. So ordered is now an attribute on the tcp element, off by default, and dispatch-timeout only applies when it is set. Unordered resumes reading when the sink accepts a message and only watches the future to log a failure, which also makes it immune to the reloaded-Minion case where the sink completes the wrong futures. No test can fail for want of ordering, because the race is a lock race that does not reproduce on demand: the two 200 message ordering tests passed in both modes, so they were never protecting it. The mechanism is pinned instead. Ordered never leaves two of a connection's messages unconfirmed at the sink, the default does, and the default carries a burst through a sink that never confirms anything. Also adds the first test to leave batch-size at its default. Every other one sets it to 1, which makes each message its own batch and is why none of this surfaced.

marshallmassengill requested review from cgorantla, christianpape, dino2gnt, indigo423 and synqotik and a lite review from Copilot August 17, 2026 21:14

github-actions Bot added the docs label Aug 17, 2026

Copilot started reviewing on behalf of marshallmassengill August 17, 2026 21:14 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Comment thread features/events/syslog/src/main/java/org/opennms/netmgt/syslogd/SyslogTcpListener.java

Comment thread syslog-tcp-status.md Outdated

Comment thread features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogTcpFrameDecoderTest.java Outdated

marshallmassengill added 2 commits August 18, 2026 08:31

marshallmassengill changed the base branch from release-36.x to develop August 18, 2026 12:46

marshallmassengill force-pushed the mm/NMS-6400-smoke branch from 587adba to 1b05962 Compare August 18, 2026 13:30

cgorantla requested changes Aug 20, 2026

View reviewed changes

marshallmassengill requested a review from cgorantla August 20, 2026 21:29

indigo423 reviewed Aug 26, 2026

View reviewed changes

marshallmassengill added 7 commits August 26, 2026 08:48

NMS-6400: Describe the TCP attributes in the shipped configuration

2b6bcbd

Commented out, so an install that is not touched keeps listening on UDP alone. The example uses unprivileged ports, since the registered 601 and 6514 need a redirect or added capabilities for a process that is not root.

NMS-6400: Document syslog ingestion over TCP and TLS

fb59b84

Covers both framings and what a mismatch looks like from the outside, since that is the failure operators will actually hit, along with the TLS attributes and the Minion keys.

marshallmassengill force-pushed the mm/NMS-6400-smoke branch from 09217fb to ab6ba8c Compare August 26, 2026 12:49

github-actions Bot requested a review from indigo423 August 26, 2026 12:49

Navigation Menu

Uh oh!

NMS-6400: OpenNMS syslogd allow receiving messages on TCP (and via TLS) - #8794

NMS-6400: OpenNMS syslogd allow receiving messages on TCP (and via TLS)#8794
marshallmassengill wants to merge 10 commits into
developfrom
mm/NMS-6400-smoke

marshallmassengill commented Aug 17, 2026

Uh oh!

dino2gnt commented Aug 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marshallmassengill commented Aug 17, 2026

Uh oh!

cgorantla left a comment

Uh oh!

marshallmassengill commented Aug 20, 2026

Uh oh!

indigo423 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

marshallmassengill commented Aug 17, 2026

External References

Uh oh!

dino2gnt commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marshallmassengill commented Aug 17, 2026

Uh oh!

cgorantla left a comment

Choose a reason for hiding this comment

Uh oh!

marshallmassengill commented Aug 20, 2026

Uh oh!

indigo423 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dino2gnt commented Aug 17, 2026 •

edited

Loading