Pulsar Data Connector docs [CTT-1428] - #2301
TomaszGaweda wants to merge 14 commits into
Conversation
TomaszGaweda
changed the title
|
Preview deployment: https://6aaae84a80d177b744955842--hazelcast-docs-uat.netlify.app |
|
|
||
| [source,java] | ||
| ---- | ||
| Pipeline p = Pipeline.reate(); |
There was a problem hiding this comment.
Pipeline.reate() does not compile - neither builder sets the required connectionSupplier or dataConnectionRef, and the sink ends with a stray backtick.
There was a problem hiding this comment.
I mean, it compiles, but fails at runtime ;p fixed
|
|
||
| [source,sql] | ||
| ---- | ||
| CREATE MAPPING my_topic <1> |
There was a problem hiding this comment.
This CREATE MAPPING omits valueFormat, so KvMetadataResolvers rejects it
| OPTIONS ( | ||
| 'keyFormat'='int', | ||
| 'valueFormat'='varchar', | ||
| 'bootstrap.servers' = '127.0.0.1:9092' |
There was a problem hiding this comment.
This example omits DATA CONNECTION and uses Kafka's bootstrap.servers, which Pulsar ignores - the first read or write reaches Utils.getClient() with neither a supplier nor DataConnectionRef and fails.
| 'keyFormat' = 'java', | ||
| 'keyJavaClass' = 'java.lang.Long', | ||
| 'valueFormat' = 'java', | ||
| 'valueJavaClass' = 'com.example.Person' |
There was a problem hiding this comment.
valueFormat='java' cannot handle com.example.Person - PulsarTable.resolveSchema() only supports the built-in types and throws UnsupportedOperationException for this class. The SQL also misses its closing );
There was a problem hiding this comment.
not sure what you mean by first part. There are tests for that - pulsarAvro is in fact "java" : https://github.com/TomaszGaweda/hazelcast-mono/blob/ed82a215237d14023c96cec39838f56631edbaca/hazelcast/extensions/pulsar/src/test/java/com/hazelcast/jet/pulsar/sql/SqlAvroTest.java#L59
| CREATE MAPPING my_topic <1> | ||
| DATA CONNECTION myPulsar <2> | ||
| OPTIONS ( | ||
| 'preferredLocalParallelism' = '2' <3> |
There was a problem hiding this comment.
PulsarTable never reads preferredLocalParallelism - it always returns LOCAL_PARALLELISM_USE_DEFAULT, so the documented value 2 has no effect. The partition guidance is also Kafka-specific - this connector uses a Shared subscription rather than one-to-one partition assignment. We need to remove this option and guidance or implement them for Pulsar
There was a problem hiding this comment.
at never reading preferredLocalParallelism - gotcha, bug.
at guidance being Kafka specific - no, it's not, Pulsar just moves assignment to the broker, but incorrect number may result in some processors getting more work
| |exactly-once | ||
|
|
||
|
|
||
| |xref:sql:mapping-to-pulsar.adoc[Apache Pulsar] |
There was a problem hiding this comment.
mapping-to-pulsar.adoc is missing from ROOT/nav.adoc, while querying-streams.adoc still says Kafka is the only SQL streaming source - the only links to the new page are in this connector table.
| </hazelcast> | ||
| ---- | ||
| <1> (Required) Broker URL, mandatory property. | ||
| <2> (Optional) Broker HTTP Service - necessary for resource listing. |
There was a problem hiding this comment.
tenant and namespace only filter listResources() - they do not restrict which topics the returned client can read or write. A fully qualified topic outside that scope is still accessible if the broker allows it.
| () -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(), | ||
| () -> Schema.JSON(Event.class), | ||
| Message::getValue).build(); | ||
| StreamSource<Event> source = PulsarSources.pulsarConsumerBuilder(json(Event.class)) |
There was a problem hiding this comment.
Switching to the consumer changes the restart behaviour described below - the current consumer acks messages even with snapshots disabled and reconnects to the same subscription. Restarting does not replay acknowledged history - Earliest only sets the position for a new subscription.
There was a problem hiding this comment.
What changes? I don't see anything in the text below that changes, maybe I am overlooking something obvious after staring at it for too long :D
3 participants
Includes:
I didn't include too much of Java API details, there are easily accessible after the release. Wanted to focus on basic concepts and ideas "why and how".