Skip to content

Commit 62c4fbb

Browse files
authored
[docs] Update Flink/Spark/Kafka connector versions and release notes (#3550)
## Versions - [x] dev - [x] 4.x - [ ] 3.x - [ ] 2.1 ## Languages - [x] Chinese - [x] English
1 parent b41f313 commit 62c4fbb

File tree

25 files changed

+340
-22
lines changed

25 files changed

+340
-22
lines changed

docs/ecosystem/doris-kafka-connector/doris-kafka-connector.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ The Doris community provides the [doris-kafka-connector](https://github.com/apac
1818
| 1.1.0 | 2.4+ | 2.0+ | 8 |
1919
| 24.0.0 | 2.4+ | 2.0+ | 8 |
2020
| 25.0.0 | 2.4+ | 2.0+ | 8 |
21+
| 26.0.0 | 2.4+ | 2.0+ | 8 |
2122

2223
## Usage
2324

docs/ecosystem/doris-kafka-connector/release-notes.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,24 @@
77

88
# Doris Kafka Connector Release Notes
99

10+
## 26.0.0
11+
12+
### Features & Improvements
13+
14+
- Enable gzip compression by default for StreamLoad [#95](https://github.com/apache/doris-kafka-connector/pull/95)
15+
- Fix JDBC connection surge caused by improper connection reuse in schema evolution [#94](https://github.com/apache/doris-kafka-connector/pull/94)
16+
- Add Confluent build scripts [#86](https://github.com/apache/doris-kafka-connector/pull/86)
17+
- Update config for ConfigDef [#85](https://github.com/apache/doris-kafka-connector/pull/85)
18+
19+
### Bug Fixes
20+
21+
- Fix the check of 'topic2table' configuration [#83](https://github.com/apache/doris-kafka-connector/pull/83)
22+
- Fix the link to the website doc in the readme [#84](https://github.com/apache/doris-kafka-connector/pull/84)
23+
24+
### Credits
25+
26+
@JNSimba @rnb-tron
27+
1028
## 25.0.0
1129

1230
### Features & Improvements

docs/ecosystem/flink-doris-connector/flink-doris-connector.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Using the Flink Connector, you can perform the following operations:
3434
| 25.0.0 | 1.15 - 1.20 | 1.0+ | 8 | - |
3535
| 25.1.0 | 1.15 - 1.20 | 1.0+ | 8 | - |
3636
| 26.0.0 | 1.15 - 1.20,2.0 - 2.2 | 1.0+ | 8(1.x),17(2.x) | - |
37+
| 26.1.0 | 1.15 - 1.20,2.0 - 2.2 | 1.0+ | 8(1.x),17(2.x) | - |
3738

3839
## Usage
3940

@@ -832,9 +833,9 @@ After starting the Flink cluster, you can directly run the following command:
832833
| Key | Default Value | Required | Comment |
833834
| --------------------------- | ------------- | -------- | ------------------------------------------------------------ |
834835
| sink.label-prefix | -- | Y | The label prefix used for Stream load import. In the 2pc scenario, it is required to be globally unique to ensure the EOS semantics of Flink. |
835-
| sink.properties.* | -- | N | Import parameters for Stream Load. For example, 'sink.properties.column_separator' = ', ' defines the column separator, and 'sink.properties.escape_delimiters' = 'true' means that special characters as delimiters, like \x01, will be converted to binary 0x01. For JSON format import, 'sink.properties.format' = 'json', 'sink.properties.read_json_by_line' = 'true'. For detailed parameters, refer to [here](../data-operate/import/import-way/stream-load-manual.md#load-configuration-parameters). For Group Commit mode, for example, 'sink.properties.group_commit' = 'sync_mode' sets the group commit to synchronous mode. The Flink connector has supported import configuration group commit since version 1.6.2. For detailed usage and limitations, refer to [group commit](../data-operate/import/group-commit-manual.md). |
836+
| sink.properties.* | -- | N | Import parameters for Stream Load. For example, 'sink.properties.column_separator' = ', ' defines the column separator, and 'sink.properties.escape_delimiters' = 'true' means that special characters as delimiters, like \x01, will be converted to binary 0x01. For JSON format import, 'sink.properties.format' = 'json', 'sink.properties.read_json_by_line' = 'true'. For detailed parameters, refer to [here](../../data-operate/import/import-way/stream-load-manual.md#load-configuration-parameters). For Group Commit mode, for example, 'sink.properties.group_commit' = 'sync_mode' sets the group commit to synchronous mode. The Flink connector has supported import configuration group commit since version 1.6.2. For detailed usage and limitations, refer to [group commit](../../data-operate/import/group-commit-manual.md). |
836837
| sink.enable-delete | TRUE | N | Whether to enable deletion. This option requires the Doris table to have the batch deletion feature enabled (enabled by default in Doris 0.15+ versions), and only supports the Unique model. |
837-
| sink.enable-2pc | TRUE | N | Whether to enable two-phase commit (2pc). The default is true, ensuring Exactly-Once semantics. For details about two-phase commit, refer to [here](../data-operate/transaction.md#streamload-2pc). |
838+
| sink.enable-2pc | TRUE | N | Whether to enable two-phase commit (2pc). The default is true, ensuring Exactly-Once semantics. For details about two-phase commit, refer to [here](../../data-operate/transaction.md#streamload-2pc). |
838839
| sink.buffer-size | 1MB | N | The size of the write data cache buffer, in bytes. It is not recommended to modify it, and the default configuration can be used. |
839840
| sink.buffer-count | 3 | N | The number of write data cache buffers. It is not recommended to modify it, and the default configuration can be used. |
840841
| sink.max-retries | 3 | N | The maximum number of retries after a Commit failure. The default is 3 times. |
@@ -1129,7 +1130,7 @@ In the whole database synchronization tool provided by the Connector, no additio
11291130
11301131
3. **errCode = 2, detailMessage = current running txns on db 10006 is 100, larger than limit 100**
11311132
1132-
This is because the concurrent imports into the same database exceed 100. It can be solved by adjusting the parameter `max_running_txn_num_per_db` in `fe.conf`. For specific details, please refer to [max_running_txn_num_per_db](../admin-manual/config/fe-config#max_running_txn_num_per_db).
1133+
This is because the concurrent imports into the same database exceed 100. It can be solved by adjusting the parameter `max_running_txn_num_per_db` in `fe.conf`. For specific details, please refer to [max_running_txn_num_per_db](../../admin-manual/config/fe-config#max_running_txn_num_per_db).
11331134
11341135
Meanwhile, frequently modifying the label and restarting a task may also lead to this error. In the 2pc scenario (for Duplicate/Aggregate models), the label of each task needs to be unique. And when restarting from a checkpoint, the Flink task will actively abort the transactions that have been pre-committed successfully but not yet committed. Frequent label modifications and restarts will result in a large number of pre-committed successful transactions that cannot be aborted and thus occupy transactions. In the Unique model, 2pc can also be disabled to achieve idempotent writes.
11351136

docs/ecosystem/flink-doris-connector/release-notes.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,16 @@
77

88
# Flink Doris Connector Release Notes
99

10+
## 26.1.0
11+
12+
### Features & Improvements
13+
14+
- Enable gz compression by default for StreamLoad [#648](https://github.com/apache/doris-flink-connector/pull/648)
15+
16+
### Credits
17+
18+
@JNSimba
19+
1020
## 26.0.0
1121

1222
### Features & Improvements

docs/ecosystem/spark-doris-connector/release-notes.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,22 @@
77

88
# Spark Doris Connector Release Notes
99

10+
## 26.0.0
11+
12+
### Improvements
13+
14+
- Enable gzip compression by default for StreamLoad writes [#356](https://github.com/apache/doris-spark-connector/pull/356)
15+
- Add log for Arrow Flight read [#354](https://github.com/apache/doris-spark-connector/pull/354)
16+
- Support http request use UTF-8 charset [#347](https://github.com/apache/doris-spark-connector/pull/347)
17+
18+
### Bug Fixes
19+
20+
- Fix column projection issue in Spark 3.3, 3.4, and 3.5 [#353](https://github.com/apache/doris-spark-connector/pull/353)
21+
22+
### Credits
23+
24+
@JNSimba @gnehil
25+
1026
## 25.2.0
1127

1228
### Improvements

docs/ecosystem/spark-doris-connector/spark-doris-connector.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Code repository: https://github.com/apache/doris-spark-connector
2020

2121
| Connector | Spark | Doris | Java | Scala |
2222
|-----------|---------------------|-------------|------|------------|
23+
| 26.0.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2324
| 25.2.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2425
| 25.1.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2526
| 25.0.1 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
@@ -416,8 +417,8 @@ Java version examples are provided under `samples/doris-demo/spark-demo/` for re
416417
| doris.sink.label.prefix | spark-doris | Import label prefix when writing in Stream Load mode. |
417418
| doris.thrift.max.message.size | 2147483647 | Maximum message size when reading data through Thrift. |
418419
| doris.fe.auto.fetch | false | Whether to automatically fetch FE information. When set to true, all FE node information will be requested based on the nodes configured in `doris.fenodes`, without needing to configure multiple nodes additionally or separately configure `doris.read.arrow-flight-sql.port` and `doris.query.port`. |
419-
| doris.read.bitmap-to-string | false | Whether to convert Bitmap type to a string composed of array indexes when reading. For specific result format, refer to function definition [BITMAP_TO_STRING](../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-string.md). |
420-
| doris.read.bitmap-to-base64 | false | Whether to convert Bitmap type to Base64 encoded string when reading. For specific result format, refer to function definition [BITMAP_TO_BASE64](../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-base64.md). |
420+
| doris.read.bitmap-to-string | false | Whether to convert Bitmap type to a string composed of array indexes when reading. For specific result format, refer to function definition [BITMAP_TO_STRING](../../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-string.md). |
421+
| doris.read.bitmap-to-base64 | false | Whether to convert Bitmap type to Base64 encoded string when reading. For specific result format, refer to function definition [BITMAP_TO_BASE64](../../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-base64.md). |
421422
| doris.query.port | - | Doris FE query port, used for overwrite write and Catalog metadata retrieval. |
422423

423424
### SQL and Dataframe Specific Configuration

i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector/doris-kafka-connector.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Doris 社区提供了 [doris-kafka-connector](https://github.com/apache/doris-ka
1818
| 1.1.0 | 2.4+ | 2.0+ | 8 |
1919
| 24.0.0 | 2.4+ | 2.0+ | 8 |
2020
| 25.0.0 | 2.4+ | 2.0+ | 8 |
21+
| 26.0.0 | 2.4+ | 2.0+ | 8 |
2122

2223
## 使用方式
2324

@@ -210,7 +211,7 @@ errors.deadletterqueue.topic.replication.factor=1
210211
| jmx | - | true | N | 通过 JMX 获取 Connector 内部监控指标,请参考:[Doris-Connector-JMX](https://github.com/apache/doris-kafka-connector/blob/master/docs/zh-CN/Doris-Connector-JMX.md) |
211212
| label.prefix | - | ${name} | N | Stream load 导入数据时的 label 前缀。默认为 Connector 应用名称。 |
212213
| auto.redirect | - | true | N | 是否重定向 StreamLoad 请求。开启后 StreamLoad 将通过 FE 重定向到需要写入数据的 BE,并且不再显示获取 BE 信息 |
213-
| sink.properties.* | - | `'sink.properties.format':'json'`, <br/>`'sink.properties.read_json_by_line':'true'` | N | Stream Load 的导入参数。<br />例如:定义列分隔符`'sink.properties.column_separator':','` <br />详细参数参考[这里](../data-operate/import/import-way/stream-load-manual.md)。 <br/><br/> **开启 Group Commit**,例如开启 sync_mode 模式的 group commit:`"sink.properties.group_commit":"sync_mode"`。Group Commit 可以配置 `off_mode``sync_mode``async_mode` 三种模式,具体使用参考:[Group-Commit](https://doris.apache.org/docs/data-operate/import/group-commit-manual/) <br/><br/> **开启部分列更新**,例如开启更新指定 col2 的部分列:`"sink.properties.partial_columns":"true"`, `"sink.properties.columns": "col2",` |
214+
| sink.properties.* | - | `'sink.properties.format':'json'`, <br/>`'sink.properties.read_json_by_line':'true'` | N | Stream Load 的导入参数。<br />例如:定义列分隔符`'sink.properties.column_separator':','` <br />详细参数参考[这里](../../data-operate/import/import-way/stream-load-manual.md)。 <br/><br/> **开启 Group Commit**,例如开启 sync_mode 模式的 group commit:`"sink.properties.group_commit":"sync_mode"`。Group Commit 可以配置 `off_mode``sync_mode``async_mode` 三种模式,具体使用参考:[Group-Commit](https://doris.apache.org/docs/data-operate/import/group-commit-manual/) <br/><br/> **开启部分列更新**,例如开启更新指定 col2 的部分列:`"sink.properties.partial_columns":"true"`, `"sink.properties.columns": "col2",` |
214215
| delivery.guarantee | `at_least_once`,<br/> `exactly_once` | at_least_once | N | 消费 Kafka 数据导入至 doris 时,数据一致性的保障方式。支持 `at_least_once` `exactly_once`,默认为 `at_least_once` 。Doris 需要升级至 2.1.0 以上,才能保障数据的 `exactly_once` |
215216
| converter.mode | `normal`,<br/> `debezium_ingestion` | normal | N | 使用 Connector 消费 Kafka 数据时,上游数据的类型转换模式。 <br/> ```normal```表示正常消费 Kafka 中的数据,不经过任何类型转换。 <br/> ```debezium_ingestion```表示当 Kafka 上游的数据通过 Debezium 等 CDC(Changelog Data Capture,变更数据捕获)工具采集时,上游数据需要经过特殊的类型转换才能支持。 |
216217
| debezium.schema.evolution | `none`,<br/> `basic` | none | N | 通过 Debezium 采集上游数据库系统(如 MySQL),发生结构变更时,可以将增加的字段同步到 Doris 中。<br/>`none`表示上游数据库系统发生结构变更时,不同步变更后的结构到 Doris 中。 <br/> `basic`表示同步上游数据库的数据变更操作。由于列结构变更是一个危险操作(可能会导致误删 Doris 表结构的列),目前仅支持同步上游增加列的操作。当列被重命名后,则旧列保持原样,Connector 会在目标表中新增一列,将重命名后的新增数据 Sink 到新列中。 |

i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector/release-notes.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,24 @@
77

88
# Doris Kafka Connector Release Notes
99

10+
## 26.0.0
11+
12+
### 功能与改进
13+
14+
- StreamLoad 默认启用 gzip 压缩 [#95](https://github.com/apache/doris-kafka-connector/pull/95)
15+
- 修复 Schema Evolution 中连接复用不当导致的 JDBC 连接激增问题 [#94](https://github.com/apache/doris-kafka-connector/pull/94)
16+
- 添加 Confluent 构建脚本 [#86](https://github.com/apache/doris-kafka-connector/pull/86)
17+
- 更新 ConfigDef 配置 [#85](https://github.com/apache/doris-kafka-connector/pull/85)
18+
19+
### Bug 修复
20+
21+
- 修复 'topic2table' 配置的检查问题 [#83](https://github.com/apache/doris-kafka-connector/pull/83)
22+
- 修复 README 中网站文档链接 [#84](https://github.com/apache/doris-kafka-connector/pull/84)
23+
24+
### 致谢
25+
26+
@JNSimba @rnb-tron
27+
1028
## 25.0.0
1129

1230
### 功能与改进

0 commit comments

Comments
 (0)