Skip to content

Commit 95f46af

Browse files
dataroaringclaude
andcommitted
Add built-in Streaming Job option for MySQL and PostgreSQL migration
Add Doris-native Streaming Job (CREATE JOB ON STREAMING) as a migration option for both MySQL and PostgreSQL docs (EN + ZH-CN). This provides continuous file-based loading from S3/object storage without external tools like Flink. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 8cedfc3 commit 95f46af

File tree

4 files changed

+60
-4
lines changed

4 files changed

+60
-4
lines changed

docs/migration/mysql-to-doris.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,21 @@ For detailed setup, see the [Flink Doris Connector](../ecosystem/flink-doris-con
7070

7171
The [JDBC Catalog](../lakehouse/catalogs/jdbc-catalog.md) allows direct querying and batch migration from MySQL. This is the simplest approach for one-time or periodic batch migrations.
7272

73-
### Option 3: DataX
73+
### Option 3: Streaming Job (Continuous File Loading)
74+
75+
Doris's built-in [Streaming Job](../data-operate/import/streaming-job.md) (`CREATE JOB ON STREAMING`) provides continuous file-based loading without external tools. Export MySQL data to S3/object storage, and the Streaming Job automatically picks up new files and loads them into Doris.
76+
77+
This option is suited for:
78+
79+
- Continuous incremental migration via file export pipelines
80+
- Environments where you prefer Doris-native features over external tools like Flink
81+
- Scenarios where MySQL data is periodically exported to object storage
82+
83+
**Prerequisites**: Data exported to S3-compatible object storage; Doris 2.1+ with Job Scheduler enabled.
84+
85+
For detailed setup, see the [Streaming Job](../data-operate/import/streaming-job.md) and [CREATE STREAMING JOB](../sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md) documentation.
86+
87+
### Option 4: DataX
7488

7589
[DataX](https://github.com/alibaba/DataX) is a widely-used data synchronization tool that supports MySQL to Doris migration via the `mysqlreader` and `doriswriter` plugins.
7690

docs/migration/postgresql-to-doris.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,21 @@ Flink CDC captures changes from PostgreSQL WAL (Write-Ahead Log) and streams the
6868

6969
For detailed setup, see the [Flink Doris Connector](../ecosystem/flink-doris-connector.md) documentation.
7070

71-
### Option 3: Export and Load
71+
### Option 3: Streaming Job (Continuous File Loading)
72+
73+
Doris's built-in [Streaming Job](../data-operate/import/streaming-job.md) (`CREATE JOB ON STREAMING`) provides continuous file-based loading without external tools. Export PostgreSQL data to S3/object storage, and the Streaming Job automatically picks up new files and loads them into Doris.
74+
75+
This option is suited for:
76+
77+
- Continuous incremental migration via file export pipelines
78+
- Environments where you prefer Doris-native features over external tools like Flink
79+
- Scenarios where PostgreSQL data is periodically exported to object storage
80+
81+
**Prerequisites**: Data exported to S3-compatible object storage; Doris 2.1+ with Job Scheduler enabled.
82+
83+
For detailed setup, see the [Streaming Job](../data-operate/import/streaming-job.md) and [CREATE STREAMING JOB](../sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md) documentation.
84+
85+
### Option 4: Export and Load
7286

7387
For air-gapped environments or when direct connectivity is not possible:
7488

i18n/zh-CN/docusaurus-plugin-content-docs/current/migration/mysql-to-doris.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,21 @@ Flink CDC 捕获 MySQL binlog 变更并流式传输到 Doris。此方法适用
7070

7171
[JDBC Catalog](../lakehouse/catalogs/jdbc-catalog.md) 允许从 MySQL 直接查询和批量迁移。这是一次性或定期批量迁移最简单的方法。
7272

73-
### 选项 3:DataX
73+
### 选项 3:Streaming Job(持续文件加载)
74+
75+
Doris 内置的 [Streaming Job](../data-operate/import/streaming-job.md)`CREATE JOB ON STREAMING`)提供无需外部工具的持续文件加载能力。将 MySQL 数据导出到 S3/对象存储,Streaming Job 会自动发现新文件并加载到 Doris。
76+
77+
此选项适用于:
78+
79+
- 通过文件导出管道进行持续增量迁移
80+
- 偏好使用 Doris 原生功能而非 Flink 等外部工具的环境
81+
- MySQL 数据定期导出到对象存储的场景
82+
83+
**前提条件**:数据已导出到 S3 兼容的对象存储;Doris 2.1+ 并启用 Job Scheduler。
84+
85+
详细设置请参考 [Streaming Job](../data-operate/import/streaming-job.md)[CREATE STREAMING JOB](../sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md) 文档。
86+
87+
### 选项 4:DataX
7488

7589
[DataX](https://github.com/alibaba/DataX) 是一个广泛使用的数据同步工具,通过 `mysqlreader``doriswriter` 插件支持 MySQL 到 Doris 的迁移。
7690

i18n/zh-CN/docusaurus-plugin-content-docs/current/migration/postgresql-to-doris.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,21 @@ Flink CDC 从 PostgreSQL WAL(预写日志)捕获变更并实时流式传输
6868

6969
详细设置请参考 [Flink Doris Connector](../ecosystem/flink-doris-connector.md) 文档。
7070

71-
### 选项 3:导出和加载
71+
### 选项 3:Streaming Job(持续文件加载)
72+
73+
Doris 内置的 [Streaming Job](../data-operate/import/streaming-job.md)`CREATE JOB ON STREAMING`)提供无需外部工具的持续文件加载能力。将 PostgreSQL 数据导出到 S3/对象存储,Streaming Job 会自动发现新文件并加载到 Doris。
74+
75+
此选项适用于:
76+
77+
- 通过文件导出管道进行持续增量迁移
78+
- 偏好使用 Doris 原生功能而非 Flink 等外部工具的环境
79+
- PostgreSQL 数据定期导出到对象存储的场景
80+
81+
**前提条件**:数据已导出到 S3 兼容的对象存储;Doris 2.1+ 并启用 Job Scheduler。
82+
83+
详细设置请参考 [Streaming Job](../data-operate/import/streaming-job.md)[CREATE STREAMING JOB](../sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md) 文档。
84+
85+
### 选项 4:导出和加载
7286

7387
适用于网络隔离环境或无法直接连接的情况:
7488

0 commit comments

Comments
 (0)