Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions src/UserGuide/Master/Tree/User-Manual/Streaming_apache.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows

In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.

To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).

#### **TabletInsertionEvent**
#### **Tablet Write Event (`TabletInsertionEvent`)**

The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.

For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
Since V2.0.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.

For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.

The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.

```java
/** TabletInsertionEvent is used to define the event of data insertion. */
Expand All @@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
}
```

#### **TsFileInsertionEvent**
#### **TsFile Write Event (`TsFileInsertionEvent`)**

The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).

IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.

Expand All @@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc

1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.

2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.

```java
/**
Expand Down
18 changes: 9 additions & 9 deletions src/UserGuide/V1.3.x/User-Manual/Streaming_apache.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows

In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.

To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).

#### **TabletInsertionEvent**
#### **Tablet Write Event (`TabletInsertionEvent`)**

The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.

For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
Since V1.3.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.

For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.

The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.

```java
/** TabletInsertionEvent is used to define the event of data insertion. */
Expand All @@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
}
```

#### **TsFileInsertionEvent**
#### **TsFile Write Event (`TsFileInsertionEvent`)**

The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).

IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.

Expand All @@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc

1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.

2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.

```java
/**
Expand Down
18 changes: 9 additions & 9 deletions src/UserGuide/dev-1.3/User-Manual/Streaming_apache.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows

In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.

To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).

#### **TabletInsertionEvent**
#### **Tablet Write Event (`TabletInsertionEvent`)**

The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.

For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
Since V1.3.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.

For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.

The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.

```java
/** TabletInsertionEvent is used to define the event of data insertion. */
Expand All @@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
}
```

#### **TsFileInsertionEvent**
#### **TsFile Write Event (`TsFileInsertionEvent`)**

The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).

IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.

Expand All @@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc

1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.

2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.

```java
/**
Expand Down
Loading
Loading