You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Versions
- [x] dev
- [x] 4.x
- [x] 3.x
- [ ] 2.1
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
Copy file name to clipboardExpand all lines: docs/lakehouse/best-practices/optimization.md
+74Lines changed: 74 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,6 +36,32 @@ Since version 4.0.2, cache warmup functionality is supported, which can further
36
36
37
37
Please refer to the **HDFS IO Optimization** section in the [HDFS Documentation](../storages/hdfs.md).
38
38
39
+
## Split Count Limit
40
+
41
+
When querying external tables (Hive, Iceberg, Paimon, etc.), Doris splits files into multiple splits for parallel processing. In some scenarios, especially when there are a large number of small files, too many splits may be generated, leading to:
42
+
43
+
1. Memory pressure: Too many splits consume a significant amount of FE memory
44
+
2. OOM issues: Excessive split counts may cause OutOfMemoryError
45
+
3. Performance degradation: Managing too many splits increases query planning overhead
46
+
47
+
You can use the `max_file_split_num` session variable to limit the maximum number of splits allowed per table scan (supported since 4.0.4):
48
+
49
+
- Type: `int`
50
+
- Default: `100000`
51
+
- Description: In non-batch mode, the maximum number of splits allowed per table scan to prevent OOM caused by too many splits.
52
+
53
+
Usage example:
54
+
55
+
```sql
56
+
-- Set maximum split count to 50000
57
+
SET max_file_split_num =50000;
58
+
59
+
-- Disable this limit (set to 0 or negative number)
60
+
SET max_file_split_num =0;
61
+
```
62
+
63
+
When this limit is set, Doris dynamically calculates the minimum split size to ensure the split count does not exceed the specified limit.
64
+
39
65
## Merge IO Optimization
40
66
41
67
For remote storage systems like HDFS and object storage, Doris optimizes IO access through Merge IO technology. Merge IO technology essentially merges multiple adjacent small IO requests into one large IO request, which can reduce IOPS and increase IO throughput.
@@ -71,3 +97,51 @@ If you find that `MergedBytes` is much larger than `RequestBytes`, it indicates
71
97
-`merge_io_read_slice_size_bytes`
72
98
73
99
Session variable, supported since version 3.1.3. Default is 8MB. If you find serious read amplification, you can reduce this parameter, such as to 64KB, and observe whether the modified IO requests and query latency improve.
100
+
101
+
## Parquet Page Cache
102
+
103
+
:::info
104
+
Supported since version 4.1.0.
105
+
:::
106
+
107
+
Parquet Page Cache is a page-level caching mechanism for Parquet files. This feature integrates with Doris's existing Page Cache framework, significantly improving query performance by caching decompressed (or compressed) data pages in memory.
108
+
109
+
### Key Features
110
+
111
+
1.**Unified Page Cache Integration**
112
+
- Shares the same underlying `StoragePageCache` framework used by Doris internal tables
113
+
- Shares memory pool and eviction policies
114
+
- Reuses existing cache statistics and RuntimeProfile for unified performance monitoring
115
+
116
+
2.**Intelligent Caching Strategy**
117
+
-**Compression Ratio Awareness**: Automatically decides whether to cache compressed or decompressed data based on the `parquet_page_cache_decompress_threshold` parameter
118
+
-**Flexible Storage Approach**: Caches decompressed data when `decompressed size / compressed size ≤ threshold`; otherwise, decides whether to cache compressed data based on `enable_parquet_cache_compressed_pages`
119
+
-**Cache Key Design**: Uses `file_path::mtime::offset` as the cache key to ensure cache consistency after file modifications
120
+
121
+
### Configuration Parameters
122
+
123
+
The following are BE configuration parameters:
124
+
125
+
-`enable_parquet_page_cache`
126
+
127
+
Whether to enable the Parquet Page Cache feature. Default is `false`.
128
+
129
+
-`parquet_page_cache_decompress_threshold`
130
+
131
+
Threshold that controls whether to cache compressed or decompressed data. Default is `1.5`. When the ratio of `decompressed size / compressed size` is less than or equal to this threshold, decompressed data will be cached; otherwise, it will decide whether to cache compressed data based on the `enable_parquet_cache_compressed_pages` setting.
132
+
133
+
-`enable_parquet_cache_compressed_pages`
134
+
135
+
Whether to cache compressed data pages when the compression ratio exceeds the threshold. Default is `true`.
136
+
137
+
### Performance Monitoring
138
+
139
+
You can view Parquet Page Cache usage through Query Profile:
140
+
141
+
```
142
+
ParquetPageCache:
143
+
- PageCacheHitCount: 1024
144
+
- PageCacheMissCount: 128
145
+
```
146
+
147
+
Where `PageCacheHitCount` indicates the number of cache hits, and `PageCacheMissCount` indicates the number of cache misses.
2. The operation will fail if the specified snapshot does not exist
2134
2134
3. The merge operation creates a new snapshot and does not delete the original snapshot
2135
2135
2136
+
### expire_snapshots
2137
+
2138
+
The `expire_snapshots` operation removes old snapshots from Iceberg tables to free up storage space and improve metadata performance. This operation follows the Apache Iceberg Spark procedure specification.
|`older_than`| String | No | Timestamp threshold for snapshot expiration. Snapshots older than this will be removed. Supports ISO datetime format (e.g., `2024-01-01T00:00:00`) or milliseconds timestamp |
2154
+
|`retain_last`| Integer | No | Number of ancestor snapshots to preserve. When specified alone, automatically sets `older_than` to current time |
2155
+
|`snapshot_ids`| String | No | Comma-separated list of specific snapshot IDs to expire |
2156
+
|`max_concurrent_deletes`| Integer | No | Size of thread pool for delete operations |
2157
+
|`clean_expired_metadata`| Boolean | No | When set to `true`, cleans up unused partition specs and schemas |
2158
+
2159
+
**Return Value:**
2160
+
2161
+
Executing the `expire_snapshots` operation returns a result set with the following 6 columns:
2162
+
2163
+
| Column Name | Type | Description |
2164
+
| ---- | ---- | ---- |
2165
+
|`deleted_data_files_count`| BIGINT | Number of deleted data files |
2166
+
|`deleted_position_delete_files_count`| BIGINT | Number of deleted position delete files |
2167
+
|`deleted_equality_delete_files_count`| BIGINT | Number of deleted equality delete files |
2168
+
|`deleted_manifest_files_count`| BIGINT | Number of deleted manifest files |
2169
+
|`deleted_manifest_lists_count`| BIGINT | Number of deleted manifest list files |
2170
+
|`deleted_statistics_files_count`| BIGINT | Number of deleted statistics files |
2171
+
2172
+
**Example:**
2173
+
2174
+
```sql
2175
+
-- Expire snapshots, keeping only the last 2
2176
+
ALTERTABLEiceberg_db.iceberg_table
2177
+
EXECUTE expire_snapshots ("retain_last"="2");
2178
+
2179
+
-- Expire snapshots older than a specific timestamp
1. This operation does not support WHERE conditions.
2195
+
2. If both `older_than` and `retain_last` are specified, both conditions apply: only snapshots older than `older_than` AND not within the most recent `retain_last` snapshots will be deleted.
2196
+
3.`snapshot_ids` can be used alone to delete specific snapshots.
2197
+
4. This operation permanently deletes snapshots and their associated data files. Use with caution.
2198
+
5. It is recommended to query the `$snapshots` system table before execution to understand the table's snapshot information.
2199
+
2136
2200
### fast_forward
2137
2201
2138
2202
The `fast_forward` operation quickly advances the current snapshot of one branch to the latest snapshot of another branch.
0 commit comments