Repository navigation
Bug: Large bulk ingest remains impractically slow on 0.16.1 for a 10M-node / 77.79M-edge graph #474
Description
Ladybug version
v0.16.1
What operating system are you using?
Ubuntu 22.04
What happened?
Summary
Large bulk ingest is still very slow for our workload even when using bulk COPY-style loading through the Python API, rather than single-row transactional inserts.
I know there is already related context in:
- DuckDB is 2x faster than LadybugDB at non-batched inserts #302
- Enhancement: native bulk-load / fast-ingest mode for replay scenarios #386
But this report is specifically about large-scale bulk load performance, not per-row or non-batched insert overhead.
Environment
- LadybugDB version:
0.16.1 - Python version:
3.12.13 - Platform:
Linux x86_64 - Binding: Python
- Workload runner: custom benchmark harness using Ladybug as an on-disk graph store
What we are doing
This is not single-row insert mode.
We:
- Create schema up front.
- Bulk load node tables from CSV using
COPY ... FROM. - Bulk load relationship tables from CSV using
COPY ... FROM. - Use an on-disk database.
- Do not create extra secondary indexes for this benchmark.
In our harness, node tables are copied directly from CSV, and relationship CSVs are rewritten into chunks and then loaded via COPY.
Dataset shape
Synthetic property graph:
- 10 node labels
- 10 relationship types
- 10,000,000 nodes
- 77,790,000 edges
- skewed degree profile
- node properties:
- 8 extra text
- 18 extra numeric
- 8 extra boolean
- edge properties:
- 4 extra text
- 10 extra numeric
- 4 extra boolean
Observed behavior
Large ingest takes roughly 19 to 20 hours before analytical queries even begin.
Representative ingest times from repeated runs:
68,619,567 ms(~19.06 h)69,380,471 ms(~19.27 h)72,656,042 ms(~20.18 h)
So in practice, just preparing the large dataset is already prohibitively expensive for benchmarking and evaluation workflows.
Why this seems distinct from #302
Issue #302 discusses poor performance for non-batched / transactional single-row inserts.
Our case is different:
- we are not inserting row-by-row
- we are doing large CSV-based bulk load
- the database still takes around 19 to 20 hours to ingest this graph
That makes it feel like there may still be a separate bottleneck in the current bulk-load path at larger scales.
Minimal reproduction shape
The exact harness is custom, but the important part is:
- create 10 node tables and 10 relationship tables
- bulk import ~10M nodes and ~77.79M relationships from CSV
- use Python API with on-disk database
- large skewed graph with many properties per record
If useful, I can provide a more minimal standalone repro script that just focuses on schema creation + CSV COPY load without the rest of the benchmark machinery.
Expected behavior
Bulk ingest should be substantially faster for this scale, or at least there should be a documented fast-ingest path that makes this practical.
Actual behavior
Large bulk ingest completes, but takes around 19 to 20 hours, which is too slow for practical large-scale benchmark preparation.
Additional note
I am intentionally keeping this issue focused on ingest time only.
In separate large runs, we also saw later query-execution instability / process kills during heavy OLAP traversal queries, but I do not want to mix that into this report unless you think the two are likely connected.
Are there known steps to reproduce?
No response
Activity
adsharma commentedon May 10, 2026
I think you're running into #97
Suggest trying the unreleased main branch and then:
adsharma commentedon May 13, 2026
@tae898 would love to see numbers before and after
enable_default_hash_index=falseadsharma commentedon May 13, 2026
#487
This was a 4M node, 35M edges graph. Or roughly half the size of your graph. Imported in < 10 secs.
tae898 commentedon May 13, 2026
Do i have to run the main branch or is it already pushed to pypi?
adsharma commentedon May 13, 2026
They get pushed on a nightly basis:
https://test.pypi.org/project/ladybug/#history
Once #487 lands, it should appear in the next nightly build.
ericyuanhui commentedon Jun 1, 2026
Does it only import slowly on tag v0.16.1? If the tag v0.15 is faster?
aheev commentedon Jun 1, 2026
try the latest release 0.17.0
ericyuanhui commentedon Jun 1, 2026
Here is my scenario:
I have two node tables, Assignment and Section, as well as one relationship table Assignment_belongsToSection_Section that defines a many-to-many relationship from Assignment to Section. The table creation statements are attached below.
CREATE NODE TABLE IF NOT EXISTS Assignment (id STRING,fullScore DOUBLE,deadline STRING,assignmentId INT64,name STRING,description STRING,title STRING,publishTime STRING, PRIMARY KEY(id)); CREATE NODE TABLE IF NOT EXISTS Section (id STRING,year INT64,semesterType STRING,enrolledCount INT64,name STRING,classTime STRING,classroom STRING,capacity INT64,sectionId INT64,semester STRING, PRIMARY KEY(id)); CREATE REL TABLE IF NOT EXISTS Assignment_belongsToSection_Section(FROMAssignmentTOSection, MANY_MANY);
Test Data & Scenarios
Each node table contains 20,000 records with a 10% duplicate rate, and the relationship table has 100,000 records, also with a 10% duplicate rate. I tested two data write modes: MERGE-based write and UNWIND-based write.
MERGE write: We use Python to generate 5,000 standalone MERGE statements and combine them into one Cypher string. This string is sent to Ladybug via Python connection APIs. Each batch contains 5,000 Cypher statements, corresponding to 5,000 rows of business data.
UNWIND + MERGE write: We assemble 5,000 rows of data into a single UNWIND statement for ingestion, which is also submitted through the Python API. Each UNWIND statement processes 5,000 data rows. In addition, UNWIND-based writing is already fully supported for relationship tables in our environment.
During actual testing on Ladybug v0.17.0 running on a 6-core 8GB server, the overall write performance is far from ideal.
merge test:
[head] method: merge batch_size: 5000 [node] table=Assignment batch=1 rows=5000 time=19.07326585s 262.15 rows/s [node] table=Assignment batch=2 rows=5000 time=18.36600639s 272.24 rows/s [node] table=Assignment batch=3 rows=5000 time=18.69019473s 267.52 rows/s [node] table=Assignment batch=4 rows=5000 time=18.29015380s 273.37 rows/s [node] table=Section batch=1 rows=5000 time=19.32236782s 258.77 rows/s [node] table=Section batch=2 rows=5000 time=20.64565340s 242.18 rows/s [node] table=Section batch=3 rows=5000 time=19.06360206s 262.28 rows/s [node] table=Section batch=4 rows=5000 time=19.82705517s 252.18 rows/s [rel] table=Assignment_belongsToSection_Section batch=1 rows=5000 time=49.52028219s 100.97 rows/s [rel] table=Assignment_belongsToSection_Section batch=2 rows=5000 time=44.66902757s 111.93 rows/s [rel] table=Assignment_belongsToSection_Section batch=3 rows=5000 time=44.56238204s 112.20 rows/s [rel] table=Assignment_belongsToSection_Section batch=4 rows=5000 time=47.50079619s 105.26 rows/s [rel] table=Assignment_belongsToSection_Section batch=5 rows=5000 time=46.87614230s 106.66 rows/s [rel] table=Assignment_belongsToSection_Section batch=6 rows=5000 time=47.89254662s 104.40 rows/s [rel] table=Assignment_belongsToSection_Section batch=7 rows=5000 time=47.09325283s 106.17 rows/s [rel] table=Assignment_belongsToSection_Section batch=8 rows=5000 time=47.27940555s 105.75 rows/s [rel] table=Assignment_belongsToSection_Section batch=9 rows=5000 time=49.05640991s 101.92 rows/s [rel] table=Assignment_belongsToSection_Section batch=10 rows=5000 time=49.20198255s 101.62 rows/s [rel] table=Assignment_belongsToSection_Section batch=11 rows=5000 time=49.93776052s 100.12 rows/s [rel] table=Assignment_belongsToSection_Section batch=12 rows=5000 time=49.71326444s 100.58 rows/s [rel] table=Assignment_belongsToSection_Section batch=13 rows=5000 time=52.03390523s 96.09 rows/s [rel] table=Assignment_belongsToSection_Section batch=14 rows=5000 time=53.60064071s 93.28 rows/s [rel] table=Assignment_belongsToSection_Section batch=15 rows=5000 time=53.29716925s 93.81 rows/s [rel] table=Assignment_belongsToSection_Section batch=16 rows=5000 time=51.97781690s 96.19 rows/s [rel] table=Assignment_belongsToSection_Section batch=17 rows=5000 time=60.17240179s 83.09 rows/s [rel] table=Assignment_belongsToSection_Section batch=18 rows=5000 time=64.09858387s 78.00 rows/s [rel] table=Assignment_belongsToSection_Section batch=19 rows=5000 time=63.01552514s 79.35 rows/s [rel] table=Assignment_belongsToSection_Section batch=20 rows=5000 time=59.42925595s 84.13 rows/s [summary] node_time_total=153.65984065s node_time_batch_sum=153.27829921s [summary] rel_time_total=1031.20340381s rel_time_batch_sum=1030.92855154s
unwind+merge test
[head] method: unwind batch_size: 5000 [node] table=Assignment batch=1 rows=5000 time=4.72409149s 1058.40 rows/s [node] table=Assignment batch=2 rows=5000 time=2.53727100s 1970.62 rows/s [node] table=Assignment batch=3 rows=5000 time=2.67977285s 1865.83 rows/s [node] table=Assignment batch=4 rows=5000 time=2.54792320s 1962.38 rows/s [node] table=Section batch=1 rows=5000 time=2.98646828s 1674.22 rows/s [node] table=Section batch=2 rows=5000 time=2.78696789s 1794.06 rows/s [node] table=Section batch=3 rows=5000 time=2.76557526s 1807.94 rows/s [node] table=Section batch=4 rows=5000 time=3.01615717s 1657.74 rows/s [rel] table=Assignment_belongsToSection_Section batch=1 rows=5000 time=13.02391199s 383.91 rows/s [rel] table=Assignment_belongsToSection_Section batch=2 rows=5000 time=13.63313705s 366.75 rows/s [rel] table=Assignment_belongsToSection_Section batch=3 rows=5000 time=14.47836645s 345.34 rows/s [rel] table=Assignment_belongsToSection_Section batch=4 rows=5000 time=14.75132768s 338.95 rows/s [rel] table=Assignment_belongsToSection_Section batch=5 rows=5000 time=12.97021881s 385.50 rows/s [rel] table=Assignment_belongsToSection_Section batch=6 rows=5000 time=12.28236449s 407.09 rows/s [rel] table=Assignment_belongsToSection_Section batch=7 rows=5000 time=13.49660885s 370.46 rows/s [rel] table=Assignment_belongsToSection_Section batch=8 rows=5000 time=13.55030614s 369.00 rows/s [rel] table=Assignment_belongsToSection_Section batch=9 rows=5000 time=12.22348837s 409.05 rows/s [rel] table=Assignment_belongsToSection_Section batch=10 rows=5000 time=12.23894914s 408.53 rows/s [rel] table=Assignment_belongsToSection_Section batch=11 rows=5000 time=13.55364571s 368.90 rows/s [rel] table=Assignment_belongsToSection_Section batch=12 rows=5000 time=13.46036084s 371.46 rows/s [rel] table=Assignment_belongsToSection_Section batch=13 rows=5000 time=12.27963348s 407.18 rows/s [rel] table=Assignment_belongsToSection_Section batch=14 rows=5000 time=12.55063193s 398.39 rows/s [rel] table=Assignment_belongsToSection_Section batch=15 rows=5000 time=13.76511641s 363.24 rows/s [rel] table=Assignment_belongsToSection_Section batch=16 rows=5000 time=13.83650285s 361.36 rows/s [rel] table=Assignment_belongsToSection_Section batch=17 rows=5000 time=12.67642549s 394.43 rows/s [rel] table=Assignment_belongsToSection_Section batch=18 rows=5000 time=12.74800529s 392.22 rows/s [rel] table=Assignment_belongsToSection_Section batch=19 rows=5000 time=13.68031790s 365.49 rows/s [rel] table=Assignment_belongsToSection_Section batch=20 rows=5000 time=13.70387687s 364.86 rows/s [summary] node_time_total=24.45065262s node_time_batch_sum=24.04422714s [summary] rel_time_total=265.11600947s rel_time_batch_sum=264.90319575s
I would like to inquire about feasible optimizations and tuning solutions. Potential optimization directions include table schema adjustment, Ladybug configuration tuning and Cypher statement optimization. Since duplicate data exists in the dataset, we are required to use either MERGE or UNWIND + MERGE syntax for data insertion.
adsharma commentedon Jun 1, 2026
adsharma commentedon Jun 1, 2026
Another short term possibility: ingest to duckdb, dedup, export parquet and ingest that.
ericyuanhui commentedon Jun 15, 2026
I used Ladybug's 'copy from' method to write data. The total data volume is around 2 billion points and 6 billion points, simulating a customer usage scenario of 10 million points per day. Among these, there are 2.5 million points and 7.5 million edges related to the points. Data is written daily, with each batch containing 100,000 records. I'm using the Python API with max_db_size set to 8T. During continuous writing, when I've written 500 million points and 1.5 billion edges, my memory usage has already reached 80GB. Even though I call db.close after writing each day's 10 million records, the memory keeps rising, and write performance drops significantly. Do you have any best practice suggestions? How can I better use Ladybug, and what strategies are there to handle memory spikes? Note that my data isn't copied all at once but is copied daily according to the business scenario, 10 million records each day.
adsharma commentedon Jun 15, 2026
Did you remember to disable hash index?
Can you share
CALL disk_info() return *;From the largest import that succeeded?
ericyuanhui commentedon Jun 16, 2026
I remember you mentioned disabling the hash index, but if I turn it off, how can I still use the hash index for fast queries? Does this disable just turn it off without checking, or does it mean the hash index isn't created at all?
#487
in this PR 487
CALL enable_default_hash_index=false;
CREATE NODE TABLE User(id INT64, PRIMARY KEY (id));
CREATE REL TABLE edges(FROM User TO User) WITH (STORAGE_DIRECTION='fwd');
COPY USER from "/tmp/livejournal/nodes_user.parquet";
COPY edges from "/tmp/livejournal/edges.parquet";
Because in my scenario I'm reading while writing, and not just writing everything and then reading the data, if I disable before creating the table, can the hash index be recreated? In the method like above, what command do I need to send to re-enable the hash index? Or, can I skip this operation, and the table will still use the hash index by default when querying? Does it rebuild the hash index in memory on the first query? Would that be more time-consuming? Kind of like how icebug-disk works? For ladybug, after mounting icebug-disk, the first startup needs to rebuild the hash index, so queries take longer. Am I understanding this correctly?
adsharma commentedon Jun 16, 2026
Hash index has a number of problems:
a) It's space inefficient. See https://gist.github.com/adsharma/c2da6591fb0888c1ff3de11c08db63a6
b) It's possible to efficiently lookup primary keys using zonemaps like duckdb does: https://duckdb.org/docs/current/sql/indexes
c) If you must have a primary key because zonemap doesn't work for you, consider ART index. The gist above gives you some estimates.
Nothing is going to auto create hash indexes. Monitor query performance to see if indexes are needed in the first place.
adsharma commentedon Jun 16, 2026
The reason you're seeing 80GB of memory is likely because you're using a giant transaction which has all the rows.
Try something like:
ericyuanhui commentedon Jun 16, 2026
actually I use it like this:
`import ladybug as lb
db = lb.Database("mydb.lbug")
conn = lb.Connection(db)
init_str = """
CALL enable_default_hash_index=false;
CREATE NODE TABLE Item (
id SERIAL PRIMARY KEY,
name STRING,
value INT64
);
"""
conn.execute(init_str)
loop cycle: {
get_csv(path)
conn.execute(f'COPY Item(name, value) FROM "{csv_path}"')
}
db.close
`
What I want to understand is, if I use this statement CALL enable_default_hash_index=false;, then my node table won't have a hash index, right? So if I want to improve my query performance, what should I do? How can I create other indexes? Another thing is, it seems that this statement CALL enable_default_hash_index=false; is only suitable for importing data into a newly created empty table. If it’s like my case, where I import a new CSV file every minute into this table, but this time the table already has data and is no longer empty, can I still import it successfully?
aheev commentedon Jun 16, 2026
Yes. You won't have an index on node tables created from that point on. And no need to run
CALL enable_default_hash_index=falseon every run, it is saved in db filemydb.lbug.There's a smaller ART index you can create if you really think query performance is bad. Indexes updated docs -> https://docs.ladybugdb.com/cypher/indexes/
Yes. Take a look at https://docs.ladybugdb.com/import/csv/
Btw how large are the CSV files?. Also this
adsharma commentedon Jun 16, 2026
Some generic advice from Gemini
Indexing Strategies for Columnar Tables with Integer Keys
In a columnar table, a traditional hash index is rarely optimal. You should instead rely on the storage engine's native sort keys, zone maps, or compression algorithms for maximum performance.
Columnar databases (like ClickHouse, Snowflake, or Amazon Redshift) process data in blocks of columns rather than row-by-row, which fundamentally changes how indexes operate. [1, 2]
Why Hash Indexes Struggle on Columnar Integer Keys
1, 2, 3...), the columnar storage block layout already acts as an implicit index. The system can mathematically calculate the exact block position without any index lookup overhead.Recommended Alternatives for Columnar Integer Keys
Because columnar tables handle data differently, you should use architectural indexing rather than secondary index structures:
Summary of the Best Approach
If your integer key is the primary identifier used to filter queries, you should set it as the Sort Key or Primary Key during table creation. This forces physical ordering on disk, giving you the near-instant point-lookup speed of a hash index while preserving the blistering range-scan capabilities native to columnar architecture. [5]
adsharma commentedon Jun 17, 2026
If you look at how postgres optimizes the bulk COPY operation:
We are different of course because of the columnar table org and CSR based REL tables. But many things are common.
Note that ART indexes are optimized with respect to WAL (they don't write every index mutation to WAL), but not optimized with respect to amount of memory used. You'll need a large (64GB) machine to create an index on a "name" column on billion rows. Memory size scales linearly with table size.
ericyuanhui commentedon Jun 17, 2026
The total number of rows in the overall CSV data is 2 billion points and 10 billion edges, similar to LDBC's SF1000. But the node CSV isn't a single file, and the rel CSV isn't a single file either. The data will be split into batches of 100k per CSV file, so there will be a lot of CSV files.