LadybugDB write throughput

[adsharma]

#301, #302 and #306 have previous discussion

But previous threads primarily focus on write throughput under:

• batched vs non-batched
• CREATE vs MERGE

But all the benchmarks are single threaded. They don't discuss concurrency, parallelism and locking.

[loganpowell]

@adsharma, I figured you might be interested in the states of the other projects. I my use of lbug doesn't require concurrent writes ATM. Thank you for the pull and maintaining this awesome project!

[adsharma]

I'm aware and not opposed to taking code from license compatible projects. We did take some fixes from ryugraph.io when they were around.

Trying to be mindful of the priorities of our user base.

[loganpowell]

@adsharma, you'll notice a new PR that folds in the concurrent writes (safe) support, but it's quite a lot. I added some benchmarking into it and included a run in the comment for the PR. There's a slight hit to the TPS for Node updates, but everything else is pretty much the same. This is a large change for not a great deal of benefit if you don't care about concurrent write safety, so I'll completely understand if you don't pull it in.

Edit:
One perceivable benefit would be to merge the functionality of the two repos and potentially pull the Vela team in on this one to reduce duplication of maintenance effort.

[adsharma]

@GlitterKill this is where we stand now:

#301 (comment)

I suspect we're doing many fsyncs in the commit path leaning towards data integrity vs performance. Bridging the gap against DuckDB will help single threaded write throughput.

[aikins01]

Following up on Logan's note about folding the Vela work in rather than running two forks. I'm the author of the concurrent multi-writer changes that landed in #371, and I'd rather contribute what we've built since than keep maintaining a separate engine.

What we have on top of what was ported:

• checkpoint crash recovery hardening: durable directory entries for WAL and shadow files, commit publication only after the WAL sync, replay of partial checkpoint installs, and fencing the database for restart when a rotated checkpoint fails partway (storage: harden concurrent checkpoint recovery Vela-Engineering/kuzu#18)
• a process-kill test matrix that crashes the checkpoint at each durable boundary in frozen and active WAL modes and checks row values, rollback visibility, and artifact cleanup on reopen
• planned next: filesystem-level fault injection with LazyFS (torn and lost unsynced writes at fsync boundaries) and an Elle list-append workload to get a checked isolation verdict for the multi-writer path instead of an MVCC claim

Two questions before I start porting anything to the lbug tree:

1. Is the checkpoint recovery work plus the crash test matrix something you'd take, given that most of it applies whether or not enableMultiWrites is on?
2. Is there a path for multi-writer to move from the debug setting to a supported option once the isolation and crash tests are in, or would you rather it stay behind the flag?

Happy to split this into small PRs in whatever order is easiest to review.

[adsharma]

@aikins01 thank you for offering. The process-kill test matrix sounds interesting.

There have been a few commits in the last week or so hardening the partial checkpoints. Please review and send anything that's not covered.

Going forward: yes, would love to enable multi writes by default so it doesn't stay behind a feature gate.

Also we just passed UBSAN and TSAN for the first time today! Better positioned than ever to enable more concurrency in the system.

[aikins01]

That's exciting news 🔥

[loganpowell]

@aikins01 welcome aboard!

[aikins01]

Thanks Logan, happy to be aboard!