Ladybug multi-file storage feature

[ericyuanhui]

Ladybug currently stores data in a single file. I noticed that the codebase already supports partitioning, yet it does not support splitting storage into separate files by partition. I’m wondering whether the team has considered enabling multi-file storage for Ladybug — for example, splitting files by table or enforcing a fixed file size threshold.

The benefits are as follows:

1. For large graph scenarios (SF300 and above), a single database file can reach 500GB or even 1TB. Operations such as opening and modifying such large files incur substantial overhead. Splitting into multiple files can mitigate this cost.
2. With separate files, the engine could load only required files on writes or queries, which would help bound memory consumption.

Is this direction under consideration? Does it have design merit?

@adsharma

[adsharma]

yet it does not support splitting storage into separate files by partition

If you run the example here, you get something like:

foo.lbdb
foo.Orders_p0.lbdb
foo.Orders_p1.lbdb
foo.Orders_p2.lbdb
foo.Orders_p3.lbdb

[ericyuanhui]

Right. I tried it yesterday and realized my understanding was incorrect. Partitioned tables get split into multiple files, but this capability only applies to node tables; relation tables do not yet support this. There is also the same underlying issue: if I configure a fixed number of partitions, for example four partitions, as data volume grows, the database file under a single partition can still become quite large. Would it be more reasonable to automatically split storage into a new file and start writing to it once the file reaches a certain size threshold, such as 1 GB, or after the table accumulates a specific number of rows?

[adsharma]

The partition syntax supports 3 types: HASH, RANGE and LIST. Only HASH and LIST are implemented.

RANGE was designed to be exactly the way you describe. Happy to collaborate on implementing it.

Not partitioning REL tables is also a missing feature. I think we should split them into intra-partition REL table and a global REL table. The intra-partition REL table would stay within the _pN.lbdb file. When partitions are load balanced, we simply copy the file.

Perhaps the global REL table becomes a bottleneck then. Happy to discuss any ideas on how to partition them.

[ericyuanhui]

Perfect, many of these points match my ideas. I will evaluate proper solutions based on your approach. Let’s keep in contact and work together on this.