Repository navigation
Discuss potential integration with the latest MemTable and WAL #35
Description
We just dded MemTable and WAL in lance: https://lance.org/format/table/mem_wal/
I am wondering if it could be beneficial for lance context to use that instead of writing to a Lance table directly, because context is mostly for a single agent which means a single writer (correct me if I am wrong), and doing many small writes is not good for performance.
With MemTable & WAL, you don't need to do small writes and then run compaction. Data is accumulated in memory, and then flushed to disk/S3 when in reasonable size. It can also build indexes while writing, so you don't have to asynchronously build index.
You lose a bit of the time travel capability, but if your record already has a timestamp per row, it should be able to achieve similar effect.
Activity
beinan commentedon Feb 4, 2026
This is a fantastic suggestion and aligns perfectly with the goals for lance-context.
You're absolutely right about the access pattern—agent context is almost always a single-writer scenario per session. The current overhead of frequent small writes to S3/disk has been a concern for real-time agent responsiveness. Moving to a MemTable + WAL approach solves the 'small file problem' elegantly and, more importantly, allows us to offer immediate consistency for indexing without the compaction tax.
Regarding the loss of native 'Time Travel': for agentic workflows, row-level timestamps are generally more intuitive for 'memory retrieval' anyway, so I don't see that as a dealbreaker.
Let me explore how we can integrate this as the default storage path for active sessions.
@dcfocus what do you think?
jja725 commentedon Feb 5, 2026
This looks a perfect fit! I can try to work on it if you guys are good. I guess branching might be a bit tricky here. Do you think we can have branching for WAL? @jackye1995
dcfocus commentedon Feb 5, 2026
It makes sense to me. For a small write that doesn't meet the "reasonable size", would it trigger a flush write beyond a certain TTL? I assume so.
vicaya commentedon Feb 9, 2026
As long as this doesn't break the existing versioning semantics for existing APIs, MemWAL would be great addition for many low latency scenarios.
beinan commentedon Feb 11, 2026
Add WAL integration #36 @jackye1995 @dcfocus I'm using bot_id + session_id as the region_id for now, but I think we could make it flexible to support varied agent applications or framework in the future
beinan commentedon Jun 8, 2026
#36 merged