Repository navigation
Add WAL provider support to Drasi.create - #6
Draft
danielgerlag wants to merge 1 commit into
Draft
danielgerlag wants to merge 1 commit into
danielgerlag wants to merge 1 commit into
Conversation
Expose redb WAL configuration across the async, sync, and config APIs, and add durable Python-source replay with fresh-process recovery coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
WAL cleanup fails for durable sources that were not successfully started, and the generated API signature is malformed.
2 open findings
What changed in this PR
Adds engine-level redb WAL support and durable Python-source recovery.
Changes:
- Exposes WAL configuration across async, sync, typed, and config APIs.
- Adds durable Python-source persistence, replay, pruning, and recovery tests.
- Updates documentation, dependencies, and stable error codes.
| File | Description |
|---|---|
Cargo.toml |
Adds redb WAL dependency. |
Cargo.lock |
Locks WAL dependency. |
CHANGELOG.md |
Records durability features. |
README.md |
Adds durable-source example. |
docs/api-audit.md |
Marks WAL support complete. |
python/drasi/_drasi.pyi |
Extends async API types. |
python/drasi/sync.py |
Extends blocking API. |
python/drasi/types.py |
Defines WAL configuration types. |
scripts/generate_api_reference.py |
Documents WAL behavior. |
src/components.rs |
Implements durable Python sources. |
src/engine.rs |
Exposes WAL and durability arguments. |
src/errors.rs |
Adds WAL error codes. |
src/stores.rs |
Parses and installs WAL providers. |
tests/e2e/test_concurrency.py |
Covers durable concurrent processing. |
tests/e2e/test_durable_and_config.py |
Updates durable fixtures. |
tests/e2e/test_python_components.py |
Tests lifecycle behavior. |
tests/e2e/test_stores.py |
Tests WAL validation and persistence. |
tests/e2e/test_sync.py |
Tests blocking WAL support. |
tests/e2e/test_wal_recovery.py |
Tests cross-process recovery. |
tests/e2e/wal_recovery_worker.py |
Provides recovery subprocess worker. |
website/content/docs/api/_index.md |
Updates generated API reference. |
website/content/docs/guides/error-handling.md |
Documents WAL errors. |
website/content/docs/guides/python-reactions.md |
Explains durable-source requirement. |
website/content/docs/guides/python-sources.md |
Documents source durability. |
website/content/docs/guides/typing.md |
Lists the WAL type. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+234
to
+239
| async fn deprovision(&self) -> Result<()> { | ||
| if let Some(wal) = self.wal.read().await.clone() { | ||
| wal.delete_wal(self.base.get_id()).await?; | ||
| } | ||
| Ok(()) | ||
| } |
| | Member | Returns | | ||
| | --- | --- | | ||
| | `add_python_source(id: str, *, auto_start: bool = True)` | `Awaitable[None]` | | ||
| | `add_python_source(self, id: str, *, auto_start: bool = True, durability: bool = False )` | `Awaitable[None]` | |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drasi.createcould persist state and query indexes, but transient sources had no engine-level WAL provider. Sources configured for durability therefore failed at startup becauseDrasiLibhad no WAL provider to inject.This change:
wal_provider={"kind": "redb", "path": ...}to async creation, the sync wrapper, typed config, andfrom_configredb, validates the directory path with stable error codes, and wiresdrasi-wal-redb0.2.10 intoDrasiLibBuilderdurability=Trueto Python sources, with append-before-return dispatch, WAL replay, confirmed-position pruning, and cleanupSourceBase, preserving push acknowledgement and query subscriptions across source stop/startVolatile Python sources now report that they cannot replay changes. Persistent queries backed by Python sources must configure a WAL provider and opt the source into durability.
Validation includes the full hermetic suite, Rust tests and Clippy, Pyright, Ruff, generated-doc checks, and default plus RocksDB build configurations.
Closes #5