Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .agents/skills/console-ui-e2e/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ when sequencing suites or CI jobs. Fix a failure with its affected spec/group,
then resume the remaining selection; use this skill for browser diagnosis.

```sh
pixi run clean-env
pixi run clean --runtime-only
pixi run test-container-e2e --scope console.operations --case <registered-owner> --concurrency 1 --node-cpus 0.25 --cpus 8 --memory-mib 16384
```

Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/debug-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Use existing runner records where available.
- Complete shared build prerequisites before parallel test jobs. Never run
concurrent builders against the same output directory; use independent
output directories when parallel builds are necessary. Perform shared
environment cleanup before dispatch; do not repeat `clean-env` while other
environment cleanup before dispatch; do not repeat `clean --runtime-only` while other
local jobs still own running fixtures. Host browser fixtures that require
stable Docker networks also share that constraint across local CI jobs;
keep their browser phases separate from other jobs' bridge creation/removal.
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/implement-requirement/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ design and testable contract; do not create a separate design draft.
commit.

Find tests with `pixi task list`; prefix server-spawning tests with
`pixi run clean-env &&`. Before push run fmt and `pixi run rs-lint`.
`pixi run clean --runtime-only &&`. Before push run fmt and `pixi run rs-lint`.

Proceed autonomously except when a design has multiple valid choices with no
clear winner, or a test still fails after five root-cause-driven attempts.
Expand Down
67 changes: 8 additions & 59 deletions .githooks/pre-commit
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
#!/bin/bash

repo_root=$(git rev-parse --show-toplevel)

echo "Running pre-commit checks..."

# Check Rust formatting
Expand All @@ -18,7 +16,7 @@ if ! pixi run cargo fmt --all -- --check; then
fi

# Collect staged C++ files
staged_cpp=$(git diff --cached --name-only --diff-filter=ACM -- 'lib/crowdb-tree/src/*.cpp' 'lib/crowdb-tree/include/crowdb-tree/*.h')
staged_cpp=$(git diff --cached --name-only --diff-filter=ACM -- 'lib/crowdb-tree/*.cpp' 'lib/crowdb-tree/*.h')

# Check C++ formatting (only staged files)
if [ -n "$staged_cpp" ]; then
Expand All @@ -39,63 +37,14 @@ if [ $? -ne 0 ]; then
exit 1
fi

# Run clang-tidy (only on staged .cpp files)
staged_cpp_src=$(git diff --cached --name-only --diff-filter=ACM -- 'lib/crowdb-tree/src/*.cpp')
if [ -n "$staged_cpp_src" ] && [ -d lib/crowdb-tree/build ]; then
echo "Running clang-tidy on staged C++ files..."
# Mirror CMake's liburing gate: reactor.cpp, block_async_page_store.cpp,
# and any file that includes reactor.h require liburing headers; on macOS
# (and anywhere liburing is missing) they are excluded from the build, so
# clang-tidy must skip them too. clang-tidy uses clang (not GCC) and can't
# find liburing.h in the pixi env, so skip these files when liburing.h is
# not at a system path. Most crowdb-tree source files transitively include
# reactor.h via crowdb-tree.h → options.h, so when liburing.h is not at a
# system path, skip clang-tidy entirely for crowdb-tree sources.
have_liburing=false
if [ -f /opt/homebrew/include/liburing.h ] || [ -f /usr/local/include/liburing.h ] || [ -f /usr/include/liburing.h ]; then
have_liburing=true
fi
cpp_files=()
for f in $staged_cpp_src; do
case "$f" in
*/reactor.cpp | */file_async_page_store.cpp | */block_async_page_store.cpp)
$have_liburing && cpp_files+=("$f")
;;
*)
if $have_liburing; then
cpp_files+=("$f")
else
# Skip: clang-tidy can't find liburing.h (transitively
# included via crowdb-tree.h → options.h → reactor.h).
echo " Skipping clang-tidy for $f (liburing.h not at system path)"
fi
;;
esac
done
if [ ${#cpp_files[@]} -gt 0 ]; then
tmpdir="$repo_root/.crowdb-runtime/artifacts/pre-commit/clang-tidy-$$"
mkdir -p "$tmpdir"
fail=0
for f in "${cpp_files[@]}"; do
base=$(basename "$f")
(pixi run clang-tidy "$f" -p lib/crowdb-tree/build 2>&1 | grep -E '^(error:|warning:|/.*: error:|/.*: warning:)' > "$tmpdir/$base.out") &
done
wait
for f in "${cpp_files[@]}"; do
base=$(basename "$f")
if [ -s "$tmpdir/$base.out" ]; then
cat "$tmpdir/$base.out"
fail=1
fi
done
rm -rf "$tmpdir"
if [ $fail -ne 0 ]; then
echo "❌ clang-tidy failed. Fix warnings before committing."
exit 1
fi
# Use the same compile database, dependency include paths and changed-header
# scope as the repository C++ gate. Preserve warnings as commit failures.
if [ -n "$staged_cpp" ]; then
echo "Running tree-lint on staged C++ files..."
if ! CT_LINT_WARNINGS_AS_ERRORS=1 pixi run tree-lint $staged_cpp; then
echo "❌ tree-lint failed. Fix diagnostics before committing."
exit 1
fi
elif [ -n "$staged_cpp_src" ] && [ ! -d lib/crowdb-tree/build ]; then
echo "⚠️ Skipping clang-tidy: lib/crowdb-tree/build not found. Run 'pixi run build' first."
fi

echo "✅ All checks passed!"
30 changes: 15 additions & 15 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ jobs:
- name: Cache compilation
uses: ./.github/actions/compilation-cache
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- name: Build C++ libs
run: pixi run build-cpp
run: pixi run build
- name: Run C++ and FFI tests
run: pixi run test-cpp
- name: Capture runner state on failure
Expand Down Expand Up @@ -117,7 +117,7 @@ jobs:
- name: Cache compilation
uses: ./.github/actions/compilation-cache
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- name: Run core component tests
run: pixi run test-core
- name: Capture runner state on failure
Expand Down Expand Up @@ -170,9 +170,9 @@ jobs:
- name: Cache compilation
uses: ./.github/actions/compilation-cache
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- name: Build C++ libs (crowdb-diskio subprocess)
run: pixi run build-cpp
run: pixi run build
- name: Run storage component tests
run: pixi run test-storage
- name: Capture runner state on failure
Expand Down Expand Up @@ -207,7 +207,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

AccessTests:
timeout-minutes: 90
Expand All @@ -228,7 +228,7 @@ jobs:
pixi-version: v0.77.0
- uses: ./.github/actions/compilation-cache
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- name: Run access component tests
run: |
pixi run test-access
Expand Down Expand Up @@ -272,7 +272,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

S3E2E:
env:
Expand Down Expand Up @@ -301,7 +301,7 @@ jobs:
artifact: ${{ needs.BuildClients.outputs.artifact }}
kind: clients
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- uses: actions/download-artifact@v4
with:
name: ${{ needs.ContainerImage.outputs.artifact }}
Expand Down Expand Up @@ -335,7 +335,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

IcebergE2E:
strategy:
Expand Down Expand Up @@ -391,7 +391,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

ConsoleTests:
runs-on: ubuntu-24.04
Expand All @@ -412,7 +412,7 @@ jobs:
- name: Cache compilation
uses: ./.github/actions/compilation-cache
- name: Clean env
run: pixi run clean-env
run: pixi run clean --runtime-only
- name: Run console tests
run: pixi run test-console
- name: Capture runner state on failure
Expand Down Expand Up @@ -447,7 +447,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

UITests:
runs-on: ubuntu-24.04
Expand Down Expand Up @@ -497,7 +497,7 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only

BuildClients:
runs-on: ubuntu-24.04
Expand Down Expand Up @@ -625,7 +625,7 @@ jobs:
- name: Verify centralized KV fixtures and protocol layer
run: |
pixi run test-container-e2e-fixture
pixi run clean-env
pixi run clean --runtime-only
pixi run test-e2e-kv
- name: Record KV verified digest
id: artifact
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/docker-preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,9 @@ jobs:
env:
CROWDB_PREVIEW_TEST_ARTIFACTS: ${{ runner.temp }}/crowdb-preview-artifacts
run: |
pixi run install-ui-deps
pixi run install-deps
pixi run bash -c 'cd e2e-test/clients/console-ui && npm ci --ignore-scripts'
pixi run clean-env
pixi run clean --runtime-only
pixi run --skip-deps test-single-node-container
pixi run test-container-e2e --profile bridge-admission --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS,CROWDB_COMPILED_TESTS "$(command -v pixi)" run --as-is test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
Expand Down Expand Up @@ -247,7 +247,7 @@ jobs:
- name: Verify centralized KV fixtures and protocol layer
run: |
pixi run test-container-e2e-fixture
pixi run clean-env
pixi run clean --runtime-only
pixi run test-container-e2e --layer kv --concurrency 2 --require-isolation --cpus 4 --memory-mib 4096
- name: Record KV verified digest
id: artifact
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/iceberg-container-ecosystem.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,4 +125,4 @@ jobs:
retention-days: 7
- name: Clean subprocesses
if: always()
run: pixi run clean-env
run: pixi run clean --runtime-only
6 changes: 3 additions & 3 deletions .github/workflows/release-container.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,9 @@ jobs:
env:
CROWDB_PREVIEW_TEST_ARTIFACTS: ${{ runner.temp }}/crowdb-preview-artifacts
run: |
pixi run install-ui-deps
pixi run install-deps
pixi run bash -c 'cd e2e-test/clients/console-ui && npm ci --ignore-scripts'
pixi run clean-env
pixi run clean --runtime-only
pixi run --skip-deps test-single-node-container
pixi run test-container-e2e --profile bridge-admission --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS,CROWDB_COMPILED_TESTS "$(command -v pixi)" run --as-is test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
Expand Down Expand Up @@ -250,7 +250,7 @@ jobs:
- name: Verify centralized KV fixtures and protocol layer
run: |
pixi run test-container-e2e-fixture
pixi run clean-env
pixi run clean --runtime-only
pixi run test-container-e2e --layer kv --concurrency 2 --require-isolation --cpus 4 --memory-mib 4096
- name: Record KV verified digest
id: artifact
Expand Down
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ and transport exposed through FFI.
- Markdown is primarily read raw. Prefer bullets; use tables for real
comparisons only. `doc/doc_index.md` always uses tables. When a table is
used, pad columns with spaces so the raw markdown aligns visually.
- Console UI changes follow `doc/design/console/design-crowdb-console-ui.md`.
Before changing visible behavior, update the UI spec and its ASCII layout
after agreement with the user; only then implement the UI change. Do not
unilaterally change the spec or introduce UI behavior that is absent from
the agreed spec. Each visible UI change needs focused test or E2E coverage.

## Skill dispatch

Expand Down
6 changes: 6 additions & 0 deletions app/crowdb-access-server/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,8 @@ impl std::fmt::Debug for IcebergLocatorSigningConfig {
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(default)]
pub struct DatasetConfig {
/// Delay before reclaiming marked Datasets and Snapshots.
pub reclaim_grace_seconds: u64,
pub listen: Option<String>,
pub max_in_flight: usize,
pub max_batch_size: usize,
Expand All @@ -256,6 +258,7 @@ pub struct DatasetConfig {
impl Default for DatasetConfig {
fn default() -> Self {
Self {
reclaim_grace_seconds: 20 * 60,
listen: None,
max_in_flight: 8,
max_batch_size: 10_000,
Expand Down Expand Up @@ -425,6 +428,9 @@ impl BaseConfig for AccessConfig {

impl AccessConfig {
fn validate_dataset(&self) -> Result<(), String> {
if self.dataset.reclaim_grace_seconds == 0 {
return Err("dataset.reclaim_grace_seconds must be positive".into());
}
if self.dataset.max_in_flight == 0 || self.dataset.max_in_flight > 64 {
return Err("dataset.max_in_flight must be between 1 and 64".into());
}
Expand Down
Loading
Loading