Skip to content

debug.md

Latest commit

 

History

History
654 lines (514 loc) · 27.6 KB

debug.md

File metadata and controls

654 lines (514 loc) · 27.6 KB
Raw

Debugging

This guide is for contributors who need to understand or troubleshoot qrypt at runtime. It starts with the common workflows, then lists the lower-level commands and endpoints.

The debug tools collect structured evidence from a running qrypt process. Most read-only commands return JSON and include a diagnostics section when qrypt can identify likely problems.

Debug output is read-only unless a command is clearly named test or bench. Secrets are masked by the driver and config layers, but local paths, filenames, remote IDs, and short error messages may still be sensitive. Please review any bundle before sharing it.

Run One Test With A Temporary Mount

Use scripts/dev/debug-mount.sh when you need to run one probe test against a real qrypt mount from the current checkout, but do not need to keep the mount running for interactive investigation. This is useful for repeatable driver and VFS regression checks such as batchupload, batchmove, and fs: the script builds qrypt, creates an isolated temporary mount point and debug socket, waits for both to become ready, runs the selected test, then unmounts and removes the temporary local files. Read caches, thumbnail caches, upload staging, driver state, logs, and temporary data are also redirected below that temporary directory instead of using the config's [storage] paths. For valid test arguments, standard output contains only the qrypt probe test result, so it can be captured or compared directly.

The selected mount must exist in the config and have test_enabled = true. The script supports single-mount tests only; use the normal qrypt mount and qrypt probe test xfer commands when testing transfers between two mounts. Use the manual workflow later in this guide when the mount must remain available for collect, watch, raw endpoints, or an interactive reproduction.

Run a test with the repository qrypt.toml:

./scripts/dev/debug-mount.sh quark-test batchmove --count 50 --size 4k
./scripts/dev/debug-mount.sh quark-test read --size 256m --samples 3 --cache-mode both --pattern both --seek-count 10 --seek-size 1m
./scripts/dev/debug-mount.sh quark-test read --size 256m --cache-mode cold --pattern seek --seek-scenario prefetch

Pass a different config before the mount name:

./scripts/dev/debug-mount.sh --config ./path/to/qrypt.toml quark-test fs --size 30m

List all available test cases or inspect the wrapper options:

./scripts/dev/debug-mount.sh quark-test --help
./scripts/dev/debug-mount.sh --help

Mental Model

There are three layers worth keeping separate:

  • Driver: talks to the cloud provider.
  • VFS: tracks pending writes, staging files, upload queues, delayed deletes, read cache, and namespace mounts.
  • Debug socket: exposes runtime state through local /v1/... endpoints.

Most runtime debug commands need a mount scope:

--mount quark-test

Repeat --mount when two or more mounts are relevant:

--mount local --mount quark

Use --all-mounts only when the whole namespace is part of the issue. It can produce much larger reports.

Start Here

Start qrypt with a local debug socket:

go run ./cmd/qrypt mount --config ./qrypt.toml

Mount always starts the built-in per-config local endpoint. Unix-like systems use a filesystem socket; Windows uses a named pipe. Then connect using the normal command:

go run ./cmd/qrypt debug collect --mount quark-test

When the same config is discoverable or passed with --config, debug commands resolve the built-in endpoint automatically:

go run ./cmd/qrypt debug collect --config ./qrypt.toml --mount quark-test

Then collect one focused report:

go run ./cmd/qrypt debug collect --mount quark-test

If a specific path is involved, include it:

go run ./cmd/qrypt debug collect /quark-test/example.bin --mount quark-test

On Unix-like systems the socket file is created with 0600 permissions. qrypt refuses to replace a live endpoint and removes stale Unix socket files during startup. Windows named pipes use the operating system's pipe access control instead of filesystem permissions.

Android and remote-device debugging uses the Android app's explicit loopback debug server and tools such as adb forward; the CLI mount endpoint is local and does not expose the debug API on the LAN.

Enable Debug Logs

When the issue involves FUSE callbacks, retries, upload failures, or timing, enable debug logs in the config:

[logging]
log_level = "debug"
log_file = "~/.qrypt/qrypt.log"
error_file = "~/.qrypt/qrypt-error.log"

The debug log includes callback names, paths, return codes, durations, and sampled runtime messages. The error log is usually the fastest place to look for warnings and failures after a reproduction.

Windows

On Windows, qrypt uses a named pipe derived from the per-config runtime identity; the CLI does not require an endpoint path:

qrypt.exe mount --config .\qrypt.toml

Keep the mount process running, then use the same path in another PowerShell window:

qrypt.exe debug collect --mount local
qrypt.exe debug collect /local/example.txt --mount local
qrypt.exe debug raw health

The runtime identity is created from the resolved qrypt configuration and is shared by mount, debug, probe, and control commands. Named pipe access is restricted by the Windows pipe security descriptor; no password is required for the default same-user local workflow.

Choose A Command

Goal Command
Collect one structured diagnosis debug collect --mount NAME
Sample state while reproducing a timing issue debug watch --mount NAME
Create a shareable zip artifact debug bundle --mount NAME --out FILE
Query one socket endpoint debug raw ENDPOINT
Verify driver credentials probe test auth --mount NAME
Verify driver CRUD behavior probe test crud --mount NAME
Verify VFS upload behavior probe test fs --mount NAME
Measure full mounted-file read latency and throughput probe test read --mount NAME --mount-point PATH
Verify same-directory batch upload caching probe test batchupload --mount NAME
Measure same-mount batch moves probe test batchmove --mount NAME
Verify resumable upload recovery probe test resume --mount NAME
Run the full contract suite probe test contract --mount NAME
Compare performance or regressions probe bench ...

probe test contract runs the complete suite against the named mount (auth, contract, crud, fs, instantupload, resume, multipart) and requires a running mount server (qrypt mount --config ./qrypt.toml). The built-in per-config probe socket is resolved automatically. The mount must have test_enabled = true in its [[mounts]] block; unlisted mounts are rejected with 403. This is the same suite the Contract Tests CI workflow runs nightly against real providers.

Use collect first when you are unsure. Use raw when you already know which endpoint you need. Use watch when the issue only appears during a time window, for example upload retries or stale reads.

Inspect Uploads And Upload

This is the most useful workflow for staging, pending upload, and slow upload problems.

If you need to trigger a controlled VFS write, run:

go run ./cmd/qrypt probe test fs --mount quark-test --size 30m

probe test fs creates temporary files through the VFS, writes data, flushes, waits for upload, reads back, and removes the test data. It writes to the selected mount, so use it only when temporary remote writes are acceptable.

For small-file and move workloads, run:

go run ./cmd/qrypt probe test batchupload --mount quark-test --count 50 --size 4k
go run ./cmd/qrypt probe test batchmove --mount quark-test --count 50 --size 4k

batchupload writes and flushes every file through VFS, waits for the upload queue to become idle, verifies the directory and sampled content, and reports the parent-directory cache miss/hit/shared counts. When upload history is available, more than one parent-cache miss fails the test. batchmove first uploads the source batch and waits for it to become idle, then measures only the sequential VFS Rename calls before verifying the empty source and the destination data. Both tests clean their randomized temporary directories.

To measure the complete mounted-file read path, including the OS filesystem, FUSE/macFUSE adapter, VFS read cache and remote driver, run:

./scripts/dev/debug-mount.sh quark-test read --size 256m --samples 3 --cache-mode both --pattern both --seek-count 10 --seek-size 1m

The wrapper supplies its temporary mount point automatically. When invoking qrypt directly, pass the real FUSE mount directory explicitly:

go run ./cmd/qrypt probe test read --mount quark-test --mount-point /Volumes/qrypt --size 256m --block-size 1m --samples 3 --cache-mode both --pattern both --seek-count 10 --seek-size 1m

The test creates and uploads a deterministic temporary file before timing, then reads it with os.Open through the supplied mount point. Setup and cleanup are excluded from the read measurements. cold clears qrypt's durable and in-memory read caches; every measured read requests OS cache bypass/drop where supported, while warm keeps qrypt's read-cache state from the preceding read. both reports both modes. Each measurement includes open and time-to-first-byte latency, total and steady throughput, peak throughput over a one-second window, data-bearing read-call latency percentiles, cache hit/miss deltas, VFS read-call count, byte count, and content-hash validation. The test fails when the supplied path bypasses qrypt (for example, when --mount-point names the backing directory instead of the FUSE mount). metrics_truncated marks a measurement whose phase-level history exceeded the bounded runtime history. The test does not use a fixed network-speed threshold.

--pattern sequential preserves the full-file test above and remains the default. --pattern seek opens the mounted file, seeks to widely spaced offsets, and loads --seek-size bytes at each offset; --pattern both runs both patterns. Seek output separates the local seek call (seek_us) from the following data load (seek_load_us) and their combined latency (seek_total_us). seek_summary reports cold/warm P50, P95, P99, max, and load throughput. Cold probes clear qrypt's read cache before every offset; warm probes read the same offset immediately after it has been primed. The result also requires vfs_offset_match=true, proving that the measured FUSE read reached qrypt at the requested seek offset. The defaults are 10 probes of 1 MiB each, with a maximum of 32 probes.

Use --seek-scenario to control the load present when each seek starts:

  • isolated is the default baseline and issues no intentional competing read.
  • prefetch first reads sequential chunks on the same file descriptor, waits until qrypt reports an active background prefetch for that file, then seeks on that descriptor.
  • concurrent starts a bounded sequential read on a second file descriptor, waits until qrypt reports its active chunk load or resulting prefetch, then seeks on the first descriptor.

For example:

./scripts/dev/debug-mount.sh quark-test read \
  --size 256m --cache-mode cold --pattern seek \
  --seek-scenario concurrent --seek-warmup-chunks 2 \
  --seek-overlap-timeout 2s

The loaded range and seek target are kept apart so the competing request does not pre-warm the bytes being measured. The test synchronizes against qrypt's active-operation state instead of sleeping for a fixed interval. Every loaded scenario reports overlap, overlap_wait_us, active_kind, active_offset, and active_requested; it fails if no matching in-flight request is observed within --seek-overlap-timeout. Every probe waits for its reads and prefetches to become idle before its events are collected; the wait is excluded from seek_load_us. This prevents old work from satisfying the next overlap check, repopulating a newly cleared cache, or being attributed to the following measurement. chunks_touched distinguishes aligned single-chunk loads from cross-chunk loads. Compare isolated, prefetch, and concurrent runs with the same size, cache mode, probe count, and seek size.

If the issue is about interrupted uploads or resumable upload sessions, run:

go run ./cmd/qrypt probe test resume --mount quark-test --size 64m

probe test resume writes one temporary file, injects a one-shot upload context canceled fault after upload progress starts, waits for qrypt to retry, then checks that the final remote file has the expected size. A passing result means the VFS upload path recovered from the cancellation. The JSON outcome field tells you whether the driver reused a resumable session or safely fell back to a fresh upload after the provider rejected the old session.

To inspect the current state without continuous sampling, use these raw endpoints:

go run ./cmd/qrypt debug raw '/v1/uploads?mount=quark-test&history=1'
go run ./cmd/qrypt debug raw '/v1/pending?mount=quark-test'
go run ./cmd/qrypt debug raw '/v1/staging?mount=quark-test'
go run ./cmd/qrypt debug raw '/v1/cache?mount=quark-test'
go run ./cmd/qrypt debug raw '/v1/events?level=warn&limit=100'
go run ./cmd/qrypt debug raw '/v1/events?level=warn&mount=quark-test'

/v1/events filters on the event's fields, not on the message text: mount= (repeatable, matching any listed mount) selects lines produced by a mount-scoped call site, and component= matches the message's leading [TAG]. A line is therefore never returned for a mount it merely mentions in its text, and path= remains a plain substring match over the message.

Mount attribution currently covers the pending-upload and read-cache stores, so their journal, eviction and cleanup warnings carry mount="NAME" in both the log text and the event field. Call sites that are not yet mount-scoped produce lines with no mount field, and /v1/events?mount= does not return them — an empty result means "no attributed line for that mount", not "that mount was quiet".

Useful fields:

  • uploads.uploads[]: active and recent upload attempts, state, retry count, uploaded bytes, and last error.
  • pending.pending[]: files waiting for upload.
  • staging.mounts[].files[]: local staging file size and whether it matches the pending size.
  • staging.mounts[].orphans[]: staging files not referenced by pending state.
  • cache.mounts[].cache.journal: pending journal size, duplicate entries, and whether compaction is recommended.
  • events.events[]: recent warnings and errors.
  • faults[]: currently armed debug upload-cancel faults. This should normally be empty outside an active probe test resume run.

For one report instead of separate raw calls:

go run ./cmd/qrypt debug collect --mount quark-test

For time-series samples during a reproduction window:

go run ./cmd/qrypt debug watch --mount quark-test --duration 120s --interval 2s

Inspect Reads And Read Cache

For stale reads, unexpectedly slow reads, or cache behavior, start with a path focused report:

go run ./cmd/qrypt debug collect /quark-test/path/file.bin --mount quark-test

Raw endpoints:

go run ./cmd/qrypt debug raw '/v1/reads?mount=quark-test&path=/quark-test/path/file.bin'
go run ./cmd/qrypt debug raw '/v1/cache?mount=quark-test&path=/quark-test/path/file.bin'
go run ./cmd/qrypt debug raw '/v1/consistency?path=/quark-test/path/file.bin'

Useful fields:

  • reads.reads[]: recent read events, source, bytes, cache hits/misses, and errors.
  • cache.mounts[].cache: read cache counters and per-file cache details.
  • consistency.report: whether the path is pending, found remotely, and size matched.

Reading The Read-Cache Replacement State

cache.mounts[].cache carries the counters visible to the eviction policy, and they answer two different questions:

  • unproven_bytes / proven_bytes split the cache by state: a chunk is unproven until a read run other than the one that admitted it touches it. A sequential stream consuming its own read-ahead only ever produces unproven content, which is why a one-pass scan shows up here rather than as hits.
  • stream_bytes is the budget view: the unproven bytes that live in the large-file class, i.e. what the sequential sub-budget is currently charging. It is bounded by a share of the large-class ceiling whenever a drain runs. Small-class unproven content has no sub-budget, so stream_bytes is normally smaller than unproven_bytes.
  • protected_bytes / probationary_bytes split the proven content by segment. A drain empties probationary before touching protected, so a cache whose protected bytes hold steady across a large scan is working as intended.
  • evicted_bytes and promotions are event counters: how much the cache has released, and how many chunks a later run has confirmed as reused.

Expect the disk footprint to look different from older builds: content read once in a single pass is held in the sub-budget rather than filling the whole large-class ceiling, so a cache serving only streaming reads will settle below its max_size instead of at it. Content that is read again is promoted and released from that sub-budget, so repeatedly used files still fill the cache.

Read events are kept in two bounded rings and merged into one append-ordered list: summaries (one per read: offset, bytes, cache hits/misses, duration, throughput, error) retain 128 entries, and details (per chunk and per window phase timing) retain 512 entries. Summaries are never evicted by a read's detail burst, so the recent reads stay visible even while a whole-file read walks every chunk; details are recency-biased and a large sequential read can flush older ones. The two rings are bounded at roughly 320 KiB per mount, allocated only once the mount has served a read.

Each mount also carries cumulative runtime.counters, which answer "how fast and how reliable overall" where the bounded histories answer "which read, and where did it spend its time":

  • ops, errors, bytes, duration_ms, mean_ms are totals since process start — they never reset, so they do not equal anything in the bounded event history and are the right baseline for spotting degradation.
  • latency.bounds_ms / latency.counts are a fixed-bucket histogram, with the last count being the overflow bucket. Derive a percentile from the counts: the reported value is that bucket's upper bound, so read it as "p95 is at most this". exceed is true when the percentile falls in the overflow bucket.
  • bytes counts what the read path materialized: a remote read counts the data it fetched, while a staging passthrough counts 0 because it only opens a local file and hands the stream back. The read event for that passthrough instead reports the bytes the consumer drained, so the two differ by design.

Inspect Cross-Mount Transfers

For copy or transfer issues, provide both endpoints and both mount names:

go run ./cmd/qrypt debug collect /local/source.bin \
  --dest /quark/archive/source.bin \
  --mount local \
  --mount quark

Useful commands:

go run ./cmd/qrypt probe test xfer --source local --dest quark --size 16m
go run ./cmd/qrypt probe test xfer --source local --dest quark --size 16m --vfs
go run ./cmd/qrypt probe bench xfer --source local --dest quark --size 16m --samples 3 --sample-interval 2s

test xfer and bench xfer create temporary remote objects. The --vfs mode tests transfer behavior through the VFS layer instead of directly through drivers.

Inspect Driver Behavior

Use driver tests when the question is about the provider API or credentials, not local staging or VFS upload.

Read-only auth check:

go run ./cmd/qrypt probe test auth --mount quark-test

Write-capable CRUD check:

go run ./cmd/qrypt probe test crud --mount quark-test

Driver state and health:

go run ./cmd/qrypt debug raw '/v1/driver?mount=quark-test'
go run ./cmd/qrypt debug raw '/v1/mounts/health?mount=quark-test'

probe test auth does not create remote objects. probe test crud creates temporary remote objects and cleans them up best-effort.

Create A Shareable Bundle

Use bundle when you want a zip file that another contributor or AI assistant can inspect:

go run ./cmd/qrypt debug bundle --mount quark-test --out /tmp/qrypt-debug.zip

For a path-specific issue:

go run ./cmd/qrypt debug bundle /quark-test/path/file.bin \
  --mount quark-test \
  --out /tmp/qrypt-path-debug.zip

For a transfer issue:

go run ./cmd/qrypt debug bundle /local/source.bin \
  --dest /quark/archive/source.bin \
  --mount local \
  --mount quark \
  --out /tmp/qrypt-transfer-debug.zip

For timing-sensitive issues, include a watch window:

go run ./cmd/qrypt debug bundle /quark-test/path/file.bin \
  --mount quark-test \
  --watch 30s \
  --out /tmp/qrypt-watch-debug.zip

Existing files are not overwritten unless --force is provided. --goroutines adds a goroutine dump and should be used only when runtime deadlock or blocking is suspected. --all-mounts can make the bundle much larger; prefer --mount NAME when the relevant mount is known.

Offline Checks

Offline checks read cache files and do not require a running mount:

go run ./cmd/qrypt fs pending --config ./qrypt.toml
go run ./cmd/qrypt fs pending --config ./qrypt.toml --verbose
go run ./cmd/qrypt fs journal --config ./qrypt.toml
go run ./cmd/qrypt fs journal --config ./qrypt.toml --json
go run ./cmd/qrypt fs journal --config ./qrypt.toml --mount quark-test

fs pending --verbose shows virtual path, expected size, staging file path, retry count, last error, and next retry time. fs journal checks pending.jsonl, missing staging files, size mismatches, and orphan staging files.

Endpoint Reference

Use debug raw when you need one endpoint directly:

go run ./cmd/qrypt debug raw health
go run ./cmd/qrypt debug raw '/v1/state?mount=quark-test'

Common endpoints:

Endpoint Purpose Size notes
/v1/health Debug socket health, including task-journal persistence status Small
/v1/runtime Go runtime and memory summary Small
/v1/debug/stacks Goroutine stack dump Text, can be large
/v1/state?mount=NAME Mount snapshot Can grow with pending, cache, events
/v1/pending?mount=NAME Pending upload files Grows with pending count
/v1/tasks?type=upload_stream_batch&state=waiting_input&limit=200 Persistent tasks with item progress and recovery linkage Grows with task and item count; use filters
/v1/task-events?after_seq=N&wait_ms=25000 Incremental task snapshots with cursor replay and gap recovery Bounded by the task event history and response batch
/v1/uploads?mount=NAME&history=1 Active and recent uploads Bounded history
/v1/ops?mount=NAME Active VFS read, chunk, window, and prefetch ops Small, current in-flight ops only
/v1/reads?mount=NAME&limit=200&since=2m Recent read events: 128 summaries plus 512 phase/chunk details per mount Bounded history, filtered server-side
/v1/staging?mount=NAME Staging files and orphan staging files Grows with staging files
/v1/cache?mount=NAME Read cache and pending journal health Grows with cache files
/v1/events?level=warn&limit=100&mount=NAME&component=CACHE Recent warnings/errors, filtered by mount and component fields Limited by limit
/v1/driver?mount=NAME&limit=200&since=2m Driver debug snapshot and metrics Driver dependent, metrics filtered server-side
/v1/mounts/health?mount=NAME Recent operation health Small
/v1/resolve?path=PATH Resolve a virtual path Small
/v1/consistency?path=PATH Compare pending and remote state for a path Small for one path

Raw endpoints can take repeated mount=NAME query parameters when multiple mounts are relevant. /v1/reads and /v1/driver also support operation, phase, op_id, remote_id, and error_only=1.

Benchmarks

Use benchmarks when you need comparable performance artifacts:

go run ./cmd/qrypt probe bench crud --mount quark-test
go run ./cmd/qrypt probe bench crud --mount quark-test --samples 3 --sample-interval 2s
go run ./cmd/qrypt probe bench fs --mount quark-test --samples 3 --sample-interval 2s
go run ./cmd/qrypt probe bench xfer --source local --dest quark --size 16m --samples 3 --sample-interval 2s
go run ./cmd/qrypt probe bench compare --base baseline.json --current current.json

Benchmark reports include summary, assessment, environment.network_probe, samples[], cases[], and events[]. With --samples, the summary includes median, p95, max duration, throughput statistics, and coefficient of variation.

bench fs reuses the VFS smoke test. It reports VFS operations such as write, flush, wait_upload, read, remove, and wait_cleanup. Its summary.vfs section includes pending/upload/delete drain state, cache hit ratio, cache errors, staging orphan counts, staging size mismatch counts, and active loader counters.

Event And Snapshot Notes

Mount state is reported as vfs.MountSnapshot:

  • identity: mount name, driver name, root, capabilities, and encryption state
  • queues: upload/delete queue and timer state
  • overlay: pending files, delayed deletes, rename overlays, restored directories, and hidden copy entries
  • upload_state: active and historical uploads
  • cache: read cache and pending journal health
  • events: recent drive.MetricEvent streams such as VFS reads and driver metrics
  • runtime: chunk, window, and prefetch activity counters

Events use drive.MetricEvent. op_id, step, and name are correlation labels. kind identifies the event family such as driver, vfs_read, vfs_upload, or transfer. operation is the low-cardinality operation name used for comparisons, and phase is the measured sub-step.

HTTP debug events must keep secrets masked. Include method, sanitized URL without query parameters, status, duration, request field names, response size, and short masked error snippets only. Do not include cookies, session keys, signed upload URLs, encrypted request blobs, or full response bodies.

Safety Notes

  • collect, watch, bundle, and normal raw GET endpoints are observation tools.
  • probe test and probe bench may create, upload, read, rename, or delete temporary remote objects.
  • probe test and probe bench only run against mounts with test_enabled = true in [[mounts]]. Transfer tests require both source and destination mounts to opt in.
  • probe test fs --size accepts bytes or k, m, g suffixes. Large values consume upload time and provider bandwidth.
  • probe test batchupload and probe test batchmove accept --count (default 50, maximum 100) and --size (default 4 KiB per file, maximum 1 MiB).
  • probe test read requires --mount-point. It accepts --size (default 256 MiB, maximum 2 GiB), --block-size (default 1 MiB), --samples (default 1, maximum 10), --cache-mode cold|warm|both (default both), --pattern sequential|seek|both (default sequential), --seek-count (default 10, maximum 32), --seek-size (default 1 MiB, maximum 16 MiB), --seek-scenario isolated|prefetch|concurrent (default isolated), --seek-warmup-chunks (default 2, maximum 16), and --seek-overlap-timeout (default 2s, range 100ms to 30s). It creates and removes a temporary remote file of the requested size.
  • probe test resume --size has the same size format. It intentionally cancels one upload attempt through the VFS debug fault injector, then waits for normal retry or resumable-upload recovery.
  • --all-mounts can significantly increase report size.
  • --goroutines is useful for blocking or deadlock investigations, but it can make bundles larger.