Repository navigation

perf: keep tokenizer workers alive for step 2 - #95

Closed

EllianCarlos wants to merge 25 commits into

masterfrom

perf/step2-4x-pr

Closed

perf: keep tokenizer workers alive for step 2#95
EllianCarlos wants to merge 25 commits into
masterfrom
perf/step2-4x-pr

Commits on Sep 28, 2026
Commits on Oct 4, 2026
- fix(blobExec): clear worker diagnostics per request
  EllianCarlos
  committed
  5db0fc1
- fix(tokenWorker): give helper stderr to its request
  EllianCarlos
  committed
  360f4d9
- fix(blobExec): refuse --tokenizer-worker outside pipelines
  EllianCarlos
  committed
  9d67aa0
- refactor(blobExec): take --tokenizer-worker=<path>
  EllianCarlos
  committed
  a4b0d8d
- refactor(blobExec): pass the budget to each worker request
  EllianCarlos
  committed
  1b4d214
- refactor(blobExec): reuse ChildRunner's lifetime and kill
  EllianCarlos
  committed
  4404bd5
- refactor(blobExec): drop the worker env map
  EllianCarlos
  committed
  8c9c563
- refactor(blobExec): drop the worker pool shutdown hook
  EllianCarlos
  committed
  e64608c
- refactor(tokenWorker): drop the BFG_WORKER_TRACE trace
  EllianCarlos
  committed
  16506b5
- test: drop test-only hooks from the worker code
  EllianCarlos
  committed
  b3a7d4c
- docs(tokenWorker): bring WORKER_PROTOCOL.md up to date
  EllianCarlos
  committed
  79a4786
- refactor(tests): split the token worker driver into checks
  EllianCarlos
  committed
  08723d1
- style(tokenWorker): drop comments that restate the code
  EllianCarlos
  committed
  653743b
- refactor(tokenWorker): split the long worker subs
  EllianCarlos
  committed
  d34faca
- refactor(blobExec): share the pool's timed read
  EllianCarlos
  committed
  dd713bc
- refactor(tokenize): take the DECL lines out of merge
  EllianCarlos
  committed
  b8d031d

Navigation Menu

Repository navigation

perf: keep tokenizer workers alive for step 2 - #95

perf: keep tokenizer workers alive for step 2#95
EllianCarlos wants to merge 25 commits into
masterfrom
perf/step2-4x-pr

Commits on Sep 28, 2026

perf(blobExec): Reuse tokenizer workers

perf(tokenizeByBlobId): Add persistent tokenizer worker

fix(tokenWorker): Silence perl exec warning on the failure fallback

ci: Run the tokenizer worker protocol tests

perf(tokenWorker): Optional per-request trace via BFG_WORKER_TRACE

perf(tokenWorker): Run the srcML chain in-process

fix: restore the exec bit on tokenize.pl and tokenBySha.pl

fix(tokenWorker): route .C and .H to C++, as tokenBySha.pl does

fix(tests): name the timeout process without exec -a

Commits on Oct 4, 2026

fix(blobExec): clear worker diagnostics per request

fix(tokenWorker): give helper stderr to its request

fix(blobExec): refuse --tokenizer-worker outside pipelines

refactor(blobExec): take --tokenizer-worker=<path>

refactor(blobExec): pass the budget to each worker request

refactor(blobExec): reuse ChildRunner's lifetime and kill

refactor(blobExec): drop the worker env map

refactor(blobExec): drop the worker pool shutdown hook

refactor(tokenWorker): drop the BFG_WORKER_TRACE trace

test: drop test-only hooks from the worker code

docs(tokenWorker): bring WORKER_PROTOCOL.md up to date

refactor(tests): split the token worker driver into checks

style(tokenWorker): drop comments that restate the code

refactor(tokenWorker): split the long worker subs

refactor(blobExec): share the pool's timed read

refactor(tokenize): take the DECL lines out of merge