Skip to content

feat: add intentional workers and scoped parallel execution - #40

Merged
fazzatti merged 9 commits into
devfrom
feat/worker-pipelines
Oct 11, 2026
Merged

fazzatti merged 9 commits into
devfrom
feat/worker-pipelines

Conversation

@fazzatti

@fazzatti fazzatti commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Convee can execute intentionally authored functions, steps and pipelines in dedicated workers while keeping the normal Step API and caller-owned plugin lifecycle. Each worker definition exposes one callable. Parallel groups have independent capacities, and nested split results retain their exact association and TypeScript types when joined.

// word-count.worker.ts
export default worker(countWords, { module: import.meta.url });

// caller
const processDocuments = parallel({
  workers: { words: { worker: wordCount.use(wordHooks), count: 3 } },
  split: ({ words }, texts: string[]) => texts.map((text) => words.job(text)),
  join: (counts) => counts,
});

Behavior

  • worker(callable, { module: import.meta.url }) is a directly imported, immutable definition. worker.use() returns a configured copy with per-job caller plugins; embedded native plugins execute inside the worker. parallel.use() surrounds the whole invocation.
  • Ordinary parallel is a reusable Step with invocation-owned workers and automatic cleanup on every path. persistentParallel owns .step, optional warmup, graceful close, terminal termination/disposal and await using; optional reopening and idle timeout control intentional reuse.
  • Named groups schedule independently. Nested arrays/tuples preserve order and inference without task-name registries, roles, recipes, explicit pool generics or a separate public pool executor.
  • Whole batches reserve one bounded capacity budget before job hooks. Slots remain reserved through caller finalization; persistent calls share capacity. Explicit job context is snapshotted and fresh per execution.
  • all aggregates unrecovered errors after every admitted job settles; collect delivers the matching outcome tree. Remote diagnostics retain bounded Convee details, causes and group/job/path identities.
  • Startup and dispatch deadlines are separate. Crashes/timeouts fail their job without replay, await termination and use a bounded, group-wide replacement budget for later work. Exhaustion closes only that group. Constructor cancellation, late replies, cleanup failures and close/reopen races have dedicated regressions.
  • Built-in Deno, Node/Bun and browser adapters share the same public API. The optional /workers/vite integration preserves worker-file syntax and emits entries for actual development and production builds.

Validation

  • Full deno task verify passes locally on Deno 2.9.7: 44 runtime groups / 823 BDD steps, public type contracts, 29 README examples, tooling, architecture, type-depth budgets, isolated source-package consumers and JSR dry run.
  • Exact production coverage: 2834/2834 lines, 1093/1093 branches, 253/253 instrumented functions — 100% each, including the Vite integration. Strict gates and Codecov project/patch targets remain unchanged.
  • 61/61 targeted mutants killed, including five worker invariants; 60,000 seeded property cases and forced-GC retention checks pass.
  • Independent real consumers pass locally on Node 24.1.0, Bun 1.4.3, and Vite 8.3.4 / Playwright 1.64.0 in Chromium 156.0.8078.4, Firefox 157.0 and WebKit 27.2, each in development and production. Every browser run ends with zero live worker handles.
  • All 12 CI jobs pass at signed head c444c5f against dev, including Deno 2.6.0/2.9.6, Node 22.12.0/24.1.0, Bun 1.4.3 and all six browser/build combinations. Downloaded CI artifacts confirm 2834/2834 lines, 1095/1095 branches and 253/253 functions, plus 61/61 mutants and zero remaining browser worker handles. Deno 2.9.6 on Linux reports two additional, fully covered branches at workers/entry.ts:4 compared with local Deno 2.9.7 on macOS; the source and strict gates are unchanged.
  • Repository suites retain unit/integration filenames, BDD registrations and alias imports. Public factories, callable methods, options and error contracts have TSDoc; README examples are checked against the implementation.

Scope

This adds optional APIs to the released core, so the planned version remains 2.2.0. The replaced registry/pool API was introduced only in this unpublished PR. No WASM, automatic job retries, implicit buffer transfer, GC-based lifecycle guarantee, merge or publication is included. Custom transports must honor cancellation and actual termination; unconfirmed cleanup is reported. Module initialization runs in both caller and workers. Workers do not provide a security sandbox, and no workload-independent speedup is claimed. WebKit conformance is not a Safari-release certification.

* Export types (#6)

* docs: improve readme

* Create main.yml (#1)

* package: add jsr.json (#2)

* Feat jsr (#3)

* package: add jsr.json

* fix: jsr path

* feat: switch to deno (#4)

* Switch to Deno (#5)

* feat: switch to deno

* fix uuid generation

* feat: add export types

* packaging: bump version to 0.1.1

* refactor: replace storeOutput with storeMetadata and update tests accordingly

* chore: update version to 0.8.0 in deno.json (#28)

* Update src/utils/reduce-pipeline-steps-to-names.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix: pipeline types and add plugin functionality for pipeline-level o… (#30)

* Fix: error handling in CreateProcess to enrich and throw unhandled er… (#32)

* feat: add BeltPlugin tests and enhance plugin types with metadata handling

* docs: update README to enhance installation, usage, and core concepts sections

* feat: enhance GitHub Actions workflow for publishing with version detection and tagging

* refactor: streamline plugin and process engine implementations with enhanced documentation

* chore: update version numbers in deno.json and deno.lock for release

* Complete V1 refactor (#35)

* Release 0.10.0 (#34)

* V1 refactor

* refactor library docs and release workflows

* chore: update version number to 1.0.0 in deno.json

* chore: update publishing configuration and version number to 1.0.1 in deno.json
* feat!: harden execution contracts and add comprehensive quality gates

Unify lifecycle drivers, isolate invocation contexts, preserve argument tuples, normalize plugin descriptors, and repair fluent/type-depth contracts.

Add deterministic, generated, concurrency, mutation, architecture, package, retention and benchmark validation across supported Deno runtimes.

BREAKING CHANGE: enforce explicit tuple transport, body-only recovery, last-completed history and defensive configuration ownership. Prepare version 2.0.0 with documented migration guidance.

* fix(ci): parse colored mutation test summaries accurately

Normalize ANSI formatting before recognizing Deno failed-test summaries. Keep infrastructure failures and timeouts distinct from assertion kills. Add eight tooling regressions to the quality gate and verify the campaign with forced color.

* ci: refresh pinned actions to supported Node 24 runtimes

Use verified official checkout 7.0.1, upload-artifact 7.0.1 and Codecov 7.0.0 revisions. Preserve scoped permissions, existing inputs and the supported Deno matrix.

* docs: make usage and testing guides release-ready

Remove branch-only release commentary and Deno-only support framing. Keep migration guidance, document the actual CI matrix separately, and describe the reusable release workflow without candidate or unpublished-package instructions.

* docs: restore the published README walkthrough and examples

Restore the live 1.0.1 README layout, badges, architecture walkthrough, sync APIs, error model, design notes and license. Retain original example use cases and add current lifecycle, tuple, context and migration guidance.\n\nType-check 21 standalone snippets without rejecting import-only teaching examples. Add six regression tests for the development-only checker and document its scope. No library runtime, API, version or release-workflow changes.
Add optional worker registries, configured recipes and roles, bounded reusable pools, and ordinary async Steps with typed split/join and error policies. Include lifecycle, real-worker, type and package-consumer coverage; prepare the additive 2.2.0 release.
@codecov-commenter

codecov-commenter commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.96%. Comparing base (af8220a) to head (c444c5f).

❌ Your project status has failed because the head coverage (99.96%) is below the target coverage (100.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff             @@
##               dev      #40      +/-   ##
===========================================
- Coverage   100.00%   99.96%   -0.04%     
===========================================
  Files           17       39      +22     
  Lines         1998     3002    +1004     
  Branches       247      550     +303     
===========================================
+ Hits          1998     3001    +1003     
- Misses           0        1       +1     
Files with missing lines Coverage Δ
src/context/runtime.ts 100.00% <100.00%> (ø)
src/error/index.ts 100.00% <100.00%> (ø)
src/error/meta.ts 100.00% <100.00%> (ø)
src/error/serialize.ts 100.00% <100.00%> (ø)
src/index.ts 100.00% <100.00%> (ø)
src/pipe/error.ts 100.00% <100.00%> (ø)
src/pipe/factory.ts 100.00% <100.00%> (ø)
src/pipe/index.ts 100.00% <100.00%> (ø)
src/pipe/pipe.ts 100.00% <100.00%> (ø)
src/plugin/error.ts 100.00% <100.00%> (ø)
... and 28 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rename all suites with explicit unit or integration categories, replace native registration with describe/it, and align task discovery and replay references. Preserve existing assertions and production sources.
Use public, source, test and tooling aliases throughout repository checks. Generate README and type-depth imports through the import map while preserving isolated package-consumer verification.
Cover late transport callbacks after pool shutdown, require 100% line/branch/function coverage, and expand public TSDoc across both entrypoints and callable surfaces. Production code and type signatures are unchanged.

Verified full local checks, isolated native-worker consumption, both stale-callback guard mutants, and rejection of incomplete coverage reports.
@fazzatti fazzatti self-assigned this Oct 11, 2026
@fazzatti fazzatti changed the title feat: add typed worker pools and parallel steps feat: add intentional workers and scoped parallel execution Oct 11, 2026
@fazzatti
fazzatti changed the base branch from main to dev October 11, 2026 20:48
@fazzatti
fazzatti merged commit 027e453 into dev Oct 11, 2026
12 checks passed
fazzatti added a commit that referenced this pull request Oct 12, 2026
* Export types (#6)

* docs: improve readme

* Create main.yml (#1)

* package: add jsr.json (#2)

* Feat jsr (#3)

* package: add jsr.json

* fix: jsr path

* feat: switch to deno (#4)

* Switch to Deno (#5)

* feat: switch to deno

* fix uuid generation

* feat: add export types

* packaging: bump version to 0.1.1

* refactor: replace storeOutput with storeMetadata and update tests accordingly

* chore: update version to 0.8.0 in deno.json (#28)

* Update src/utils/reduce-pipeline-steps-to-names.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix: pipeline types and add plugin functionality for pipeline-level o… (#30)

* Fix: error handling in CreateProcess to enrich and throw unhandled er… (#32)

* feat: add BeltPlugin tests and enhance plugin types with metadata handling

* docs: update README to enhance installation, usage, and core concepts sections

* feat: enhance GitHub Actions workflow for publishing with version detection and tagging

* refactor: streamline plugin and process engine implementations with enhanced documentation

* chore: update version numbers in deno.json and deno.lock for release

* Complete V1 refactor (#35)

* Release 0.10.0 (#34)

* V1 refactor

* refactor library docs and release workflows

* chore: update version number to 1.0.0 in deno.json

* chore: update publishing configuration and version number to 1.0.1 in deno.json

* feat: add intentional workers and scoped parallel execution (#40)

* Update publish configuration (#37)

* Export types (#6)

* docs: improve readme

* Create main.yml (#1)

* package: add jsr.json (#2)

* Feat jsr (#3)

* package: add jsr.json

* fix: jsr path

* feat: switch to deno (#4)

* Switch to Deno (#5)

* feat: switch to deno

* fix uuid generation

* feat: add export types

* packaging: bump version to 0.1.1

* refactor: replace storeOutput with storeMetadata and update tests accordingly

* chore: update version to 0.8.0 in deno.json (#28)

* Update src/utils/reduce-pipeline-steps-to-names.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/pipeline/index.unit.test.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix: pipeline types and add plugin functionality for pipeline-level o… (#30)

* Fix: error handling in CreateProcess to enrich and throw unhandled er… (#32)

* feat: add BeltPlugin tests and enhance plugin types with metadata handling

* docs: update README to enhance installation, usage, and core concepts sections

* feat: enhance GitHub Actions workflow for publishing with version detection and tagging

* refactor: streamline plugin and process engine implementations with enhanced documentation

* chore: update version numbers in deno.json and deno.lock for release

* Complete V1 refactor (#35)

* Release 0.10.0 (#34)

* V1 refactor

* refactor library docs and release workflows

* chore: update version number to 1.0.0 in deno.json

* chore: update publishing configuration and version number to 1.0.1 in deno.json

* feat!: harden Convee runtime and add comprehensive quality gates (#38)

* feat!: harden execution contracts and add comprehensive quality gates

Unify lifecycle drivers, isolate invocation contexts, preserve argument tuples, normalize plugin descriptors, and repair fluent/type-depth contracts.

Add deterministic, generated, concurrency, mutation, architecture, package, retention and benchmark validation across supported Deno runtimes.

BREAKING CHANGE: enforce explicit tuple transport, body-only recovery, last-completed history and defensive configuration ownership. Prepare version 2.0.0 with documented migration guidance.

* fix(ci): parse colored mutation test summaries accurately

Normalize ANSI formatting before recognizing Deno failed-test summaries. Keep infrastructure failures and timeouts distinct from assertion kills. Add eight tooling regressions to the quality gate and verify the campaign with forced color.

* ci: refresh pinned actions to supported Node 24 runtimes

Use verified official checkout 7.0.1, upload-artifact 7.0.1 and Codecov 7.0.0 revisions. Preserve scoped permissions, existing inputs and the supported Deno matrix.

* docs: make usage and testing guides release-ready

Remove branch-only release commentary and Deno-only support framing. Keep migration guidance, document the actual CI matrix separately, and describe the reusable release workflow without candidate or unpublished-package instructions.

* docs: restore the published README walkthrough and examples

Restore the live 1.0.1 README layout, badges, architecture walkthrough, sync APIs, error model, design notes and license. Retain original example use cases and add current lifecycle, tuple, context and migration guidance.\n\nType-check 21 standalone snippets without rejecting import-only teaching examples. Add six regression tests for the development-only checker and document its scope. No library runtime, API, version or release-workflow changes.

* Add onFinally lifecycle hooks in Convee 2.1 (#39)

* feat: add typed worker pools and parallel steps

Add optional worker registries, configured recipes and roles, bounded reusable pools, and ordinary async Steps with typed split/join and error policies. Include lifecycle, real-worker, type and package-consumer coverage; prepare the additive 2.2.0 release.

* test: classify suites and use BDD throughout

Rename all suites with explicit unit or integration categories, replace native registration with describe/it, and align task discovery and replay references. Preserve existing assertions and production sources.

* refactor: use aliases for repository imports

Use public, source, test and tooling aliases throughout repository checks. Generate README and type-depth imports through the import map while preserving isolated package-consumer verification.

* test: enforce full coverage and document public API contracts

Cover late transport callbacks after pool shutdown, require 100% line/branch/function coverage, and expand public TSDoc across both entrypoints and callable surfaces. Production code and type signatures are unchanged.

Verified full local checks, isolated native-worker consumption, both stale-callback guard mutants, and rejection of incomplete coverage reports.

* feat: redesign intentional workers and parallel lifetimes

* feat: add Rust WASM steps and typed bindings (#41)

* feat: add Rust component WASM steps with explicit lifecycles

* test(wasm): establish owned-value ABI conformance matrix

* docs: consolidate consumer guidance in README

* fix: upload verified production coverage to Codecov (#43)

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants