Bump mistralai to 2.x (dev merged in)#1007
Open
lchoquel wants to merge 71 commits into
Open
Conversation
…ge-4) dev's #969 reconciled runtime bridge makes merge-4's bridge work redundant, so this branch is rebranched off dev and carries only the mistralai 2.x HOLD group on top of the current dev — no bridge/tracing/graph reconciliation. - Mistral provider (config, factory, llm/extract workers, llms) migrated to the mistralai 2.x client layout (mistralai.client.*); the message-is-None guard in mistral_llm_worker is 2.x-only. - mistralai>=2.4.4 + instructor git-fork pin in [tool.uv.sources]. Gated: do not publish until instructor #2298 ships mistralai-2.x support to PyPI. - test_transport_retry_wiring keeps dev's #969 gateway-max-retries tests and adopts the 2.x RetryConfig import path. - uv.lock regenerated (mistralai 2.4.9; instructor from the fork). make agent-check + make agent-test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-repo master status/verification/next-steps doc for the mistralai 2.x work (both feature/mistralai-2x-bump here and feature/Mistral-native in the plugin): branch state, a verification checklist the user can run, the publish gate, and the un-gate procedure for when instructor #2298 ships mistralai-2.x to PyPI. Folds the pre-execution handoff into the track folder as background-rationale.md and indexes the track in wip/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings post-#969 dev (through v0.34.0: keyword-only enforcement #991, concept_ref rename #990, recursive design #970, protocol unification #985, Temporal determinism #987, test-coverage grind #989) onto the mistralai 2.x bump. Conflict resolutions: - mistral_factory.py (make_simple_messages): kept dev's correct ordering (system message before user message) on top of the branch's 2.x type (list[ChatCompletionRequestMessage]). The bottom-appended system block auto-resolved away via dev's move; consistent with dev's CHANGELOG fix. - CHANGELOG.md: kept the branch's [Unreleased] mistralai-2.x entry above dev's [v0.34.0] release section. - wip/README.md: kept both track entries (keyword-only-arguments + mistralai-2x-bump). Post-merge fixes (semantic, beyond textual conflicts): - Migrated dev's three new Mistral test files (test-grind #989) from the 1.x SDK layout to 2.x: `mistralai.models` -> `mistralai.client.models` and `mistralai import Mistral` -> `mistralai.client import Mistral`. Verification: uv lock --check clean; mistralai 2.x pin (2.4.9) and the instructor git-fork source preserved; make agent-check green (pyright, mypy, keyword-only guard); full make agent-test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QutaVv26fHKxFFi5orP3q
- Introduced README.md outlining the plugin architecture, including orchestrators and inference backends as plugins. - Created inference-backends-as-plugins.md detailing the plan to convert inference SDK wrappers into registerable plugins. - Established orchestrators-as-plugins.md to define the strategy for making orchestrators discoverable plugins. - Developed temporal-as-plugin.md to outline the extraction of the Temporal integration into a first-class plugin.
- Introduced a comprehensive implementation plan for the Pipelex plugin system, outlining the rationale, phases, decisions, and deliverables. - Defined the plugin enablement model, emphasizing the separation of presence and activation. - Structured the rollout into distinct phases, including renaming existing components, integrating inference families, and externalizing the Temporal plugin. - Established a clear decision log and resolved open questions to guide the implementation process. - Included checkpoints for each phase to ensure progress and maintain quality.
Pure rename, no behavior change. Kills the three-way "plugin" overload so the new plugin-system vocabulary is unambiguous. - Plugin -> ModelHandle (plugins/plugin.py -> plugins/model_handle.py) - PluginSdkRegistry -> SdkClientRegistry (+ get_sdk_instance/set_sdk_instance -> get/set); module plugin_sdk_registry.py -> sdk_client_registry.py - PluginManager -> SdkClientManager (.plugin_sdk_registry -> .sdk_client_registry); module plugin_manager.py -> sdk_client_manager.py - PluginFactoryAbstract -> BackendExtrasFactory (module backend_extras_factory.py) - hub get_/set_plugin_manager -> get_/set_sdk_client_manager; fix the bogus "PluginManager2 is not initialized" message - fix the four wrong-path Plugin imports (llm/img_gen/extract pulled Plugin from plugin_sdk_registry; search used the correct path) - carry the type rename through the variables/params that held it: plugin -> model_handle across the worker factories and in-tree adapters - tests/fixtures updated to match (plugin= -> model_handle=, fixtures plugin_for_* -> model_handle_for_*, plugin_fixtures.py -> model_handle_fixtures.py) D4: SdkClientManager stays a thin wrapper holding one SdkClientRegistry; collapsing it is a deferred P3 follow-up. agent-check / tb / agent-test all green. Code-review (pure-rename scope): clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHgQXAoK1DKSwubHg8Doej
Builds the discovery/contract/registry seam and migrates the LLM inference family onto it. No behavior change in dispatch (one deliberate registry-miss error reword). Seam (pipelex/plugins/): - contract.py: PipelexPlugin Protocol + PLUGIN_API_VERSION; side-effect-free register invariant. - inference_backend_registry.py: InferenceFamily, MakeWorkerFn, the (family,sdk) registry, and require_sdk (the DRY find_spec guard, moved into make_worker). - orchestrator_registry.py: OrchestratorProtocol + registry (skeleton; wired in Phase 3). - registrar.py: PluginRegistrar accumulator with per-plugin attribution + PluginDiscovery; duplicate (family,sdk)/mode/slot fail loud naming both. - discovery.py: pure, idempotent build_registrar (built-ins + entry points, version check, plugins.disabled denylist, fail-loud BrokenPluginError). - builtins.py + per-vendor *_plugin.py (openai/gateway/portkey/anthropic/ mistral/bedrock/google): each register() adds (LLM, sdk) backends whose lazy make_worker closures reproduce the old match arms byte-for-byte. - exceptions.py: the named fail-loud plugin errors. LLM migration: llm_worker_factory's match collapses to get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)(...). SdkClientRegistry.get_or_create DRYs the get-or-set cache dance. Boot: build_registrar runs in setup() after the gateway/model checks and before the content-generator/router/run hub setup points; the two registries are stored on the hub. Registrar held on self for Phase 3 slot/CLI/teardown apply-points. Control surface (D7): PluginsConfig (disabled list) + [plugins] in both pipelex.toml files; `pipelex plugins list` command. Import-cycle break: PipelexPipeRunInput/Output extracted to import-light runtime_bridge/payloads.py; bridge re-exports them. Tests: plugin discovery contract conformance (version/duplicate/broken/idempotent/ denylist), subprocess import-light guard, and the LLM factory routing tests now exercise the real closures through the real registry. agent-check / tb / agent-test all green. Code-review (xhigh) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHgQXAoK1DKSwubHg8Doej
Captures the full SDK -> (vendor plugin · worker · client build) map for the ImgGen/Extract/Search families, the new vendor plugins to create vs existing ones to extend, the huggingface find_spec bug fix (C10), the stateless arms, and the search reporting_delegate normalization (C9). No code change — lets a fresh session start Phase 2 without re-reading the three worker factories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHgQXAoK1DKSwubHg8Doej
…the inference-backend seam Collapses the remaining three inference worker-factory `match` statements (`img_gen`, `extract`, `search`) into `get_inference_backend_registry().lookup(family, sdk)`, exactly like the Phase 1 LLM factory. Core now names no inference backend by import or string in any of the four families. The registry holds 30 (family, sdk) backends from 15 built-in plugins. New vendor plugins (each import-light, SDKs lazy-imported inside closures): fal, huggingface, blackboxai, openrouter, azure_rest, docling, pypdfium2, linkup. Extended vendor plugins (one register() now spans families): - gateway: now serves all four families (img_gen ×2, extract, search) - mistral: + extract - google, openai: + img_gen codex C10 fix: the old `huggingface_img_gen` arm imported `huggingface_hub` with no `find_spec` guard. The huggingface plugin now guards it with `require_sdk(spec="huggingface_hub", extra="huggingface")`; a huggingface error-parity case in the img-gen factory test proves the fix. codex C9 normalization: `make_search_worker` now takes `reporting_delegate` and no longer reaches into the hub; `search_generate._make_search_worker` supplies `get_report_delegate()`. Removes search's hidden hub coupling with no production behavior change. Tests: img-gen routing test rewired to build the real registry from BUILTIN_PLUGINS; new test_inference_backend_coverage pins the full 30-key round-trip + cross-family vendors; import-light guard now also blocks docling + linkup. Docs: docs/under-the-hood/inference-backend-plugins.md — Inference SPI reference, seam walkthrough, and a copy-pasteable authoring example (added to mkdocs nav). No behavior change in dispatch except the uniform registry-miss wording. agent-check clean (pyright 0, mypy 0); agent-test green. Code-review clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHgQXAoK1DKSwubHg8Doej
Three confirmed review-agent findings on the new inference-backend plugin seam: - require_sdk: wrap the find_spec probe in try/except ModuleNotFoundError so a dotted spec whose parent package is entirely absent (e.g. google.genai with no google distribution installed) surfaces as MissingDependencyError with the pipelex[<extra>] hint, not a raw import error. (greptile P1) - Linkup search backend: guard the optional linkup SDK with require_sdk before importing the top-level-linkup-importing worker, mirroring the extract backend; add a search-specific missing-dependency message. (greptile P1) - Plugin discovery: denylist external entry points by entry-point name BEFORE load(), so a broken/dependency-missing installed plugin can be recovered via plugins.disabled instead of raising BrokenPluginError at load. Working externals remain disable-able by either their entry-point name or their .name. (greptile P1 + codex P2) Regression test added for each; under-the-hood plugin docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHgQXAoK1DKSwubHg8Doej
Three CONFIRMED gaps from the xhigh /code-review pass over the refactor/Plugins diff, plus the deferred follow-ups doc: - import-light guard: blocklist was missing the migrated plugins' deferred SDKs; add openai, portkey_ai, pypdfium2 (verified none are imported at registration time, so the invariant still holds). - InferenceBackendRegistry.lookup raised a bare NotImplementedError on a registry miss; replace with a structured InferenceBackendNotFoundError (PluginError) so it carries error_domain/type_uri. Update both factory miss-tests. - require_sdk named ALL specs on a partial multi-spec miss; collect and name only the absent one(s) so a user who already has one SDK is not told to reinstall it. Add a multi-spec regression test. Remaining review findings (latent footguns reachable only via external entry-point plugins / future callers + quality cleanups) recorded in wip/plugins/phase-2-review-deferred.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed items Append a dated post-checkpoint delta to the Phase 2 as-built: the whole-PR xhigh /code-review pass (#997), the three pre-merge fixes landed in 94fa908 (structured lookup error, import-light blocklist, multi-spec require_sdk), and a pointer to wip/plugins/phase-2-review-deferred.md flagging the two footguns that bear on Phase 3 (build_registrar dropping cli/slot/teardown collections; search reporting_delegate default None). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o integration) Invert orchestrator dispatch so the runtime bridge, boot/teardown, and CLI name no integration — by import or by string. Dispatch now goes through a plugin-contributed OrchestratorRegistry keyed by execution mode. - bridge.py: the `match execution_mode:` collapses to get_orchestrator_registry().get_optional(mode).run(...), else MissingOrchestratorError(mode). DIRECT/Temporal/Mistral bodies extracted verbatim into DirectOrchestrator (core plugin) and the in-tree Temporal plugin's orchestrators; serialize helpers moved to runtime_bridge/serialization.py to break the bridge→bootstrap→pipelex import cycle. - pipelex.py: the four `if temporal.is_enabled:` boot blocks become _resolve_hub_slot with precedence explicit-param > plugin slot-claim thunk > core default (codex C8); teardown runs registered callbacks LIFO. No temporal import remains. - _cli.py: the two hard temporal-command imports + static registration are gone; commands are harvested at CLI-build via build_registrar. add_cli_command now declares an import_path (resolved dynamically) so a builtin plugin can contribute a Pipelex-booting command without an import cycle. The harvest config load is side-effect-free with a package-default fallback. - Temporal plugin (in-tree, BUILTIN): always contributes the TEMPORAL_* orchestrators + worker/setup-temporal-namespace commands; iff temporal.is_enabled, claims the content-generator/task-manager/router/run hub slots and a teardown callback — each a deferred thunk, so register imports no temporalio (D5). - MissingOrchestratorError(mode) replaces MissingPipelexTemporalExtraError + MissingMistralWorkflowsPluginError, mapping each mode to its exact install hint (codex C7). - Orchestrator SPI published (docs/under-the-hood/orchestrator-plugins.md). Tests: orchestrator dispatch-by-mode (fake) + per-mode error parity, injection-precedence + teardown-LIFO, CLI-command contribution + broken-config harvest fallback; import-light guard extended with temporalio. Full agent-test green; Temporal integration suite green through the seam (156 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al console script - Updated documentation and test references to reflect the renaming of CLI commands from `pipelex setup-temporal-namespace` to `pipelex-temporal setup-namespace`. - Introduced a new smoke test for the `pipelex-temporal` console script to ensure it exposes the `worker` and `setup-namespace` commands. - Removed the CLI-command contribution seam, transitioning the `worker` and `setup-temporal-namespace` commands to a dedicated `pipelex-temporal` script. - Updated various code paths and tests to accommodate the new command structure and ensure compatibility with the existing functionality. - Documented the decision to drop the CLI-command seam and the rationale behind moving to a standalone console script for Temporal operations.
Option A (drop the CLI-command harvest seam; Temporal worker/setup-namespace ship as the standalone pipelex-temporal console script) landed as 989c9be. Flip the plan doc to a faithful as-built record (status implemented, checkpoint PASSED with gate results) and add the "Option A — as-built" note + cold-start status to TODOS.md so a fresh session can resume cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tch seam) Collapse model_lists.py's `match sdk:` into a registry lookup, so core names no integration across every enumerated seam. Each backend plugin now registers an optional ListModelsFn via add_model_lister, mirroring the inference-backend and orchestrator registries from Phases 1-3. - New plugins/model_lister_registry.py: ListModelsFn + ModelListerRegistry keyed by sdk (get_optional -> a soft miss = unsupported-for-listing, not an error). - registrar.add_model_lister + model_listers accumulator; DuplicateModelListerError. - ModelListingUnsupportedError (core soft signal); the anthropic lister translates its vendor AnthropicSDKUnsupportedError into it, so core names no anthropic symbol. - Hub + pipelex.py setup() wire the registry; the five vendor plugins grow lister closures (openai x4 keys, bedrock x2; mistral/bedrock sync, the rest async). - Behavior byte-equivalent: the find_spec guards already live inside each list_*_models fn; any_listed threads identically; failures still wrap to PipelexCLIError. Tests: registry round-trip coverage + behavioral dispatch + import-light guard. Docs: the inference-backend SPI page gains the model-listing capability; two error pages regenerated. CHECKPOINT 4: agent-check clean, agent-test green, /code-review byte-equivalent (no blockers). Three pre-existing unenumerated core->vendor couplings dispositioned in wip/plugins/phase-4-residual-core-vendor-couplings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…elper + count cleanup Post-checkpoint follow-ups from the xhigh workflow /code-review of Phase 4 (6a07965). The review re-confirmed the model-listing inversion is byte-equivalent to the old `match`; these are the two cheap reuse/altitude cleanups it surfaced, plus a deferred-tradeoffs record. - registrar: collapse the three add_* menu methods (add_inference_backend / add_model_lister / add_orchestrator) onto a shared generic _add() that owns the dup-check -> store -> source-attribution -> contribution sequence, mirroring the existing _claim() helper. Each method supplies its keyed store, parallel sources dict, contribution label, and an on_duplicate factory for its distinctly-typed Duplicate*Error. A future 6th seam reuses _add instead of a fourth copy. Typed via _RegistryKeyT/_RegistryValueT — no type safety lost. - TODOS.md: drop the hardcoded item-counts from the Phase 4 as-built (workspace "Never hardcode counts" rule); add a pointer to the new follow-ups doc. - wip/plugins/phase-4-review-followups.md: record the four design-tradeoff findings deferred (listing/disable coupling, bedrock key skew, any_listed SPI leak, ListModelsFn loose typing), each with a why-deferred + revisit-if. Gates: make agent-check clean (pyright 0 / mypy 0 / keyword-only pass); targeted plugins + model-backends suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019aGUg464jUnCsALuf88r89
…hema to core The Temporal config *schema* must leave the externalizable `pipelex.temporal` namespace before `pipelex/temporal/` can move out, because core `configs.py` imports it unconditionally. This is the prerequisite the design missed (codex C6); it makes the rest of Phase 5 a packaging move. - git mv pipelex/temporal/config_temporal.py -> pipelex/system/configuration/config_temporal.py (whole schema; already import-light via the TYPE_CHECKING RetryPolicy=Any placeholder). - New pipelex/system/configuration/exceptions.py holds the two exceptions the schema raises: TemporalConfigError(ValueError, PipelexError) + WorkerTaskQueueUnknownError. Re-based off PipelexError (not the runtime TemporalFlowError, which stays with pipelex/temporal/ for the cut-over); byte-equivalent (error_domain=None, ValueError mixin, isinstance/str/args all unchanged — TemporalFlowError is never imported/caught anywhere). - temporal/exceptions.py keeps the runtime/workflow errors + the worker-boot config subclasses, which now subclass the core TemporalConfigError. - Repointed every importer (configs.py:14 + temporal source + ~20 test modules); the one hard pipelex.temporal import in core config is gone. - Regenerated docs/errors (2 moved error pages + the temporal->system macro regrouping); updated the two optional-dep regression tests' module path. Behavior-neutral, in-pipelex only, zero external importers of the moved symbols. Gates: make tb · make agent-check (pyright 0/mypy clean) · make agent-test green. Flagged (pre-existing, deferred): (ValueError, PipelexError) exceptions never set .message so to_error_report() raises — also affects the untouched ConfigModelError. See wip/plugins/valueerror-mixin-message-unset-latent-bug.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019aGUg464jUnCsALuf88r89
ValueError.__init__ (the BaseException C slot) shadowed PipelexError.__init__ in the MRO, so self.message was never set and to_error_report() raised AttributeError on these instances. Reorder bases so the message-setting PipelexError base resolves first: - TemporalConfigError(PipelexError, ValueError) - ConfigModelError(FatalError, ValueError) The Worker*/SearchAttribute* subclasses inherit the fix transitively; the two (PipeComposeError, ValueError/TypeError) compose mixins already had the safe order. isinstance(_, ValueError) is unaffected by base order, so Pydantic still wraps validator raises into ValidationError; str()/.args unchanged. One behavior change, at one site: ConfigModelError now runs the restored TracebackMessageError.__init__, so it logs at its single misuse raise site (ConfigModel.transform_dict_str_to_enum) — appropriate fatal-error behavior, matching its ConfigValidationError/FatalError siblings. Add tests/unit/pipelex/test_pipelex_error_message_init.py: an MRO guard that sweeps every loaded PipelexError subclass asserting PipelexError precedes ValueError/TypeError, plus .message/to_error_report() regression assertions. Surfaced by the xhigh /code-review of Phase 5 Step 0b (C6); the deferred wip note is removed and the TODOS.md as-built flipped to FIXED. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019aGUg464jUnCsALuf88r89
…ipelex-temporal (pre-publish)
Move the Temporal integration out of core into the new sibling distribution
`pipelex-temporal` (top-level package `pipelex_temporal`), discovered through the
existing `pipelex.plugins` entry-point seam. Local, reversible work only — the
downstream pin flips and publishing stay deferred (publish/release-gated).
Core cut:
- Drop TemporalPlugin from BUILTIN_PLUGINS (temporal is now an external entry-point dist)
- Remove the `temporal` extra, the `pipelex-temporal` console script, and the
`temporal` pytest marker from pyproject.toml
- git rm pipelex/temporal/ + the temporal unit/integration test trees
- MissingOrchestratorError install hint: 'pipelex[temporal]' -> pipelex-temporal (+ tests)
- Stale `pipelex[temporal]` docstrings/comments -> pipelex-temporal
Test reconciliation:
- Split the mixed bridge/boot tests: core keeps the DIRECT/core-mode methods, the
temporal-mode methods travel to pipelex-temporal
- Keep test_cv_batch_screening (DIRECT e2e) in core, decoupled from the temporal tree
- Restore the shared .mthds bundles to tests/integration/pipelex/error_handling/bundles/
- Relocate test_config_temporal_optional_dep + test_non_retryable_baseline_pins back
to core (they validate the core-relocated config schema)
Option-A fold-in: delete dead load_base_config_dict / ensure_global_if_missing.
Docs: install instructions + removed error-name references -> pipelex-temporal.
Gates: make agent-check clean (pyright 0 errors / mypy 0 errors / keyword-only),
make tb green, full make agent-test green ("All tests passed").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019aGUg464jUnCsALuf88r89
…gacy references and update documentation
- Deleted `repro_runner_registry_bug.py` and `submit_dry_validate.py` scripts from the `.claude/skills/temporal-e2e-validate/scripts` directory as they are no longer needed. - Removed the `temporal-test-crate` skill documentation from `SKILL.md`. - Cleaned up the Makefile by removing all Temporal-related targets and commands, including those for starting the Temporal server and running tests. - Updated `TODOS.md` to reflect the removal of Temporal skills and targets, and to document the finalization of the local pass for the core and `pipelex-temporal`. - Adjusted `wip/plugins/phase-5-cutover-review-followups.md` to indicate the successful relocation of Temporal skills and the removal of dead tooling from the core.
- Adjusted mypy settings in .vscode/settings.json for improved configuration. - Introduced a new HTML document detailing the Pipelex Plugin System and its architecture, including the transition to a plugin-based approach for optional integrations.
… distribution status
…date (Phase V0) Mirror the per-call OrchestratorRegistry seam for /validate: a BundleValidatorProtocol + BundleValidatorRegistry keyed by PipelexExecutionMode, an add_bundle_validator registrar method (reusing the _add dup-guard), the hub getter + boot wiring, MissingBundleValidatorError with per-mode install hints, and a core DirectBundleValidator (registered by the direct plugin) that validates in-process and returns the verdict as a value — PipelexValidationReport on valid, the ValidateBundleError-derived ErrorReport on invalid. The seam's valid arm is typed at the MTHDS-protocol ValidationReport (a leaf type), not the concrete PipelexValidationReport, so the hub-reachable registry stays import-acyclic (reportImportCycles gates on TYPE_CHECKING edges too) — mirroring how the orchestrator seam returns the leaf PipelexPipeRunOutput. The concrete envelope is recovered at the API edge. DirectBundleValidator still returns the precise type (a covariant narrowing). First step of reversing/extending decision F2 (validate always DIRECT in-process): adds the generic, orchestrator-agnostic dispatch seam. The Temporal validator and the API dispatch+route land in their own repos. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgkGZcadLLZfWMH8br4XAZ
…t + reverse F2) Mark the parent plan's F2 decision REVERSED/EXTENDED (in the orchestrator-agnostic-runner plan and in TODOS: locked-decisions F2 + the Phase C as-built). Append the V2 (pipelex-api dispatch+route+F2 reversal) and V3 (bookkeeping/spec/conformance) as-built and the V-A/V-B checkpoint records to the orchestrator-dispatched-validate plan. Capture the clean-context review's one deferred design tradeoff (start/validate_verdict mode-dispatch duplication). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgkGZcadLLZfWMH8br4XAZ
…import it Greptile P1: the verdict alias was bound only under TYPE_CHECKING, so an external validator plugin that annotates its validate_bundles with BundleValidationVerdict would fail at import (the name is unbound at runtime). Both arms — the MTHDS-protocol ValidationReport and base_exceptions ErrorReport — are leaf types w.r.t. the hub, so the alias binds at module-level runtime without re-introducing the import cycle that keeps the concrete PipelexValidationReport off this hub-reachable seam. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgkGZcadLLZfWMH8br4XAZ
…lidate (all deferred) Capture the gstack pre-landing /review of PR #998 across all three repos. Verdict: landable, no critical findings. All five informational findings deferred (none a clear win): MISTRAL_NATIVE /validate unserviceable (out of scope — Phase V-Mistral), the verdict-narrowing cast vs a future bare-ValidationReport (excluded future-plugin guard; fails loud, type-checked for in-tree validators), and three test-coverage symmetry gaps (transitively covered / untested-by-convention). Each carries a revisit trigger; the three pre-existing deferred notes were re-confirmed, not re-opened. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgkGZcadLLZfWMH8br4XAZ
…leted validate work The sibling /validate plan is now done and changed pipelex-api/pipeline.py (dropped the ApiRunner.validate override → validate_verdict, shifted line numbers, adjusted imports). Refresh the execute plan's §2 anchors to the post-validate tip (72c0efc), recommend branching the execute work off that tip (not the stale a39841e) to stack cleanly and avoid a pipeline.py merge, and cross-reference the boot-active executor constraint the validate work surfaced (relevant when /execute dispatches temporal_blocking). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgkGZcadLLZfWMH8br4XAZ
…ndings (Phase E0) `/execute` now dispatches by execution_mode (implemented in pipelex-api on feature/Execute-per-request-mode, off the validate tip). Update the plan's status to Phase E0 DONE with an As-built section (final signature, the output-mapping hinge decision, test evidence), and add the consolidated deferred/flagged-findings note: the accepted DIRECT serialize→rehydrate round-trip, the strict=False rationale, a pre-existing /start DIRECT-path resource leak (flagged, not fixed), absorbed pre-existing /validate OpenAPI + config-doc drift, plus the independent code-review pass's altitude / OpenAPI-schema / is_completed findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
review) Greptile (P1) / Codex (P2) flagged that /execute's OpenAPI request schema omitted the execution_mode override it now honors. Fixed for /execute (PipelexApiExecuteRequest); deferred-findings §6 now records the /execute half as resolved and narrows the deferral to the pre-existing /start half (out of this PR's scope), with the exact follow-up fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
…(PR #27 Codex P2) Codex flagged that /execute's advertised execution_mode references #/$defs/PipelexExecutionMode, unresolvable at the document root. It is a pre-existing artifact-generation pattern (/execute and /start already emit dangling #/$defs/ refs via openapi_extra + model_json_schema; only typed-body routes like /validate get hoisted to components). Recorded the deferral + the cross-endpoint clean fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
…y (Phase E0) gstack pre-landing /review: /execute-by-execution_mode is merge-ready, no blockers, no code fixes warranted. Capture the one new deferred finding (#9): a failing-pipe /execute now double-wraps the error message + carries an unwound pipe_stack (pipe failures route through DirectOrchestrator → PipelexBridgeDispatchError, hitting the base's except PipelexError arm rather than except PipeRouterError) — message/log fidelity only, no wire-status/classification regression, and a property of the shared orchestrator seam spanning /start and /validate too, so a cross-endpoint follow-up, not a /execute fix. Plus minor no-action notes. Mark the plan status MERGE-READY. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
…ollow-ons DONE
Correct the stale status block ("commit pending / nothing pushed") to reflect Phase C
committed (a39841e) + pushed, and add a "Follow-on extensions" section tracking the two
post-C plans now done: orchestrator-dispatched /validate (reverses/extends F2; 3 repos on
feature/Orchestrator-dispatched-validate) and per-request execution_mode on /execute
(extends F1; PR #27, merge-ready). Restate current branches + the editable-pin CI gotcha.
NEXT = Phase D, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
Paste-ready session-resume orientation for the orchestrator-agnostic-runner effort: what's true, commands to verify it against the repos (branches/tips/PR #27/gates), what's done, and NEXT = Phase D. Linked from the TODOS intro. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
…w fixes (7e4bc13) A third post-Phase-C follow-on landed on feature/Execute-per-request-mode: /start now derives the fire-and-forget sibling of the deployment's single synchronous execution_mode (temporal_blocking -> temporal_fire_and_forget; direct/mistral dispatched unchanged), so one configured mode stays coherent across /start, /execute, and /validate. Records the xhigh-workflow /code-review of f5e3ef7 and the fixes it drove (7e4bc13): ApiConfig rejects a configured f&f mode (fail-fast at load), corrected the mistral_native workflow_id docs/comments (non-null run id, not None), and added the missing pure-mapping + mistral_native end-to-end tests. Genericized the "two follow-ons" counts and refreshed the branch/tip + push state (f5e3ef7 + 7e4bc13 + this update are not pushed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bktem3rLWRE2R7dmkEfbf
…hestration_mode` and `DeliveryMode`
…pen token) + DeliveryMode (closed enum)
Replace the flat PipelexExecutionMode enum — which conflated *which orchestrator
runs the pipe* with *whether the caller waits* — with two orthogonal axes:
- orchestration_mode: an OPEN string token. Core owns only "direct"; every other
token (e.g. "temporal", "mistralai-workflows") is owned by the plugin that
registers it. The orchestrator/validator registries and the registrar stores are
re-keyed by str (OrchestrationMode alias). Validation is the registry lookup —
a miss raises a generic MissingOrchestratorError that names no orchestrator (D-F).
- DeliveryMode {BLOCKING, FIRE_AND_FORGET}: a closed core StrEnum, endpoint-set and
passed as a parameter to OrchestratorProtocol.run. Orchestrators advertise
supports_fire_and_forget so a caller can be told honestly when async is impossible.
Delete execution_mode.py (enum + its 3 properties); add orchestration_mode.py +
delivery_mode.py. Split PipelexPipeRunInput.execution_mode into orchestration_mode
(str) + delivery (DeliveryMode). bridge.py dispatches on the token and threads
delivery; _validate_input's delivery-target requirement now keys on
delivery is FIRE_AND_FORGET. Generic Missing{Orchestrator,BundleValidator}Error
messages (string compare on the core "direct" token, no enum match). DirectOrchestrator
gains supports_fire_and_forget=False and accepts+ignores delivery (in-process always
blocks). Update the orchestrator-plugins SPI doc for the two-axis model.
Phase 1 of the orchestration-mode / delivery split (core → temporal → api). Gates:
make agent-check, make tb, make agent-test all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…ivery split + Phase 1 as-built Follow-up to bd4e528 (Checkpoint 1). Clean-context /code-review flagged three user-facing Mistral-workflows doc pages still describing the deleted execution_mode / PipelexExecutionMode model (the §9 sweep had been .py-only): - distributed-execution/mistral-workflows/execution-modes.md — rewritten to the two-axis orchestration_mode (open token) + delivery model; retitled "Orchestration & Delivery"; generic missing-orchestrator wording; mistralai-workflows token. - index.md / your-first-pipelex-workflow.md / installation.md — field/token refs + link labels. - reliability/durable-execution.md — mistral_native → mistralai-workflows. Plus the Phase 1 as-built / checkpoint log appended to the plan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…tart-honesty risk Record in the orchestration-mode/delivery-split plan: Phase 2 (pipelex-temporal collapse, commit 01d7dc3) as-built and its clean-context review verdict (no material issues), and mark the §8 /start-honesty conformance RISK resolved — the MTHDS-Protocol spec defers execute/start to pipelex-api's OpenAPI/docs, no conformance test POSTs the runner /start, and the hosted platform runs the async-capable temporal mode, so honest 4xx on a direct-only base breaks nothing. Status: Phases 1+2 done; next = Phase 3 (pipelex-api, THE gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…api) Capture a fresh-session handoff for the orchestration-mode/delivery split: - exact branch/tip per repo (core 919c093, temporal 01d7dc3, api 06cd5e6 = clean cut point); - Phases 1+2 done + reviewed; §8 /start-honesty risk resolved; - a "⏸️ COLD-START RESUME — paused before Phase 3" block in the plan §10 with the locked Phase 3 design decisions (honest /start = 400; restructure start() to capability-check before pipeline_run_setup; _OrchestratorPipeRun gains delivery; resolve_orchestration_mode; ApiConfig/error_types/models/api.toml renames; tests + openapi + docs) and the ready-to-execute checklist pointing at §3.3/§9; - parent TODOS.md follow-on bullet flipped to IN PROGRESS. pipelex-api branch is cut off 06cd5e6 with a clean tree (two partial edits reverted) so Phase 3 starts from scratch against the plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…ators The bridge forwarded delivery=FIRE_AND_FORGET to any resolved orchestrator without consulting supports_fire_and_forget, so direct + fire-and-forget + a valid delivery target passed the delivery-target gate, ran blocking to completion, returned is_completed=True, and never invoked the delivery target — block + false-ack + silent drop. Found by the combined Phases 1-2 clean-context /code-review; resolves the §8 Tier-3 open question. Resolve the orchestrator up front and add a capability gate in _validate_input (checked before the delivery-target gate) that rejects FIRE_AND_FORGET on any orchestrator with supports_fire_and_forget=False — the Tier-3 counterpart of /start's D-E check. A valid delivery target does not rescue a blocking-only orchestrator (the mode, not the target, is the problem). This is the first read site of supports_fire_and_forget in landed code. - bridge.py: resolve orchestrator before the scoped library / pipe-job build; _validate_input gains the orchestrator param + the capability gate - tests: split FF validation into capability-vs-target gates; seam test proving a blocking-only orchestrator's run() is never awaited for FF; trace-context nulling test updated for the resolve-first reorder - docs/under-the-hood/orchestrator-plugins.md: worked example now describes the single collapsed TemporalOrchestrator, not the deleted two-class shape - tracker: §8 Tier-3 marked RESOLVED; Phase 4 Mistral scope corrected to a real code migration (verified anchors) not a token rename; post-checkpoint hardening note + cold-start state updated Gates: make agent-check green (pyright 0 / mypy 0 / ruff / plxt / keyword-only); targeted runtime_bridge + plugins suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…oint-3 review Record the Phase 3 as-built (pipelex-api dd84a9a): the orchestration_mode/delivery split landed across api_config, error_types, wire models, routes (honest /start capability gate before library load), validate, OpenAPI, docs, CHANGELOG. Gates green (pyright/mypy 0, 319 tests, openapi-check clean); clean-context /code-review found no must-fix issues. Reconcile the pin deviation (branch was on a git-SHA, not editable as the cold-start note assumed — flipped to editable ../_plugins for local gates; carry-forward = flip to a pushed git SHA before any PR). Tracker is cold-start ready for Phase 4 (Mistral plugin migration + bookkeeping). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
…i-2x-bump Brings the orchestration_mode (open token) + DeliveryMode split into the mistralai-2x-bump core so the pipelex-mistralai-workflows plugin (which pins this worktree editable) can migrate to the new OrchestratorProtocol surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154Kn5AhmT3LfrQfA1nWLMG
Brings this branch up to dev's finished plugin/runtime_bridge architecture
("Refactor/plugins 5", #1006) while preserving this branch's unique payload,
the mistralai-2.x SDK bump. dev carried the completed form of the same
refactor this branch held in transitional state, so the resolution is
effectively "dev + mistral-2.x bump + reconciled bookkeeping".
Conflict resolution:
- runtime_bridge: took dev throughout — accepted its deletion of bridge.py and
the legacy primitives (dispatch moved to the closed pipelex-transport lib;
the in-process arm is now DirectOrchestrator, which already carries this
branch's fire-and-forget rejection). No in-repo callers dangle. Also removed
the HEAD-only test_orchestrator_dispatch.py, which imported the deleted
bridge.py (same category as the bridge tests dev deleted). The runtime_bridge
source and test trees are now identical to dev.
- test_data.py: took dev's extraction of the ErrorReport parity fixtures into
the shared pipelex.test_extras.error_report_parity source of truth.
- plugins/builtins.py: took dev (only a stale doc-comment differed).
- mistralai-2.x bump (pipelex/plugins/mistral/*): preserved intact — dev never
touched these, so they auto-merged.
- Distributed-execution docs: accepted dev's consolidation to a single
capability page; the deep self-install guides this branch had polished are
dropped (mkdocs redirect_maps already point them at the index).
- docs/errors/*: took dev; `make generate-error-pages` confirms 0 changes (the
merged error set equals dev's).
- CHANGELOG / TODOS: union-merged — kept dev's narrative and folded in the
mistral-2.x [Unreleased] entry; dev's TODOS status already narrates this
branch's merge. wip/plugins/* trackers: took dev's more-advanced state.
Gates green: make tb, make agent-check (pyright 0, mypy 0, keyword-only pass).
Follow-up (cross-repo, non-blocking): the deleted pipe_classification helpers
and the dropped bridge-dispatch test belong with pipelex-transport / the
pipelex-mistralai-workflows plugin if still needed there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JEHK3NPZw4v5NNtN2BdFdF
Greptile SummaryThis PR moves the Mistral integration to the 2.x SDK layout. The main changes are:
Confidence Score: 4/5This is close, but the packaging issue should be fixed before it reaches a release path.
pyproject.toml and pipelex/plugins/mistral/mistral_llms.py Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+19
to
+23
| known_models = [model for model in models_list if isinstance(model, (BaseModelCard, FTModelCard))] | ||
| if not known_models: | ||
| msg = "No models found" | ||
| raise MistralModelListingError(msg) | ||
| return sorted(known_models, key=lambda model: model.id) |
There was a problem hiding this comment.
BaseModelCard or FTModelCard. Since the dependency allows any mistralai>=2.4.4, a 2.x SDK response can include another valid model-card class from models.list().data; this branch would drop it before sorting and can raise MistralModelListingError("No models found") for an account that has listable models. The lister should preserve cards with the fields this code needs, or handle unknown card types without treating the response as empty.
Prompt To Fix With AI
This is a comment left during a code review.
Path: pipelex/plugins/mistral/mistral_llms.py
Line: 19-23
Comment:
**Keep valid models** The new filter only keeps objects that are instances of `BaseModelCard` or `FTModelCard`. Since the dependency allows any `mistralai>=2.4.4`, a 2.x SDK response can include another valid model-card class from `models.list().data`; this branch would drop it before sorting and can raise `MistralModelListingError("No models found")` for an account that has listable models. The lister should preserve cards with the fields this code needs, or handle unknown card types without treating the response as empty.
How can I resolve this? If you propose a fix, please make it concise.![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e30c09e598
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| huggingface = ["huggingface_hub>=0.23,<1.0.0"] | ||
| linkup = ["linkup-sdk>=0.12.0"] | ||
| mistralai = ["mistralai>=1.12.0"] | ||
| mistralai = ["mistralai>=2.4.4"] |
There was a problem hiding this comment.
When this package is installed from wheel/sdist or with an installer that does not honor [tool.uv.sources], this extra now pulls mistralai 2.x while the published metadata still only requires released instructor>=1.13. The Mistral structured-output path unconditionally calls instructor.from_mistral(...), and the commit’s own notes say the compatible instructor fork is only supplied through uv sources, so pipelex[mistralai] consumers outside that uv-locked setup get a broken structured Mistral install as soon as this metadata is published or installed without sources.
Useful? React with 👍 / 👎.
Removes the feature_config layer (always use ReportingManager), streamlines interpreter TOML syntax-error reporting, and tidies the wip/ folder (history move). # Conflicts: # wip/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings the mistralai 2.x SDK bump to
dev. The Mistral provider (LLM + OCR/extract workers, factory, config) is migrated to the mistralai 2.x client layout (mistralai.client.*), makingpipelex[mistralai]co-installable with the Mistral Workflows SDK (mistralai-workflows), which resolves mistralai 2.x.devhas been merged into this branch (merge commit), so this PR is conflict-free and up to date. The net contribution overdevis the mistral 2.x bump plus reconciled bookkeeping — dev's "Refactor/plugins 5" (#1006) already carried the finished form of this branch's runtime_bridge / orchestration-mode-delivery-split work, which was resolved in favor of dev during the merge.This work is gated from publish until 567-labs/instructor#2298 ships to PyPI. Until then,
instructoris pinned to a git fork via[tool.uv.sources]. Local/editable use is unaffected, but this must not be merged into a release line and shipped to PyPI before that upstream release lands.Verification
make tb✅ (boot/config load)make agent-check✅ (pyright 0, mypy 0 across 2067 files, ruff/plxt/keyword-only pass, no dangling imports of dev-deleted modules)make agent-test✅ (full suite)🤖 Generated with Claude Code
https://claude.ai/code/session_01JEHK3NPZw4v5NNtN2BdFdF
Summary by cubic
Upgrades
mistralaito 2.x and migrates the Mistral provider (LLM + OCR/extract) tomistralai.client.*, keepingpipelex[mistralai]co‑installable withmistralai-workflows. Removes the feature flag layer (reporting is always on), improves TOML parse error reporting, and adds WIP docs tracking the 2.x bump and publish gate.Dependencies
mistralai>=2.4.4(lock resolves 2.4.9); switch imports tomistralai.client.*across provider and tests.instructorto a git fork via[tool.uv.sources]; do not release until upstream adds 2.x support (feat: support mistralai v2 567-labs/instructor#2298).mistralai.client.errors,.utils,.types; adjust model listing to returnBaseModelCard | FTModelCardand filter unknown types.Bug Fixes
messageis None and surface a transient error to trigger retry.Written for commit efb1af2. Summary will update on new commits.