API use plugin system

[lchoquel]

Summary by cubic

Make the API orchestrator‑agnostic and add per‑request backend selection. Also fixes startup teardown so a config warmup failure cleanly shuts down the Pipelex singleton.

• New Features

  • New deployment config in api/api.toml: orchestration_mode (open token; default direct) and allow_request_orchestration_mode_override (default false).
  • Per-request orchestration_mode supported on /execute, /start, and /validate when allowed; unknown tokens are rejected.
  • /execute and /validate run with blocking delivery; /start is fire‑and‑forget and returns 400 when the resolved backend cannot run async (e.g. direct).
  • Dispatch goes through the OrchestratorRegistry and BundleValidatorRegistry; the base imports no orchestrator SDKs.
  • Error handling uses plugin‑provided HTTP error mappers; new error types for override policy and start‑mode capability.
  • OpenAPI/docs updated.
  • Dev: added .codex environment and .gitignore entries for local workflows.

• Migration

  • The base image no longer includes Temporal; all runs are in‑process by default. To use distributed execution, add an orchestrator plugin (e.g. pipelex-temporal) and set orchestration_mode accordingly in ~/.pipelex/api_override.toml (or env‑specific files).
  • Remove the [temporal] block from .pipelex/pipelex.toml; orchestration is now controlled via api.toml.
  • If you need per-request backend selection, set allow_request_orchestration_mode_override = true.
  • On direct, use /execute until an async‑capable orchestrator is configured; /start will return 400.

^{Written for commit d4c3095. Summary will update on new commits.}

[greptile-apps]

Greptile Summary

This PR makes pipeline execution choose an orchestration backend per deployment or per allowed request. The main changes are:

• Added api.toml and ApiConfig for orchestration_mode and override policy.
• Routed /execute, /start, and /validate through orchestrator and validator registries.
• Removed direct Temporal imports from the base API path.
• Added plugin-provided HTTP error mapper registration.
• Updated request schemas, OpenAPI docs, user docs, and unit tests for orchestration mode.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
api/api_config.py	Adds deployment config loading and per-request orchestration mode override policy.
api/routes/pipelex/pipeline.py	Moves execute, start, and validate dispatch through orchestration registries.
api/exception_handlers.py	Replaces built-in Temporal exception handling with plugin-provided HTTP error mappers.
api/main.py	Warms API config at startup and registers plugin HTTP error mappers.
api/schemas/models.py	Adds request-level orchestration mode fields to API schemas.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[API request] --> B[Read request orchestration_mode]
    B --> C[Resolve against api.toml policy]
    C --> D{Endpoint}
    D -->|/execute| E[OrchestratorRegistry.run BLOCKING]
    D -->|/start| F[Check supports_fire_and_forget]
    F --> G[OrchestratorRegistry.run FIRE_AND_FORGET]
    D -->|/validate| H[BundleValidatorRegistry.validate]
    E --> I[Canonical execute response]
    G --> J[Start ack]
    H --> K[Validation verdict envelope]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[API request] --> B[Read request orchestration_mode]
    B --> C[Resolve against api.toml policy]
    C --> D{Endpoint}
    D -->|/execute| E[OrchestratorRegistry.run BLOCKING]
    D -->|/start| F[Check supports_fire_and_forget]
    F --> G[OrchestratorRegistry.run FIRE_AND_FORGET]
    D -->|/validate| H[BundleValidatorRegistry.validate]
    E --> I[Canonical execute response]
    G --> J[Start ack]
    H --> K[Validation verdict envelope]

Loading

_{Reviews (1): Last reviewed commit: "refactor: orchestration_mode + endpoint-..." | Re-trigger Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ddcb31b48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Publish the actual Pipelex source requirement

This change uses APIs from the unreleased Pipelex git revision, but the package metadata still tells non-uv installers to resolve PyPI pipelex==0.35.0; [tool.uv.sources] is uv-specific and is not part of the dependency requirement consumed by standard pip/build installs. Any environment installing this project without uv source handling will get 0.35.0 and then fail at import/startup when the new pipelex.hub/runtime_bridge symbols are missing, so the dependency should be a PEP 508 direct reference or a released version that contains these APIs.

Useful? React with 👍 / 👎.