diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
new file mode 100644
index 0000000..e707a4e
--- /dev/null
+++ b/.github/workflows/docs.yml
@@ -0,0 +1,45 @@
+name: ci
+on:
+ push:
+ branches:
+ - main
+ pull_request:
+ branches:
+ - main
+permissions:
+ contents: read
+jobs:
+ check:
+ if: github.event_name == 'pull_request'
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: 3.x
+ - uses: astral-sh/setup-uv@v6
+ - run: PYTHONPATH=src uv run --no-project --with mkdocs --with mkdocs-material --with mkdocstrings-python mkdocs build --strict
+
+ deploy:
+ if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write
+ steps:
+ - uses: actions/checkout@v4
+ - name: Configure Git Credentials
+ run: |
+ git config user.name github-actions[bot]
+ git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+ - uses: actions/setup-python@v5
+ with:
+ python-version: 3.x
+ - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+ - uses: actions/cache@v4
+ with:
+ key: mkdocs-material-${{ env.cache_id }}
+ path: ~/.cache
+ restore-keys: |
+ mkdocs-material-
+ - run: pip install mkdocs-material mkdocstrings-python
+ - run: PYTHONPATH=src mkdocs gh-deploy --force --strict
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..8f6c219
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1 @@
+Refer to the documentation in the `docs/` directory to learn more about the project and how to make contributions.
diff --git a/README.md b/README.md
index 5051e40..0edfbc1 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,9 @@
Reproducible benchmarks for coding agents and models using Harbor
+[](https://redhat-et.github.io/coding_agent_bench/)
+[](https://github.com/redhat-et/coding_agent_bench/tags)
+
## Features
- Complete instructions for running popular benchmarks with open models
@@ -10,47 +13,6 @@ Reproducible benchmarks for coding agents and models using Harbor
- Leaderboards for popular benchmarks with instructions for reproducing results
- Full manifests for deploying open models on OpenShift with vLLM
-## Table of Contents
-
-- [Features](#features)
-- [Leaderboards](#leaderboards)
- - [SWE-Bench Verified (pass@1, N=500)](#swe-bench-verified-pass1-n500)
- - [SWE-Bench Pro - Ansible Tasks (pass@1, N=96)](#swe-bench-pro---ansible-tasks-pass1-n96)
- - [Terminal Bench 2.0 (pass@1, N=87)](#terminal-bench-20-pass1-n87)
-- [CLI Utility](#cli-utility)
- - [Prerequisites](#prerequisites)
- - [Run a Benchmark](#run-a-benchmark)
- - [Use Agent Skills](#use-agent-skills)
-- [Queue Service](#queue-service)
- - [Set up the service](#set-up-the-service)
- - [Use the service](#use-the-service)
- - [(Optional) Connect to Nebius](#optional-connect-to-nebius)
-- [Harbor Command Examples](#harbor-command-examples)
- - [Claude Code vLLM](#claude-code-vllm)
- - [Codex vLLM](#codex-vllm)
- - [OpenClaw vLLM](#openclaw-vllm)
- - [OpenCode vLLM](#opencode-vllm)
- - [OpenHands vLLM](#openhands-vllm)
- - [Pi vLLM](#pi-vllm)
- - [Qwen Code vLLM](#qwen-code-vllm)
- - [Claude Code Anthropic](#claude-code-anthropic)
- - [Claude Code VertexAI](#claude-code-vertexai)
- - [Codex OpenAI](#codex-openai)
-- [Deploy models with vLLM](#deploy-models-with-vllm)
-- [SWE-Bench Acceleration](#swe-bench-acceleration)
- - [Use accelerated images for SWE-bench-verified](#use-accelerated-images-for-swe-bench-verified)
- - [Pre-pull base images](#pre-pull-base-images)
-- [Run with Openshift](#run-with-openshift)
- - [Run Tasks in Openshift (Orchestrate Locally)](#run-tasks-in-openshift-orchestrate-locally)
- - [Run Tasks and Orchestrate in Openshift](#run-tasks-and-orchestrate-in-openshift)
-- [WIP](#wip)
- - [Run with Gemini and Gemini CLI](#run-with-gemini-and-gemini-cli)
- - [Run with vLLM and Gemini CLI](#run-with-vllm-and-gemini-cli)
-- [Developers](#developers)
- - [Bumping the Project Version](#bumping-the-project-version)
- - [Deploying Changes](#deploying-changes)
-
-
## Leaderboards
@@ -101,656 +63,6 @@ More coming soon...
† - Cost estimates for OSS models are calculated by ($4 per A100 GPU hour × agent benchmark duration).
-## CLI Utility
-
-The CLI utility will help you configure and run a benchmark jobs with Harbor for self-hosted models.
-It automatically constructs and runs the Harbor job command for your specified benchmark, agent, self-hosted model.
-
-### Prerequisites
-
-- Install dependencies with uv
-
- ```bash
- uv sync
- ```
-
-- [Set up a vLLM server](#deploy-models-with-vllm), or other Anthropic- and OpenAI-compatible server
-- Select a benchmark from among the options in [Harbor Hub](https://hub.harborframework.com/)
-
-### Run a Benchmark
-
-The following is the minimal configuration needed to run a job with the CLI:
-
-```sh
-uv run coding-agent-bench run \
- --agent \
- --dataset \
- --model-name \
- --server-url
-```
-
-For example, to run `swe-bench/swe-bench-verified` in Claude Code against a self-hosted model:
-
-```sh
-uv run coding-agent-bench run \
- --agent claude-code \
- --dataset scale-ai/swe-bench-pro \
- --model-name my-model \
- --server-url http://my.server.url
-```
-
-If you want to see a preview of Harbor command that would be run for a given set of arguments without actually running the job, add the `--dry-run` flag.
-
-> [!note]
-> Additional configuration options are available, use `uv run coding-agent-bench run --help` to see them.
-
-### Use Agent Skills
-
-Pass one or more skill directories or Git sources with the repeatable `--skill`
-option (`--skills` is an alias). Harbor installs the resolved skills into the
-agent used by the benchmark.
-
-To test a skill from your local filesystem:
-
-```sh
-uv run coding-agent-bench run \
- --agent opencode \
- --dataset swe-bench/swe-bench-verified \
- --model-name my-model \
- --server-url http://my.server.url \
- --skill ./my-skills
-```
-
-Git sources make skills easy to share and reproduce. Repeat the option to test
-multiple skill collections, for example Superpowers together with Caveman:
-
-```sh
-uv run coding-agent-bench run \
- --agent claude-code \
- --dataset swe-bench/swe-bench-verified \
- --model-name my-model \
- --server-url http://my.server.url \
- --skill obra/superpowers \
- --skill juliusbrussee/caveman
-```
-
-Harbor accepts `org/name` and `org/name@ref` shorthand and HTTP(S) Git URLs.
-Repository shorthand loads skills from the repository's `skills/` directory.
-Use a full URL such as
-`https://github.com/org/repo/tree/[/` to select another directory.
-Pin a tag or named branch with `@ref` (or in the full URL) when comparing
-benchmark runs. Harbor resolves the reference to a commit and records that
-commit in the job lock file for reproducibility.
-
-## Queue Service
-
-The queue service is a FastAPI application that can be deployed on OpenShift to queue and run benchmarks automatically.
-Benchmark results are stored to MinIO for later review.
-
-```mermaid
-sequenceDiagram
- Requestor->>Queue Service: Request Benchmark Run
- Queue Service->>Queue Service: Wait in Queue
- Queue Service->>Openshift Job: Start Benchmark Job
- Openshift Job->>Harbor Orchestrator Pod: Start Harbor Run
- loop For Each Task
- Harbor Orchestrator Pod->>Task Pod: Run Task Pod
- Task Pod->>Harbor Orchestrator Pod: Save Results
- end
- Harbor Orchestrator Pod->>MinIO: Save Benchmark Results
- Harbor Orchestrator Pod->>Openshift Job: Complete
- Queue Service-->>Openshift Job: Poll for completion
- Openshift Job->>Queue Service: Complete
- Queue Service->>Openshift Job: Cleanup
- Queue Service->>Queue Service: Start Next Job
-```
-
-### Set up the service
-
-1. Log in to your cluster and project:
- ```sh
- oc login --server= --token=
- oc project
- ```
-2. Create the MinIO service for artifact storage:
- ```sh
- oc apply -f deploy/harbor-minio.yml
- ```
- Note: the default username and password are `(minioadmin, minioadmin)`.
- You can update this in the deployment file if needed.
-3. Create the orchestrator and task service accounts:
- ```sh
- oc apply -f deploy/harbor-orchestrator-sa.yml
- oc apply -f deploy/harbor-task-sa.yml
- ```
-4. Create a secret file named `job-queue-secret` with the queue service's
- `API_KEY` and any queue or Nebius settings, then apply it:
- ```yaml
- apiVersion: v1
- kind: Secret
- metadata:
- name: job-queue-secret
- stringData:
- API_KEY:
- type: Opaque
- ```
- If the intake CronJob is deployed, create its separate poller secret:
- ```yaml
- apiVersion: v1
- kind: Secret
- metadata:
- name: intake-poller-secret
- stringData:
- JOB_QUEUE_URL: https://
- GOOGLE_SHEET_ID:
- SENDER_EMAIL: ace-model-evals@redhat.com
- AUTO_APPROVE: 'false'
- type: Opaque
- ```
-5. Create the queue service:
- ```sh
- oc apply -f deploy/job-queue-service.yml
- ```
-6. (Optional) To run jobs against OpenRouter (`server_url: openrouter`), create
- an `openrouter-api-key` secret. Job pods mount it automatically (it is
- optional, so non-OpenRouter jobs are unaffected):
- ```yaml
- apiVersion: v1
- kind: Secret
- metadata:
- name: openrouter-api-key
- stringData:
- OPENROUTER_API_KEY:
- type: Opaque
- ```
- The queue service itself also needs `OPENROUTER_API_KEY` in its environment
- to validate OpenRouter jobs at request time. Add it to `job-queue-secret`
- (which the service already loads) or `envFrom` the `openrouter-api-key`
- secret in `deploy/job-queue-service.yml`.
-
- The queue listens on HTTPS inside the cluster. OpenShift's service-serving
- certificate operator creates the `job-queue-tls` Secret referenced by the
- Deployment, and the Route uses re-encryption so traffic remains encrypted
- from the router to the queue pod. Wait for that Secret to appear before
- troubleshooting pod startup:
- ```sh
- oc get secret job-queue-tls
- ```
-
-Get the route for the deployed service:
-
-```sh
-oc get route job-queue-route --output jsonpath='{.spec.host}'
-```
-
-Set `JOB_QUEUE_URL` in `intake-poller-secret` to this HTTPS route before
-applying `deploy/intake-cronjob.yml`.
-
-Check that the application is live by visiting the docs:
-
-```sh
-export JOB_QUEUE_URL="https://$(oc get route job-queue-route --output jsonpath='{.spec.host}')"
-open $JOB_QUEUE_URL/docs
-```
-
-### Use the service
-
-Queue up a new benchmark task. Skills submitted to the queue must be public Git
-sources because the OpenShift job cannot access paths on the requestor's local
-filesystem:
-
-```sh
-curl -X POST $JOB_QUEUE_URL/jobs \
- -d '{"job_name": "test", "agent": "pi", "dataset": "swe-bench/swe-bench-verified", "model_name": "qwen3.6-27b", "server_url": "", "n_tasks": 1, "skills": ["obra/superpowers@]["]}' \
- -H "Content-Type: application/json" \
- -H "X-API-Key: "
-```
-
-```json
-{
- "message":"Job created.",
- "job_id":"b5ef13c8-8909-4bf1-b5b1-43354e9f395c",
- ...
-}
-```
-
-View the queued/running/completed tasks:
-
-```sh
-open $JOB_QUEUE_URL/ui
-```
-
-Or list them from the API:
-
-```sh
-curl $JOB_QUEUE_URL/jobs -H "X-API-Key: "
-```
-
-Cancel a running or queued job:
-
-```sh
-curl -X DELETE $JOB_QUEUE_URL/jobs/ -H "X-API-Key: "
-```
-
-### (Optional) Connect to Nebius
-
-The queue service supports starting and stopping vLLM server instances automatically using [Nebius](https://nebius.com/).
-When enabled, the queue service will create a Nebius VM, SSH into the instance, start the vLLM server for the next model in the queue, run the benchmark job against that model, swap the running model for the next model in the queue, then spin down the Nebius VM when the queue is empty.
-
-To connect the queue service to Nebius, first [set up an AI Cloud account](https://docs.nebius.com/signup-billing/sign-up).
-
-After setting up the account, [install the CLI](https://docs.nebius.com/cli/install) and [login to your account](https://docs.nebius.com/cli/configure).
-
-Then run the following commands to create a service account in your project:
-
-```sh
-# Create a service account
-export SA_ID=$(nebius iam service-account create \
- --name \
- --format json | jq -r '.metadata.id')
-
-# Create and attach an authorized key to the service account
-nebius iam auth-public-key generate \
- --service-account-id $SA_ID \
- --output ~/.nebius/$SA_ID-credentials.json
-```
-
-Once the service account is created, you can update your job queue secret with the following environment variables needed for Nebius:
-
-```yaml
-apiVersion: v1
-kind: Secret
-metadata:
- name: job-queue-secret
-stringData:
- API_KEY:
- NEBIUS_ENABLED: '1'
- NEBIUS_SERVICE_ACCOUNT_CREDS: |
-
- NEBIUS_PARENT_ID:
- NEBIUS_TENANT_ID:
- NEBIUS_SERVICE_ACCOUNT_ID:
- NEBIUS_SUBNET_ID:
- NEBIUS_INSTANCE_NAME_PREFIX: job-queue-worker
- NEBIUS_IDLE_TIMEOUT_SECONDS: '600'
- HF_TOKEN:
-type: Opaque
-```
-
-When creating a job, set `server_url` to `nebius-` to use a managed Nebius instance with the specified GPU resource (e.g. `nebius-h200`, `nebius-b200`). Available resources are defined in `RESOURCE_CONFIG_REGISTRY`.
-
-## Harbor Command Examples
-
-**Prerequisites:**
-
-- Install [Harbor](https://www.harborframework.com/docs/getting-started)
-- [Set up a vLLM server](#deploy-models-with-vllm), or other Anthropic- and OpenAI-compatible server
-- Set your benchmark in your environment from among the options in [Harbor Hub](https://hub.harborframework.com/), e.g.:
-
- ```bash
- export BENCHMARK='swe-bench/swe-bench-verified'
- ```
-
-- If you need to filter tasks in your benchmark by name, add the `-i` flag with your glob pattern to your `harbor run` command, e.g. `-i "*ansible*"`
-
-**Directory:**
-
-| Harness | Model Server | Example | Status |
-| ----------- | ------------ | ------------------------------ | --------- |
-| Claude Code | vLLM | [Link](#claude-code-vllm) | Validated |
-| Codex | vLLM | [Link](#codex-vllm) | Testing |
-| OpenClaw | vLLM | [Link](#openclaw-vllm) | Validated |
-| OpenCode | vLLM | [Link](#opencode-vllm) | Validated |
-| OpenHands | vLLM | [Link](#openhands-vllm) | Validated |
-| Pi | vLLM | [Link](#pi-vllm) | Validated |
-| Qwen Code | vLLM | [Link](#qwen-code-vllm) | Validated |
-| Claude Code | Anthropic | [Link](#claude-code-anthropic) | Validated |
-| Claude Code | VertexAI | [Link](#claude-code-vertexai) | Validated |
-| Codex | OpenAI | [Link](#codex-openai) | Validated |
-
-> [!note]
-> To use with a locally hosted model (e.g. llama.cpp) use a vLLM example and set `SERVER_URL=http://host.docker.internal:`
-
-### Claude Code vLLM
-
-Set the following variables in your environ:
-
-```bash
-export SERVER_URL=
-export MODEL_NAME=
-```
-
-Then run:
-
-```bash
-harbor run --agent claude-code -d $BENCHMARK \
- --ae ANTHROPIC_BASE_URL=$SERVER_URL \
- --ae ANTHROPIC_API_KEY='sk-no-key-required' \
- --ae ANTHROPIC_MODEL=$MODEL_NAME \
- --ae ANTHROPIC_DEFAULT_OPUS_MODEL=$MODEL_NAME \
- --ae ANTHROPIC_DEFAULT_SONNET_MODEL=$MODEL_NAME \
- --ae ANTHROPIC_DEFAULT_HAIKU_MODEL=$MODEL_NAME
-```
-
-### Codex vLLM
-
-Set the following variables in your environ:
-
-```bash
-export SERVER_URL=
-export MODEL_NAME=
-```
-
-Use the utility script to create the `config.toml` file:
-
-```bash
-uv run scripts/codex_config_toml.py $MODEL_NAME $SERVER_URL
-```
-
-Then run:
-
-```bash
-harbor run --agent codex -d $BENCHMARK \
- -m vllm/$MODEL_NAME \
- --ae CODEX_HOME=/root/.codex/ \
- --mounts-json '[ { "type": "bind", "source":"/path/to/coding-agent-bench/config.toml", "target": "/root/.codex/config.toml" } ]'
-```
-
-### OpenClaw vLLM
-
-Set the following variables in your environ:
-
-```sh
-export MODEL_NAME=
-export SERVER_URL=
-export OPENAI_BASE_URL=$SERVER_URL/v1
-export OPENAI_API_KEY='NONE'
-```
-
-Then run:
-
-```sh
-harbor run --agent openclaw -p $DATASET_DIR/swe-bench-verified \
- -m openai/$MODEL_NAME \
- --agent-kwarg thinking=off \
- --n-concurrent 8
-```
-
-### OpenCode vLLM
-
-Set the following variables in your environ:
-
-```bash
-export MODEL_NAME=
-```
-
-Set the content of your OpenCode config in your environ. Remember to replace the `` with your vLLM server url and the `` with your served model name:
-
-```bash
-export OPENCODE_CONFIG_CONTENT='{"$schema":"https://opencode.ai/config.json","model":"vllm/","provider":{"vllm":{"npm":"@ai-sdk/openai-compatible","name":"vLLM","options":{"baseURL":""},"models":{"":{"name":"","limit":{"context":196500,"output":65500}}}}}}'
-```
-
-Then run:
-
-```sh
-harbor run --agent opencode -p $DATASET_DIR/swe-bench-verified \
- -m vllm/$MODEL_NAME \
- --ae "OPENCODE_CONFIG_CONTENT=$OPENCODE_CONFIG_CONTENT"
-```
-
-### OpenHands vLLM
-
-Set the following variables in your environ:
-
-```bash
-export MODEL_NAME=
-export SERVER_URL=
-export LLM_API_KEY="NONE"
-```
-
-Then run:
-
-```bash
-harbor run -d $BENCHMARK \
- -a openhands-sdk \
- -m hosted_vllm/$MODEL_NAME \
- --ae HOSTED_VLLM_API_BASE=$SERVER_URL/v1
-```
-
-### Pi vLLM
-
-Set the following variables in your environ:
-
-```bash
-export MODEL_NAME=
-```
-
-Create a `models.json` file with your vLLM server information:
-
-```bash
-export PI_MODELS_JSON='{ "providers": { "vllm": { "baseUrl": "", "api": "openai-completions", "apiKey": "NONE", "models": [{ "id": "gemma4-26b", "name": "", "contextWindow": 262000 }] } } }'
-echo $PI_MODELS_JSON > models.json
-```
-
-Then run:
-
-```bash
-harbor run --agent pi -d $BENCHMARK \
- -m vllm/$MODEL_NAME \
- --ae PI_OFFLINE=1 \
- --ae PI_CODING_AGENT_DIR=/root/.pi/agent \
- --mounts-json '[ { "type": "bind", "source":"/path/to/models.json", "target": "/root/.pi/agent/models.json" } ]'
-```
-
-### Qwen Code vLLM
-
-Set the following variables in your environ:
-
-```bash
-export MODEL_NAME=
-export SERVER_URL=
-export OPENAI_BASE_URL=$SERVER_URL/v1
-export OPENAI_API_KEY='NONE'
-```
-
-Then run:
-
-```bash
-harbor run --agent qwen-coder -d $BENCHMARK \
- -i $DATASET_PATTERN \
- -m $MODEL_NAME
-```
-
-### Claude Code Anthropic
-
-Copy `.env.example` to `.env` and set the following variables:
-
-```
-ANTHROPIC_API_KEY=
-```
-
-Then run:
-
-```bash
-set -a
-source .env
-
-harbor run --agent claude-code -d $BENCHMARK \
- -m claude-opus-4-8
-```
-
-### Claude Code VertexAI
-
-Set the following variables in your environ:
-
-```bash
-export CLOUD_ML_REGION=
-export ANTHROPIC_VERTEX_PROJECT_ID=
-export ANTHROPIC_MODEL=
-```
-
-Then run:
-
-```bash
-harbor run --agent claude-code -d $BENCHMARK \
- --ae CLAUDE_CODE_USE_VERTEX=1 \
- --ae CLOUD_ML_REGION=$CLOUD_ML_REGION \
- --ae ANTHROPIC_VERTEX_PROJECT_ID=$ANTHROPIC_VERTEX_PROJECT_ID \
- --ae ANTHROPIC_MODEL=$ANTHROPIC_MODEL \
- --ae GOOGLE_APPLICATION_CREDENTIALS='/app/.config/gcloud/application_default_credentials.json' \
- --mounts-json '[ { "type": "bind", "source":"~/.config/gcloud/application_default_credentials.json", "target": "/app/.config/gcloud/application_default_credentials.json" } ]'
-```
-
-### Codex OpenAI
-
-Copy `.env.example` to `.env` and set the following variables:
-
-```
-OPENAI_API_KEY=
-```
-
-Then run:
-
-```bash
-set -a
-source .env
-
-harbor run --agent codex -d $BENCHMARK \
- -m gpt-5.5
-```
-
-## Deploy models with vLLM
-
-Check out [`deploy/qwen-all-in-one.yml`](./deploy/qwen-all-in-one.yml) for a sample vLLM deployment of [RedHatAI/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4).
-
-Apply to your cluster by running:
-
-```sh
-oc apply -f deploy/qwen-all-in-one.yml
-```
-
-## SWE-Bench Acceleration
-
-### Use accelerated images for SWE-bench-verified
-
-1. Download the SWE-Bench-Verified tasks
-
-```sh
-harbor download swe-bench/swe-bench-verified
-```
-
-2. Replace images with the accelerated ones from [Epoch AI](https://epoch.ai/blog/swebench-docker)
-
-```sh
-uv run scripts/replace_swe_bench_images.py
-```
-
-### Pre-pull base images
-
-1. Download the dataset
-
-```sh
-harbor download
-```
-
-2. Pull all the base images
-
-```sh
-uv run scripts/pull_images.py
-```
-
-## Run with Openshift
-
-### Run Tasks in Openshift (Orchestrate Locally)
-
-Login to your cluster and select a project:
-
-```bash
-oc login --token= --server=
-oc project
-```
-
-Create ServiceAccounts and RoleBindings to run tasks:
-
-```bash
-oc apply -f deploy/harbor-task-sa.yml
-```
-
-Then in your `harbor` command, add the flag:
-
-```bash
---env openshift
-```
-
-### Run Tasks and Orchestrate in Openshift
-
-Login to your cluster and select a project:
-
-```bash
-oc login --token= --server=
-oc project
-```
-
-Create ServiceAccounts and RoleBindings to run tasks and orchestrate:
-
-```bash
-oc apply -f deploy/harbor-task-sa.yml
-oc apply -f deploy/harbor-orchestrator-sa.yml
-```
-
-Create a MinIO deployment to store your job results:
-
-```bash
-oc apply -f deploy/harbor-minio.yml
-```
-
-Using the CLI, start a job with the `--remote` flag enabled and set `--environment openshift`, e.g.:
-
-```bash
-uv run coding-agent-bench run \
- --agent claude-code \
- --dataset scale-ai/swe-bench-pro \
- --model-name my-model \
- --server-url http://my.server.url \
- --remote \
- --environment openshift
-```
-
-Skills used by a remote OpenShift Job must be public Git sources. Local paths
-are rejected because they are not available inside the orchestrator pod. The
-pod also needs outbound network access to the Git host.
-
-```bash
-uv run coding-agent-bench run \
- --agent claude-code \
- --dataset swe-bench/swe-bench-verified \
- --model-name my-model \
- --server-url http://my.server.url \
- --skill obra/superpowers@][ \
- --remote \
- --environment openshift
-```
-
-## WIP
-
-### Run with Gemini and Gemini CLI
-
-```bash
-export GOOGLE_CLOUD_PROJECT=""
-
-harbor run --agent gemini-cli -d $BENCHMARK \
- -m $MODEL_NAME
-```
-
-### Run with vLLM and Gemini CLI
-
-```bash
-harbor run --agent gemini-cli -d $BENCHMARK \
- --ae GOOGLE_GEMINI_BASE_URL=$SERVER_URL \
- --ae GEMINI_MODEL=$MODEL_NAME \
- -m $MODEL_NAME
-```
-
## Developers
### Bumping the Project Version
diff --git a/docs/cli/index.md b/docs/cli/index.md
new file mode 100644
index 0000000..205f4ee
--- /dev/null
+++ b/docs/cli/index.md
@@ -0,0 +1,4 @@
+# CLI Utility
+
+The CLI utility will help you configure and run a benchmark jobs with Harbor for self-hosted models.
+It automatically constructs and runs the Harbor job command for your specified benchmark, agent, self-hosted model.
diff --git a/docs/cli/usage.md b/docs/cli/usage.md
new file mode 100644
index 0000000..42cdf78
--- /dev/null
+++ b/docs/cli/usage.md
@@ -0,0 +1,158 @@
+# CLI Usage
+
+## Prerequisites
+
+- Install dependencies with uv
+
+ ```bash
+ uv sync
+ ```
+
+- [Set up a vLLM server](#deploy-models-with-vllm), or other Anthropic- and OpenAI-compatible server
+- Select a benchmark from among the options in [Harbor Hub](https://hub.harborframework.com/)
+
+## Run a Benchmark
+
+The following is the minimal configuration needed to run a job with the CLI:
+
+```sh
+uv run coding-agent-bench run \
+ --agent \
+ --dataset \
+ --model-name \
+ --server-url
+```
+
+For example, to run `swe-bench/swe-bench-verified` in Claude Code against a self-hosted model:
+
+```sh
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset scale-ai/swe-bench-pro \
+ --model-name my-model \
+ --server-url http://my.server.url
+```
+
+If you want to see a preview of Harbor command that would be run for a given set of arguments without actually running the job, add the `--dry-run` flag.
+
+> [!note]
+> Additional configuration options are available, use `uv run coding-agent-bench run --help` to see them.
+
+## Use Agent Skills
+
+Pass one or more skill directories or Git sources with the repeatable `--skill`
+option (`--skills` is an alias). Harbor installs the resolved skills into the
+agent used by the benchmark.
+
+To test a skill from your local filesystem:
+
+```sh
+uv run coding-agent-bench run \
+ --agent opencode \
+ --dataset swe-bench/swe-bench-verified \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --skill ./my-skills
+```
+
+Git sources make skills easy to share and reproduce. Repeat the option to test
+multiple skill collections, for example Superpowers together with Caveman:
+
+```sh
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset swe-bench/swe-bench-verified \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --skill obra/superpowers \
+ --skill juliusbrussee/caveman
+```
+
+Harbor accepts `org/name` and `org/name@ref` shorthand and HTTP(S) Git URLs.
+Repository shorthand loads skills from the repository's `skills/` directory.
+Use a full URL such as
+`https://github.com/org/repo/tree/][/` to select another directory.
+Pin a tag or named branch with `@ref` (or in the full URL) when comparing
+benchmark runs. Harbor resolves the reference to a commit and records that
+commit in the job lock file for reproducibility.
+
+## Run with Openshift
+
+### Prerequisites
+
+Login to your cluster and select a project:
+
+```bash
+oc login --token= --server=
+oc project
+```
+
+Create ServiceAccounts and RoleBindings to run tasks and orchestrate:
+
+```bash
+oc apply -f deploy/harbor-task-sa.yml
+oc apply -f deploy/harbor-orchestrator-sa.yml
+```
+
+Create a MinIO deployment to store your job results:
+
+```bash
+oc apply -f deploy/harbor-minio.yml
+```
+
+### Run Tasks in Openshift (Orchestrate Locally)
+
+Using the CLI, start a job and set `--environment openshift`, e.g.:
+
+```bash
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset scale-ai/swe-bench-pro \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --environment openshift
+```
+
+Skills used by a remote OpenShift Job must be public Git sources. Local paths
+are rejected because they are not available inside the orchestrator pod. The
+pod also needs outbound network access to the Git host.
+
+```bash
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset swe-bench/swe-bench-verified \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --skill obra/superpowers@][ \
+ --environment openshift
+```
+
+
+### Run Tasks and Orchestrate in Openshift
+
+Using the CLI, start a job with the `--remote` flag enabled and set `--environment openshift`, e.g.:
+
+```bash
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset scale-ai/swe-bench-pro \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --remote \
+ --environment openshift
+```
+
+Skills used by a remote OpenShift Job must be public Git sources. Local paths
+are rejected because they are not available inside the orchestrator pod. The
+pod also needs outbound network access to the Git host.
+
+```bash
+uv run coding-agent-bench run \
+ --agent claude-code \
+ --dataset swe-bench/swe-bench-verified \
+ --model-name my-model \
+ --server-url http://my.server.url \
+ --skill obra/superpowers@][ \
+ --remote \
+ --environment openshift
+```
diff --git a/docs/developer/documentation.md b/docs/developer/documentation.md
new file mode 100644
index 0000000..224362e
--- /dev/null
+++ b/docs/developer/documentation.md
@@ -0,0 +1,23 @@
+# Documentation
+
+Documentation should be updated with any major feature updates, or to correct the documentation if it is outdated or incorrect.
+
+You are encouraged to write the documentation yourself, but you may use an agent if you wish.
+
+Please keep the documentation readable - shorter, more user-friendly documentation is preferred over detailed documentation.
+
+Do not write documentation for simple changes or bugfixes, unless it invalidates any existing documentation.
+
+## Previewing Changes
+
+Preview the documentation locally:
+
+```sh
+uv run mkdocs serve --strict
+```
+
+Any changes you make to the docs will automatically be reflected in the local server.
+
+## Updating the Documentation Site
+
+Any merge into the `main` branch will automatically rebuild and update the documentation site.
diff --git a/docs/developer/index.md b/docs/developer/index.md
new file mode 100644
index 0000000..fbe295d
--- /dev/null
+++ b/docs/developer/index.md
@@ -0,0 +1,24 @@
+# Developer Documentation
+
+## Prerequisites
+
+- Python >= 3.12
+- [uv](https://docs.astral.sh/uv/getting-started/installation/)
+
+## Getting Started
+
+Clone and install the project:
+
+```sh
+git clone https://github.com/redhat-et/coding_agent_bench.git
+cd coding_agent_bench
+
+uv venv
+uv sync
+```
+
+Copy `.env.example` to `.env`:
+
+```sh
+cp .env.example .env
+```
diff --git a/docs/developer/new_benchmark.md b/docs/developer/new_benchmark.md
new file mode 100644
index 0000000..e69de29
diff --git a/docs/developer/new_harness.md b/docs/developer/new_harness.md
new file mode 100644
index 0000000..e69de29
diff --git a/docs/developer/new_model.md b/docs/developer/new_model.md
new file mode 100644
index 0000000..5e6e262
--- /dev/null
+++ b/docs/developer/new_model.md
@@ -0,0 +1,135 @@
+# Add a New Model
+
+This will walk you through the two step process of adding a new model to the project: first validating the model and then registering it.
+
+## Validate a New Model
+
+Before a model can be added to the project, it must be validated to be able to run on at least one Nebius instances.
+
+### Process Flow
+
+1. Create GPU instance on Nebius (via CLI)
+2. SSH in and run setup commands
+3. Start vLLM model server
+4. Run benchmark(s)
+5. Delete Nebius instance when done
+
+### Prerequisites
+
+- [Install the Nebius CLI](https://docs.nebius.com/cli/install) and [log in](https://docs.nebius.com/cli/configure)
+- Join our Nebius organization (Reach out to Taylor Agarwal)
+- Create a service account and attach an authorized key
+
+```shell
+export SA_NAME=
+export PROJECT_ID=project-i00hz9y8pr00mf5rzvw82k
+
+# Create Service Account
+export SA_ID=$(nebius iam service-account create \
+ --name $SA_NAME \
+ --format json | jq -r '.metadata.id')
+
+# Create and attach an authorized key
+nebius iam auth-public-key generate \
+ --service-account-id $SA_ID \
+ --output ~/.nebius/$SA_ID-credentials.json
+
+# Configure the SA as an editor in the project
+export EDITOR_GROUP_ID=$(nebius iam group get-by-name \
+ --name editors --parent-id $PROJECT_ID \
+ --format json | jq -r '.metadata.id')
+
+nebius iam group-membership create \
+ --parent-id $EDITOR_GROUP_ID \
+ --member-id $SA_ID
+```
+
+### Step 1: Create an SSH Key Pair
+
+```shell
+ssh-keygen -t ed25519 -f ~/.ssh/nebius
+
+export USER_DATA=$(jq -Rrs '.' <
+export SA_ID=
+export PROJECT_ID=project-i00hz9y8pr00mf5rzvw82k
+
+# Check if the instance already exists
+nebius compute instance get-by-name --name $INSTANCE_NAME
+
+# Create the instance if it does not exist
+nebius compute v1 instance create \
+ --parent-id $PROJECT_ID \
+ --name $INSTANCE_NAME \
+ --service-account-id $SA_ID \
+ --resources-platform gpu-b200-sxm-a \
+ --resources-preset 1gpu-20vcpu-224gb \
+ --network-interfaces '[{"subnetId":"vpcsubnet-i00y73e2kyyjze510h","name":"eth0","ipAddress":{},"publicIpAddress":{}}]' \
+ --boot-disk-attach-mode read_write \
+ --boot-disk-managed-disk-name $INSTANCE_NAME-boot-disk \
+ --boot-disk-managed-disk-size-bytes 1374389534720 \
+ --boot-disk-managed-disk-block-size-bytes 4096 \
+ --boot-disk-managed-disk-type network_ssd \
+ --boot-disk-managed-disk-source-image-family-image-family ubuntu24.04-cuda13.0 \
+ --boot-disk-device-id boot-disk \
+ --cloud-init-user-data "$USER_DATA" \
+ --reservation-policy-policy forbid \
+ --preemptible-on-preemption stop
+
+# Get the public IP Address for the instance
+export PUBLIC_IP_ADDRESS=$(nebius compute instance get-by-name \
+ --name $INSTANCE_NAME \
+ --format json \
+ | jq -r '.status.network_interfaces[0].public_ip_address.address | split("/")[0]')
+
+echo $PUBLIC_IP_ADDRESS
+```
+
+### Step 3: SSH into Instance
+
+```shell
+ssh -i ~/.ssh/nebius $USER@$PUBLIC_IP_ADDRESS
+```
+
+### Step 4: Set HF_TOKEN (optional, speeds up model downloads)
+
+```shell
+export HF_TOKEN=
+```
+
+### Step 5: Run Your vLLM Command
+
+Refer to the examples in the [Validated Models](../validated_models.md).
+
+You can find the commands for many models in [vLLM's Recipes](https://recipes.vllm.ai/).
+
+### Step 6: Connect to Running Instance
+
+```shell
+VLLM_SERVER_PUBLIC_URL=http://$PUBLIC_IP_ADDRESS:8000
+echo "The vLLM server is located at $VLLM_SERVER_PUBLIC_URL"
+```
+
+## Register the Model in `coding_agent_bench`
+
+Once the model has been validated, it can be added to `coding_agent_bench` to be used in benchmark runs on managed Nebius instances.
+
+First, add a new child class of `ModelConfig` with the model-specific arguments to `src/coding_agent_bench/models/configs.py`.
+
+Then register the new class in `MODEL_CONFIGS` and `MODEL_REGISTRY` in `src/coding_agent_bench/models/__init__.py`.
+
+Please also add the model to the [Validated Models docs](../validated_models.md#validated-vllm-commands).
diff --git a/docs/developer/swe_bench_acceleration.md b/docs/developer/swe_bench_acceleration.md
new file mode 100644
index 0000000..14826ac
--- /dev/null
+++ b/docs/developer/swe_bench_acceleration.md
@@ -0,0 +1,29 @@
+# SWE-Bench Acceleration
+
+## Use accelerated images for SWE-bench-verified
+
+1. Download the SWE-Bench-Verified tasks
+
+```sh
+harbor download swe-bench/swe-bench-verified
+```
+
+2. Replace images with the accelerated ones from [Epoch AI](https://epoch.ai/blog/swebench-docker)
+
+```sh
+uv run scripts/replace_swe_bench_images.py
+```
+
+## Pre-pull base images
+
+1. Download the dataset
+
+```sh
+harbor download
+```
+
+2. Pull all the base images
+
+```sh
+uv run scripts/pull_images.py
+```
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000..5b358c8
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,13 @@
+# Coding Agent Bench
+
+Reproducible benchmarks for coding agents and models using Harbor
+
+
+
+## Features
+
+- Complete instructions for running popular benchmarks with open models
+- CLI utility to simplify benchmark runs against self-hosted models
+- Deployable queue service for scheduling benchmark runs in OpenShift
+- Leaderboards for popular benchmarks with instructions for reproducing results
+- Full manifests for deploying open models on OpenShift with vLLM
diff --git a/docs/queue-service/index.md b/docs/queue-service/index.md
new file mode 100644
index 0000000..482990b
--- /dev/null
+++ b/docs/queue-service/index.md
@@ -0,0 +1,22 @@
+# Queue Service
+
+The queue service is a FastAPI application that can be deployed on OpenShift to queue and run benchmarks automatically.
+Benchmark results are stored to MinIO for later review.
+
+```mermaid
+sequenceDiagram
+ Requestor->>Queue Service: Request Benchmark Run
+ Queue Service->>Queue Service: Wait in Queue
+ Queue Service->>Openshift Job: Start Benchmark Job
+ Openshift Job->>Harbor Orchestrator Pod: Start Harbor Run
+ loop For Each Task
+ Harbor Orchestrator Pod->>Task Pod: Run Task Pod
+ Task Pod->>Harbor Orchestrator Pod: Save Results
+ end
+ Harbor Orchestrator Pod->>MinIO: Save Benchmark Results
+ Harbor Orchestrator Pod->>Openshift Job: Complete
+ Queue Service-->>Openshift Job: Poll for completion
+ Openshift Job->>Queue Service: Complete
+ Queue Service->>Openshift Job: Cleanup
+ Queue Service->>Queue Service: Start Next Job
+```
diff --git a/docs/queue-service/intake_poller.md b/docs/queue-service/intake_poller.md
new file mode 100644
index 0000000..b9911ac
--- /dev/null
+++ b/docs/queue-service/intake_poller.md
@@ -0,0 +1,13 @@
+If the intake CronJob is deployed, create its separate poller secret:
+ ```yaml
+ apiVersion: v1
+ kind: Secret
+ metadata:
+ name: intake-poller-secret
+ stringData:
+ JOB_QUEUE_URL: https://
+ GOOGLE_SHEET_ID:
+ SENDER_EMAIL: ace-model-evals@redhat.com
+ AUTO_APPROVE: 'false'
+ type: Opaque
+ ```
\ No newline at end of file
diff --git a/docs/queue-service/local.md b/docs/queue-service/local.md
new file mode 100644
index 0000000..33ce984
--- /dev/null
+++ b/docs/queue-service/local.md
@@ -0,0 +1,55 @@
+# Queue Service Local Setup [Development]
+
+This setup is primarily for developer testing of the queue service.
+
+The queue service API server runs locally, but all other components still run on OpenShift.
+
+## Steps
+
+1. Clone the repository and install dependencies
+
+ ```sh
+ git clone https://github.com/redhat-et/coding_agent_bench.git
+ cd coding_agent_bench
+
+ uv venv
+ uv sync
+ ```
+
+2. Copy the `.env.example` file to `.env` and fill in the `API_KEY` variable
+
+ ```sh
+ cp .env.example .env
+ ```
+
+ (Optional) If using Nebius, add environment variables for Nebius
+
+3. Log into your OpenShift cluster and project, or create a new project
+
+ ```sh
+ oc project coding-agent-leaderboard
+ ```
+
+4. Create the MinIO service for artifact storage:
+
+ ```sh
+ oc apply -f deploy/harbor-minio.yml
+ ```
+
+ Note: the default username and password are `(minioadmin, minioadmin)`.
+ You can update this in the deployment file if needed.
+
+5. Apply the service accounts for the orchestrator and task pods
+
+ ```sh
+ oc apply -f deploy/harbor-orchestrator-sa.yml
+ oc apply -f deploy/harbor-task-sa.yml
+ ```
+
+6. Start the queue service locally
+
+ ```sh
+ uv run uvicorn coding_agent_bench.api:app --port 8080
+ ```
+
+7. Open the UI at [http://localhost:8080](http://localhost:8080) and test your features
diff --git a/docs/queue-service/nebius.md b/docs/queue-service/nebius.md
new file mode 100644
index 0000000..e28633c
--- /dev/null
+++ b/docs/queue-service/nebius.md
@@ -0,0 +1,105 @@
+# Nebius Setup
+
+The queue service supports starting and stopping vLLM server instances automatically using [Nebius](https://nebius.com/).
+When enabled, the queue service will create a Nebius VM, SSH into the instance, start the vLLM server for the next model in the queue, run the benchmark job against that model, swap the running model for the next model in the queue, then spin down the Nebius VM when the queue is empty.
+
+To connect the queue service to Nebius, first [set up an AI Cloud account](https://docs.nebius.com/signup-billing/sign-up).
+
+After setting up the account, [install the CLI](https://docs.nebius.com/cli/install) and [login to your account](https://docs.nebius.com/cli/configure).
+
+Then run the following commands to create a service account in your project:
+
+```sh
+# Create a service account
+export SA_ID=$(nebius iam service-account create \
+ --name \
+ --format json | jq -r '.metadata.id')
+
+# Create and attach an authorized key to the service account
+nebius iam auth-public-key generate \
+ --service-account-id $SA_ID \
+ --output ~/.nebius/$SA_ID-credentials.json
+```
+
+## Setup
+
+### Openshift
+
+Once the service account is created, you can update your job queue secret with the following environment variables needed for Nebius:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+ name: job-queue-secret
+stringData:
+ API_KEY:
+ NEBIUS_ENABLED: '1'
+ NEBIUS_SERVICE_ACCOUNT_CREDS: |
+
+ NEBIUS_PARENT_ID:
+ NEBIUS_TENANT_ID:
+ NEBIUS_SERVICE_ACCOUNT_ID:
+ NEBIUS_SUBNET_ID:
+ NEBIUS_INSTANCE_NAME_PREFIX: job-queue-worker
+ NEBIUS_IDLE_TIMEOUT_SECONDS: '600'
+ HF_TOKEN:
+type: Opaque
+```
+
+### Local
+
+Set the following environment variables in your `.env`:
+
+```
+NEBIUS_ENABLED=1
+NEBIUS_SERVICE_ACCOUNT_CREDS_PATH=
+NEBIUS_USER=
+NEBIUS_SSH_PUBLIC_KEY_PATH=
+NEBIUS_SSH_PRIVATE_KEY_PATH=
+NEBIUS_PARENT_ID=
+NEBIUS_TENANT_ID=
+NEBIUS_SERVICE_ACCOUNT_ID=
+NEBIUS_SUBNET_ID=
+NEBIUS_INSTANCE_NAME_PREFIX=cab-worker
+NEBIUS_IDLE_TIMEOUT_SECONDS=600
+HF_TOKEN=
+```
+
+## Usage
+
+When creating a job, set `server_url` to `nebius-` to use a managed Nebius instance with the specified GPU resource (e.g. `nebius-h200`, `nebius-b200`).
+Available resources are defined in `RESOURCE_CONFIG_REGISTRY`.
+The options are listed below as well, but use the `RESOURCE_CONFIG_REGISTRY` as the source of truth.
+
+When creating a job, set `model_name` to one of the options in `MODEL_REGISTRY`.
+The options are listed below as well, but use the `MODEL_REGISTRY` as the source of truth.
+
+For example:
+
+```sh
+curl -X POST $JOB_QUEUE_URL/jobs \
+ -d '{"job_name": "test", "agent": "pi", "dataset": "swe-bench/swe-bench-verified", "model_name": "Qwen/Qwen3.6-27B", "server_url": "nebius-b200", "n_tasks": 1}' \
+ -H "Content-Type: application/json" \
+ -H "X-API-Key: "
+```
+
+## Supported Resources
+
+| Hardware | Supported | Resource Key |
+| -------------- | --------- | -------------- |
+| gpu-h200-sxm | ✅ | `nebius-h200` |
+| gpu-b200-sxm | ✅ | `nebius-b200` |
+| gpu-b200-sxm-a | ✅ | `nebius-b200a` |
+
+## Supported Models
+
+| Model | Supported | Resource Key |
+| ------------------------------------------------ | --------- | -------------------------------------------------- |
+| Qwen/Qwen3.8-27B | ✅ | `Qwen/Qwen3.8-27B` |
+| Qwen/Qwen3.8-27B-FP8 | ✅ | `Qwen/Qwen3.8-27B-FP8` |
+| RedHatAI/gemma-4-31B-it-FP8-block | ✅ | `RedHatAI/gemma-4-31B-it-FP8-block` |
+| RedHatAI/gpt-oss-120b | ✅ | `RedHatAI/gpt-oss-120b` |
+| RedHatAI/Mistral-Small-4-119B-2603-NVFP4 | ✅ | `RedHatAI/Mistral-Small-4-119B-2603-NVFP4` |
+| RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 | ✅ | `RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` |
+| RedHatAI/Qwen3.6-27B-FP8 | ✅ | `RedHatAI/Qwen3.6-27B-FP8` |
diff --git a/docs/queue-service/openshift.md b/docs/queue-service/openshift.md
new file mode 100644
index 0000000..44f0623
--- /dev/null
+++ b/docs/queue-service/openshift.md
@@ -0,0 +1,86 @@
+# Queue Service OpenShift Setup
+
+## Steps
+
+1. Log in to your cluster and project:
+
+ ```sh
+ oc login --server= --token=
+ oc project
+ ```
+
+2. Create the MinIO service for artifact storage:
+
+ ```sh
+ oc apply -f deploy/harbor-minio.yml
+ ```
+
+ Note: the default username and password are `(minioadmin, minioadmin)`.
+ You can update this in the deployment file if needed.
+
+3. Create the orchestrator and task service accounts:
+
+ ```sh
+ oc apply -f deploy/harbor-orchestrator-sa.yml
+ oc apply -f deploy/harbor-task-sa.yml
+ ```
+
+4. Create a secret file named `job-queue-secret` with the queue service's `API_KEY` and any queue or Nebius settings, then apply it:
+
+ ```yaml
+ apiVersion: v1
+ kind: Secret
+ metadata:
+ name: job-queue-secret
+ stringData:
+ API_KEY:
+ type: Opaque
+ ```
+
+5. Create the queue service:
+
+ ```sh
+ oc apply -f deploy/job-queue-service.yml
+ ```
+
+6. (Optional) To run jobs against OpenRouter (`server_url: openrouter`), create
+ an `openrouter-api-key` secret. Job pods mount it automatically (it is
+ optional, so non-OpenRouter jobs are unaffected):
+ ```yaml
+ apiVersion: v1
+ kind: Secret
+ metadata:
+ name: openrouter-api-key
+ stringData:
+ OPENROUTER_API_KEY:
+ type: Opaque
+ ```
+ The queue service itself also needs `OPENROUTER_API_KEY` in its environment
+ to validate OpenRouter jobs at request time. Add it to `job-queue-secret`
+ (which the service already loads) or `envFrom` the `openrouter-api-key`
+ secret in `deploy/job-queue-service.yml`.
+
+ The queue listens on HTTPS inside the cluster. OpenShift's service-serving
+ certificate operator creates the `job-queue-tls` Secret referenced by the
+ Deployment, and the Route uses re-encryption so traffic remains encrypted
+ from the router to the queue pod. Wait for that Secret to appear before
+ troubleshooting pod startup:
+ ```sh
+ oc get secret job-queue-tls
+ ```
+
+Get the route for the deployed service:
+
+```sh
+oc get route job-queue-route --output jsonpath='{.spec.host}'
+```
+
+Set `JOB_QUEUE_URL` in `intake-poller-secret` to this HTTPS route before
+applying `deploy/intake-cronjob.yml`.
+
+Check that the application is live by visiting the docs:
+
+```sh
+export JOB_QUEUE_URL="https://$(oc get route job-queue-route --output jsonpath='{.spec.host}')"
+open $JOB_QUEUE_URL/docs
+```
diff --git a/docs/queue-service/usage.md b/docs/queue-service/usage.md
new file mode 100644
index 0000000..acc4f12
--- /dev/null
+++ b/docs/queue-service/usage.md
@@ -0,0 +1,52 @@
+# Queue Service Usage
+
+Before using the below commands, make sure you have set the `JOB_QUEUE_URL` in your environment.
+
+## Submit A Job
+
+Queue up a new benchmark task:
+
+```sh
+# Request
+curl -X POST $JOB_QUEUE_URL/jobs \
+ -d '{"job_name": "test", "agent": "pi", "dataset": "swe-bench/swe-bench-verified", "model_name": "qwen3.6-27b", "server_url": "", "n_tasks": 1}' \
+ -H "Content-Type: application/json" \
+ -H "X-API-Key: "
+
+# Response
+# {
+# "message":"Job created.",
+# "job_id":"b5ef13c8-8909-4bf1-b5b1-43354e9f395c",
+# ...
+# }
+```
+
+OR
+
+Open the UI and submit a job there:
+
+```sh
+open $JOB_QUEUE_URL/ui
+```
+
+## View the Queue
+
+View the queued/running/completed tasks:
+
+```sh
+open $JOB_QUEUE_URL/ui
+```
+
+Or list them from the API:
+
+```sh
+curl $JOB_QUEUE_URL/jobs -H "X-API-Key: "
+```
+
+## Cancel Jobs
+
+Cancel a running or queued job:
+
+```sh
+curl -X DELETE $JOB_QUEUE_URL/jobs/ -H "X-API-Key: "
+```
diff --git a/docs/validated_harbor_commands.md b/docs/validated_harbor_commands.md
new file mode 100644
index 0000000..c8202d8
--- /dev/null
+++ b/docs/validated_harbor_commands.md
@@ -0,0 +1,259 @@
+# Harbor Validated Commands
+
+## Prerequisites:
+
+- Install [Harbor](https://www.harborframework.com/docs/getting-started)
+- [Set up a vLLM server](#deploy-models-with-vllm), or other Anthropic- and OpenAI-compatible server
+- Set your benchmark in your environment from among the options in [Harbor Hub](https://hub.harborframework.com/), e.g.:
+
+ ```bash
+ export BENCHMARK='swe-bench/swe-bench-verified'
+ ```
+
+- If you need to filter tasks in your benchmark by name, add the `-i` flag with your glob pattern to your `harbor run` command, e.g. `-i "*ansible*"`
+
+## Directory:
+
+| Harness | Model Server | Example | Status |
+| ----------- | ------------ | ------------------------------ | --------- |
+| Claude Code | vLLM | [Link](#claude-code-vllm) | Validated |
+| Codex | vLLM | [Link](#codex-vllm) | Testing |
+| OpenClaw | vLLM | [Link](#openclaw-vllm) | Validated |
+| OpenCode | vLLM | [Link](#opencode-vllm) | Validated |
+| OpenHands | vLLM | [Link](#openhands-vllm) | Validated |
+| Pi | vLLM | [Link](#pi-vllm) | Validated |
+| Qwen Code | vLLM | [Link](#qwen-code-vllm) | Validated |
+| Claude Code | Anthropic | [Link](#claude-code-anthropic) | Validated |
+| Claude Code | VertexAI | [Link](#claude-code-vertexai) | Validated |
+| Codex | OpenAI | [Link](#codex-openai) | Validated |
+
+> [!note]
+> To use with a locally hosted model (e.g. llama.cpp) use a vLLM example and set `SERVER_URL=http://host.docker.internal:`
+
+## Claude Code vLLM
+
+Set the following variables in your environ:
+
+```bash
+export SERVER_URL=
+export MODEL_NAME=
+```
+
+Then run:
+
+```bash
+harbor run --agent claude-code -d $BENCHMARK \
+ --ae ANTHROPIC_BASE_URL=$SERVER_URL \
+ --ae ANTHROPIC_API_KEY='sk-no-key-required' \
+ --ae ANTHROPIC_MODEL=$MODEL_NAME \
+ --ae ANTHROPIC_DEFAULT_OPUS_MODEL=$MODEL_NAME \
+ --ae ANTHROPIC_DEFAULT_SONNET_MODEL=$MODEL_NAME \
+ --ae ANTHROPIC_DEFAULT_HAIKU_MODEL=$MODEL_NAME
+```
+
+## Codex vLLM
+
+Set the following variables in your environ:
+
+```bash
+export SERVER_URL=
+export MODEL_NAME=
+```
+
+Use the utility script to create the `config.toml` file:
+
+```bash
+uv run scripts/codex_config_toml.py $MODEL_NAME $SERVER_URL
+```
+
+Then run:
+
+```bash
+harbor run --agent codex -d $BENCHMARK \
+ -m vllm/$MODEL_NAME \
+ --ae CODEX_HOME=/root/.codex/ \
+ --mounts-json '[ { "type": "bind", "source":"/path/to/coding-agent-bench/config.toml", "target": "/root/.codex/config.toml" } ]'
+```
+
+## OpenClaw vLLM
+
+Set the following variables in your environ:
+
+```sh
+export MODEL_NAME=
+export SERVER_URL=
+export OPENAI_BASE_URL=$SERVER_URL/v1
+export OPENAI_API_KEY='NONE'
+```
+
+Then run:
+
+```sh
+harbor run --agent openclaw -p $DATASET_DIR/swe-bench-verified \
+ -m openai/$MODEL_NAME \
+ --agent-kwarg thinking=off \
+ --n-concurrent 8
+```
+
+## OpenCode vLLM
+
+Set the following variables in your environ:
+
+```bash
+export MODEL_NAME=
+```
+
+Set the content of your OpenCode config in your environ. Remember to replace the `` with your vLLM server url and the `` with your served model name:
+
+```bash
+export OPENCODE_CONFIG_CONTENT='{"$schema":"https://opencode.ai/config.json","model":"vllm/","provider":{"vllm":{"npm":"@ai-sdk/openai-compatible","name":"vLLM","options":{"baseURL":""},"models":{"":{"name":"","limit":{"context":196500,"output":65500}}}}}}'
+```
+
+Then run:
+
+```sh
+harbor run --agent opencode -p $DATASET_DIR/swe-bench-verified \
+ -m vllm/$MODEL_NAME \
+ --ae "OPENCODE_CONFIG_CONTENT=$OPENCODE_CONFIG_CONTENT"
+```
+
+## OpenHands vLLM
+
+Set the following variables in your environ:
+
+```bash
+export MODEL_NAME=
+export SERVER_URL=
+export LLM_API_KEY="NONE"
+```
+
+Then run:
+
+```bash
+harbor run -d $BENCHMARK \
+ -a openhands-sdk \
+ -m hosted_vllm/$MODEL_NAME \
+ --ae HOSTED_VLLM_API_BASE=$SERVER_URL/v1
+```
+
+## Pi vLLM
+
+Set the following variables in your environ:
+
+```bash
+export MODEL_NAME=
+```
+
+Create a `models.json` file with your vLLM server information:
+
+```bash
+export PI_MODELS_JSON='{ "providers": { "vllm": { "baseUrl": "", "api": "openai-completions", "apiKey": "NONE", "models": [{ "id": "gemma4-26b", "name": "", "contextWindow": 262000 }] } } }'
+echo $PI_MODELS_JSON > models.json
+```
+
+Then run:
+
+```bash
+harbor run --agent pi -d $BENCHMARK \
+ -m vllm/$MODEL_NAME \
+ --ae PI_OFFLINE=1 \
+ --ae PI_CODING_AGENT_DIR=/root/.pi/agent \
+ --mounts-json '[ { "type": "bind", "source":"/path/to/models.json", "target": "/root/.pi/agent/models.json" } ]'
+```
+
+## Qwen Code vLLM
+
+Set the following variables in your environ:
+
+```bash
+export MODEL_NAME=
+export SERVER_URL=
+export OPENAI_BASE_URL=$SERVER_URL/v1
+export OPENAI_API_KEY='NONE'
+```
+
+Then run:
+
+```bash
+harbor run --agent qwen-coder -d $BENCHMARK \
+ -i $DATASET_PATTERN \
+ -m $MODEL_NAME
+```
+
+## Claude Code Anthropic
+
+Copy `.env.example` to `.env` and set the following variables:
+
+```
+ANTHROPIC_API_KEY=
+```
+
+Then run:
+
+```bash
+set -a
+source .env
+
+harbor run --agent claude-code -d $BENCHMARK \
+ -m claude-opus-4-8
+```
+
+## Claude Code VertexAI
+
+Set the following variables in your environ:
+
+```bash
+export CLOUD_ML_REGION=
+export ANTHROPIC_VERTEX_PROJECT_ID=
+export ANTHROPIC_MODEL=
+```
+
+Then run:
+
+```bash
+harbor run --agent claude-code -d $BENCHMARK \
+ --ae CLAUDE_CODE_USE_VERTEX=1 \
+ --ae CLOUD_ML_REGION=$CLOUD_ML_REGION \
+ --ae ANTHROPIC_VERTEX_PROJECT_ID=$ANTHROPIC_VERTEX_PROJECT_ID \
+ --ae ANTHROPIC_MODEL=$ANTHROPIC_MODEL \
+ --ae GOOGLE_APPLICATION_CREDENTIALS='/app/.config/gcloud/application_default_credentials.json' \
+ --mounts-json '[ { "type": "bind", "source":"~/.config/gcloud/application_default_credentials.json", "target": "/app/.config/gcloud/application_default_credentials.json" } ]'
+```
+
+## Codex OpenAI
+
+Copy `.env.example` to `.env` and set the following variables:
+
+```
+OPENAI_API_KEY=
+```
+
+Then run:
+
+```bash
+set -a
+source .env
+
+harbor run --agent codex -d $BENCHMARK \
+ -m gpt-5.5
+```
+
+## WIP
+
+### Run with Gemini and Gemini CLI
+
+```bash
+export GOOGLE_CLOUD_PROJECT=""
+
+harbor run --agent gemini-cli -d $BENCHMARK \
+ -m $MODEL_NAME
+```
+
+### Run with vLLM and Gemini CLI
+
+```bash
+harbor run --agent gemini-cli -d $BENCHMARK \
+ --ae GOOGLE_GEMINI_BASE_URL=$SERVER_URL \
+ --ae GEMINI_MODEL=$MODEL_NAME \
+ -m $MODEL_NAME
+```
diff --git a/docs/validated_models.md b/docs/validated_models.md
new file mode 100644
index 0000000..44218a7
--- /dev/null
+++ b/docs/validated_models.md
@@ -0,0 +1,210 @@
+# Validated Models
+
+Models are validated on Nebius with the following settings
+
+gpu-b200-sxm:
+- 1x B200 (180 GB vRAM)
+- 20 vCPU
+- 224 GB RAM
+- Preemptive
+- ME-west1
+
+| Model | Hardware | KV Cache | Max Concurrency |
+| ---------------------------------------------------------------------------------------------------- | ------------ | -------- | --------------- |
+| [Qwen/Qwen3.8-27B](#qwenqwen38-27b) | gpu-b200-sxm | FP8 | 12x |
+| [Qwen/Qwen3.8-27B-FP8](#qwenqwen38-27b-fp8) | gpu-b200-sxm | FP8 | 15x |
+| [RedHatAI/Qwen3.8-27B-INT4](#redhataiqwen38-27b-int4) | gpu-b200-sxm | FP8 | 16.49x |
+| [RedHatAI/Qwen3.6-27B-FP8](#redhataiqwen36-27b-fp8) | gpu-b200-sxm | FP8 | 29x |
+| [RedHatAI/gemma-4-31B-it-FP8-block](#redhataigemma-4-31b-it-fp8-block) | gpu-b200-sxm | FP8 | |
+| [RedHatAI/Mistral-Small-4-119B-2603-NVFP4](#redhataimistral-small-4-119b-2603-nvfp4) | gpu-b200-sxm | auto | |
+| [RedHatAI/gpt-oss-120b](#redhataigpt-oss-120b) | gpu-b200-sxm | FP8 | 40x |
+| [RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4](#redhatainvidia-nemotron-3-super-120b-a12b-nvfp4) | gpu-b200-sxm | FP8 | 14x |
+
+
+## Validated vLLM Commands
+
+### Qwen/Qwen3.8-27B
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model Qwen/Qwen3.8-27B \
+ --max-model-len 262144 \
+ --tensor-parallel-size 1 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype fp8 \
+ --enable-auto-tool-choice \
+ --tool-call-parser qwen3_coder \
+ --reasoning-parser qwen3 \
+ --mm-encoder-tp-mode data
+```
+
+### Qwen/Qwen3.8-27B-FP8
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model Qwen/Qwen3.8-27B-FP8 \
+ --max-model-len 262144 \
+ --tensor-parallel-size 1 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype fp8 \
+ --enable-auto-tool-choice \
+ --tool-call-parser qwen3_coder \
+ --reasoning-parser qwen3 \
+ --mm-encoder-tp-mode data
+```
+
+### RedHatAI/Qwen3.8-27B-INT4
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/Qwen3.8-27B-INT4 \
+ --max-model-len 262144 \
+ --tensor-parallel-size 1 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype fp8 \
+ --enable-auto-tool-choice \
+ --tool-call-parser qwen3_coder \
+ --reasoning-parser qwen3 \
+ --mm-encoder-tp-mode data
+```
+
+### RedHatAI/Qwen3.6-27B-FP8
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/Qwen3.6-27B-FP8 \
+ --dtype auto \
+ --max-model-len 131072 \
+ --trust-remote-code \
+ --tensor-parallel-size 1 \
+ --gpu-memory-utilization 0.9 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype fp8 \
+ --enable-auto-tool-choice \
+ --reasoning-parser qwen3 \
+ --tool-call-parser qwen3_coder \
+ --default-chat-template-kwargs '{"enable_thinking": true}'
+```
+
+### RedHatAI/gemma-4-31B-it-FP8-block
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/gemma-4-31B-it-FP8-block \
+ --dtype auto \
+ --max-model-len 262144 \
+ --trust-remote-code \
+ --tensor-parallel-size 1 \
+ --gpu-memory-utilization 0.9 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype fp8 \
+ --enable-auto-tool-choice \
+ --reasoning-parser gemma4 \
+ --tool-call-parser gemma4 \
+ --default-chat-template-kwargs '{"enable_thinking": true}'
+
+```
+
+### RedHatAI/Mistral-Small-4-119B-2603-NVFP4
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/Mistral-Small-4-119B-2603-NVFP4 \
+ --dtype auto \
+ --max-model-len 131072 \
+ --trust-remote-code \
+ --tensor-parallel-size 1 \
+ --gpu-memory-utilization 0.9 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --kv-cache-dtype auto \
+ --enable-auto-tool-choice \
+ --reasoning-parser mistral \
+ --tool-call-parser mistral \
+ --default-chat-template-kwargs '{"reasoning_effort": "high"}' \
+ --limit-mm-per-prompt '{"image": 0}'
+```
+
+### RedHatAI/gpt-oss-120b
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/gpt-oss-120b \
+ --dtype auto \
+ --kv-cache-dtype fp8 \
+ --tensor-parallel-size 1 \
+ --gpu-memory-utilization 0.9 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --enable-auto-tool-choice \
+ --tool-call-parser openai
+```
+
+### RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
+
+```shell
+sudo docker run --runtime nvidia --gpus all \
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
+ --env "HF_TOKEN=$HF_TOKEN" \
+ -p 8000:8000 \
+ --ipc=host \
+ vllm/vllm-openai:v0.24.0 \
+ --model RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
+ --dtype auto \
+ --kv-cache-dtype fp8 \
+ --tensor-parallel-size 1 \
+ --gpu-memory-utilization 0.9 \
+ --async-scheduling \
+ --enable-chunked-prefill \
+ --enable-prefix-caching \
+ --enable-auto-tool-choice \
+ --reasoning-parser nemotron_v3 \
+ --tool-call-parser qwen3_coder
+```
diff --git a/mkdocs.yaml b/mkdocs.yaml
new file mode 100644
index 0000000..25d3326
--- /dev/null
+++ b/mkdocs.yaml
@@ -0,0 +1,69 @@
+site_name: "Coding Agent Bench"
+
+repo_url: https://github.com/redhat-et/coding_agent_bench
+
+theme:
+ name: "material"
+ icon:
+ logo: material/chart-box
+ features:
+ # Add tabs
+ - navigation.tabs
+ - navigation.tabs.sticky
+ palette:
+ # Palette toggle for light mode
+ - media: "(prefers-color-scheme: light)"
+ scheme: default
+ toggle:
+ icon: material/brightness-7
+ name: Switch to dark mode
+
+ # Palette toggle for dark mode
+ - media: "(prefers-color-scheme: dark)"
+ scheme: slate
+ toggle:
+ icon: material/brightness-4
+ name: Switch to light mode
+
+plugins:
+ - search
+ - mkdocstrings
+
+nav:
+ - Home:
+ - Home: index.md
+ - Validated Harbor Commands: validated_harbor_commands.md
+ - Validated Models: validated_models.md
+ - CLI:
+ - Overview: cli/index.md
+ - Usage: cli/usage.md
+ - Queue Service:
+ - Overview: queue-service/index.md
+ - Setup:
+ - Openshift: queue-service/openshift.md
+ - Local (Developer): queue-service/local.md
+ - Nebius (Optional): queue-service/nebius.md
+ - Usage: queue-service/usage.md
+ - Intake Poller: queue-service/intake_poller.md
+ - Developer:
+ - Overview: developer/index.md
+ - Adding Features:
+ - New Model: developer/new_model.md
+ - New Harness: developer/new_harness.md
+ - New Benchmark: developer/new_benchmark.md
+ - Documentation: developer/documentation.md
+
+markdown_extensions:
+ - admonition
+ - attr_list
+ - pymdownx.details
+ - pymdownx.emoji:
+ emoji_index: !!python/name:material.extensions.emoji.twemoji
+ emoji_generator: !!python/name:material.extensions.emoji.to_svg
+ - pymdownx.inlinehilite
+ - pymdownx.snippets
+ - pymdownx.superfences:
+ custom_fences:
+ - name: mermaid
+ class: mermaid
+ format: !!python/name:pymdownx.superfences.fence_code_format
diff --git a/pyproject.toml b/pyproject.toml
index b7694a7..a14f75e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -30,6 +30,8 @@ build-backend = "uv_build"
dev = [
"bump-my-version>=1.5.1",
"bump2version>=1.0.1",
+ "mkdocs-material>=9.7.7",
+ "mkdocstrings-python>=2.0.8",
"pytest>=9.1.1",
"ruff>=0.15.12",
]
diff --git a/src/coding_agent_bench/models/__init__.py b/src/coding_agent_bench/models/__init__.py
index e729db2..8b60091 100644
--- a/src/coding_agent_bench/models/__init__.py
+++ b/src/coding_agent_bench/models/__init__.py
@@ -9,6 +9,7 @@
RedHatAI_Qwen3_6_27B_FP8,
)
+# Please also update docs/queue-service/nebius.md & docs/validated_models.md when making changes here
MODEL_CONFIGS: list[type[ModelConfig]] = [
Qwen_Qwen3_8_27B,
Qwen_Qwen3_8_27B_FP8,
diff --git a/src/coding_agent_bench/nebius_utils.py b/src/coding_agent_bench/nebius_utils.py
index 314549c..1c37242 100644
--- a/src/coding_agent_bench/nebius_utils.py
+++ b/src/coding_agent_bench/nebius_utils.py
@@ -60,6 +60,7 @@ class H200x8(ResourceConfig):
platform = "gpu-h200-sxm"
preset = "8gpu-128vcpu-1600gb"
+# Please also update docs/queue-service/nebius.md when making changes here.
RESOURCE_CONFIGS: list[ResourceConfig] = [
B200,
B200A,
diff --git a/uv.lock b/uv.lock
index 721d2b4..57e2a6f 100644
--- a/uv.lock
+++ b/uv.lock
@@ -186,6 +186,28 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/86/9f/1b37f7e20a2ba4ca3dc99117d7d3ceaee7dfa1f09bc8aeb4eebe0a85bcbd/awscli-1.46.1-py3-none-any.whl", hash = "sha256:68701ad24347c63b5b145b7aa32391ce7e04f328057dd5aa0537a07c0d0b7cc3", size = 20232487, upload-time = "2026-08-27T21:30:32.54Z" },
]
+[[package]]
+name = "babel"
+version = "2.18.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" },
+]
+
+[[package]]
+name = "backrefs"
+version = "8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ec/56/4744bcd0c82184e80c52b0ac4076c261a8ffa1f1b343ff2f6e89ce0e1cef/backrefs-8.0.tar.gz", hash = "sha256:b556cd7d36c3a3a2f256b89590b176b8eddfb73bcfaee3a3ddd84ea66d21ce50", size = 7013081, upload-time = "2026-07-26T19:54:24.638Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/fd/9bf53b6a6f6f519ffaac765df2f2a25e5c2fc6d32cfd2b2747099e72c911/backrefs-8.0-py310-none-any.whl", hash = "sha256:4a627b817fd2dce43b79ab48da63613340509381cd8ce0897078a0bce79a2ab8", size = 380377, upload-time = "2026-07-26T19:54:17.457Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/29/4bd7ae72a2634da00379c2b3bcc5439e7c94620235c6afea8af15229a973/backrefs-8.0-py311-none-any.whl", hash = "sha256:f0c35cf0102ba6b6070c12a492be3c1c1d3f5839529784b9a9565d6d04569a01", size = 392169, upload-time = "2026-07-26T19:54:18.782Z" },
+ { url = "https://files.pythonhosted.org/packages/29/13/232505664e8e2a0c7a2eb0c505cfade9d715538f89a5d62bc4c272968f62/backrefs-8.0-py312-none-any.whl", hash = "sha256:87f0fae8c5f207fe9f4b2887efc71d42f4900ac78faa1af08d675ef303692dc5", size = 398084, upload-time = "2026-07-26T19:54:19.954Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/69/47a3dc20abc4fa5486655fde681bd55e63211b46c886d8c02223d6468431/backrefs-8.0-py313-none-any.whl", hash = "sha256:601ce68ca12385dbda06ce264406b4c4210cf5b79fd0fd627592365c92f29a88", size = 400040, upload-time = "2026-07-26T19:54:21.194Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/cf/e5f9b68a5b0e939a2fb933a66c20180d0c9241bf8927f7a47fa48c1675e9/backrefs-8.0-py314-none-any.whl", hash = "sha256:9ec96efa080938be92323e8e730e57718c9c88eb15ad70bbef4e1766df591408", size = 411903, upload-time = "2026-07-26T19:54:23.221Z" },
+]
+
[[package]]
name = "bracex"
version = "3.0.1"
@@ -414,6 +436,8 @@ dependencies = [
dev = [
{ name = "bump-my-version" },
{ name = "bump2version" },
+ { name = "mkdocs-material" },
+ { name = "mkdocstrings-python" },
{ name = "pytest" },
{ name = "ruff" },
]
@@ -438,6 +462,8 @@ requires-dist = [
dev = [
{ name = "bump-my-version", specifier = ">=1.5.1" },
{ name = "bump2version", specifier = ">=1.0.1" },
+ { name = "mkdocs-material", specifier = ">=9.7.7" },
+ { name = "mkdocstrings-python", specifier = ">=2.0.8" },
{ name = "pytest", specifier = ">=9.1.1" },
{ name = "ruff", specifier = ">=0.15.12" },
]
@@ -707,6 +733,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e5/22/4222d7ddf3da30f363edaa98e329c2bce6c65497c9cb2810931c8b2c0fbc/fsspec-2026.6.0-py3-none-any.whl", hash = "sha256:02e0b71817df9b2169dc30a16832045764def1191b43dcff5bb85bdee212d2a1", size = 203949, upload-time = "2026-06-16T01:57:26.358Z" },
]
+[[package]]
+name = "ghp-import"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "python-dateutil" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d9/29/d40217cbe2f6b1359e00c6c307bb3fc876ba74068cbab3dde77f03ca0dc4/ghp-import-2.1.0.tar.gz", hash = "sha256:9c535c4c61193c2df8871222567d7fd7e5014d835f97dc7b7439069e2413d343", size = 10943, upload-time = "2022-05-02T15:47:16.11Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" },
+]
+
[[package]]
name = "google-api-core"
version = "2.34.0"
@@ -790,6 +828,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9a/51/186c02b8549b69ccda44429cf6ff5081e4b61a602ddfe6a8020d1be31d1b/googleapis_common_protos-1.75.1-py3-none-any.whl", hash = "sha256:28a1934bcd33b9c9da66ac301a0a4227e3367f095a17d0375cb98f0a09d93b79", size = 300626, upload-time = "2026-08-06T06:23:46.696Z" },
]
+[[package]]
+name = "griffelib"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/27/af/018c10bc9edd42b6ef6db2e96b09542050d5253f9b195e74bc910b2d13ab/griffelib-2.3.0.tar.gz", hash = "sha256:7b0952caf5bca6afa4bb5ee8c6a2d183fe3f21b62efc5f6c7243cb2b26d2d115", size = 234534, upload-time = "2026-09-04T15:08:17.472Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/63/e876e789525063c840ccfa8857febdabd6523bcef9ce7eb979b9305ea895/griffelib-2.3.0-py3-none-any.whl", hash = "sha256:1b8f9cd525681c26b1d6d574faa1371651e8459ca51d209684f50b8096ae06e0", size = 169423, upload-time = "2026-09-04T15:08:12.956Z" },
+]
+
[[package]]
name = "gspread"
version = "6.2.1"
@@ -1179,6 +1226,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/3d/4f/141cf1162eeee762fc839c335e3f78fc309a65f2385023479a20b09e680a/litellm-1.92.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:8f03047a73154f33c2733899e4bf9b5ed129e2e345caa305895a318452e4a807", size = 19289770, upload-time = "2026-07-12T01:15:57.136Z" },
]
+[[package]]
+name = "markdown"
+version = "3.10.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/29/6f/da4c6aea59b3001f2e8c0ec7497475aadaf3b021c10cab5b2858f0f32b26/markdown-3.10.3.tar.gz", hash = "sha256:3589362618f743188b4d955b874402bc814f4f83f544dc207719f4baa7d9c45f", size = 372596, upload-time = "2026-07-30T19:05:29.005Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/69/4a5af2bc115a9a33fefe51709749de8262be3f9ba063d1753a837cdbc49c/markdown-3.10.3-py3-none-any.whl", hash = "sha256:fa6c92a00a4a3c98b22728c64a935ae1928250ae65058a6ded814d2cc29a4cea", size = 110757, upload-time = "2026-07-30T19:05:27.883Z" },
+]
+
[[package]]
name = "markdown-it-py"
version = "4.2.0"
@@ -1263,6 +1319,129 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
]
+[[package]]
+name = "mergedeep"
+version = "1.3.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3a/41/580bb4006e3ed0361b8151a01d324fb03f420815446c7def45d02f74c270/mergedeep-1.3.4.tar.gz", hash = "sha256:0096d52e9dad9939c3d975a774666af186eda617e6ca84df4c94dec30004f2a8", size = 4661, upload-time = "2021-02-05T18:55:30.623Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307", size = 6354, upload-time = "2021-02-05T18:55:29.583Z" },
+]
+
+[[package]]
+name = "mkdocs"
+version = "1.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "ghp-import" },
+ { name = "jinja2" },
+ { name = "markdown" },
+ { name = "markupsafe" },
+ { name = "mergedeep" },
+ { name = "mkdocs-get-deps" },
+ { name = "packaging" },
+ { name = "pathspec" },
+ { name = "pyyaml" },
+ { name = "pyyaml-env-tag" },
+ { name = "watchdog" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bc/c6/bbd4f061bd16b378247f12953ffcb04786a618ce5e904b8c5a01a0309061/mkdocs-1.6.1.tar.gz", hash = "sha256:7b432f01d928c084353ab39c57282f29f92136665bdd6abf7c1ec8d822ef86f2", size = 3889159, upload-time = "2024-08-30T12:24:06.899Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/22/5b/dbc6a8cddc9cfa9c4971d59fb12bb8d42e161b7e7f8cc89e49137c5b279c/mkdocs-1.6.1-py3-none-any.whl", hash = "sha256:db91759624d1647f3f34aa0c3f327dd2601beae39a366d6e064c03468d35c20e", size = 3864451, upload-time = "2024-08-30T12:24:05.054Z" },
+]
+
+[[package]]
+name = "mkdocs-autorefs"
+version = "1.4.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown" },
+ { name = "markupsafe" },
+ { name = "mkdocs" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/52/c0/f641843de3f612a6b48253f39244165acff36657a91cc903633d456ae1ac/mkdocs_autorefs-1.4.4.tar.gz", hash = "sha256:d54a284f27a7346b9c38f1f852177940c222da508e66edc816a0fa55fc6da197", size = 56588, upload-time = "2026-02-10T15:23:55.105Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/28/de/a3e710469772c6a89595fc52816da05c1e164b4c866a89e3cb82fb1b67c5/mkdocs_autorefs-1.4.4-py3-none-any.whl", hash = "sha256:834ef5408d827071ad1bc69e0f39704fa34c7fc05bc8e1c72b227dfdc5c76089", size = 25530, upload-time = "2026-02-10T15:23:53.817Z" },
+]
+
+[[package]]
+name = "mkdocs-get-deps"
+version = "0.2.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mergedeep" },
+ { name = "platformdirs" },
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ce/25/b3cccb187655b9393572bde9b09261d267c3bf2f2cdabe347673be5976a6/mkdocs_get_deps-0.2.2.tar.gz", hash = "sha256:8ee8d5f316cdbbb2834bc1df6e69c08fe769a83e040060de26d3c19fad3599a1", size = 11047, upload-time = "2026-03-10T02:46:33.632Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl", hash = "sha256:e7878cbeac04860b8b5e0ca31d3abad3df9411a75a32cde82f8e44b6c16ff650", size = 9555, upload-time = "2026-03-10T02:46:32.256Z" },
+]
+
+[[package]]
+name = "mkdocs-material"
+version = "9.7.7"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "babel" },
+ { name = "backrefs" },
+ { name = "colorama" },
+ { name = "jinja2" },
+ { name = "markdown" },
+ { name = "mkdocs" },
+ { name = "mkdocs-material-extensions" },
+ { name = "paginate" },
+ { name = "pygments" },
+ { name = "pymdown-extensions" },
+ { name = "requests" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f1/cd/c05d3a530ba7934f144fb45f7203cd236adc25c7bdcc34673d202f4b0278/mkdocs_material-9.7.7.tar.gz", hash = "sha256:c0649c065b1b0512d60aad8c10f947f8e455284475239b364b610f2deb4d0855", size = 4097923, upload-time = "2026-07-17T16:21:33.156Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ad/21/17c1bc9e6f47c972ad66fb2ac2568f99f90f1207eeb6fc3b34d094dba7b5/mkdocs_material-9.7.7-py3-none-any.whl", hash = "sha256:8ea9bb1737a5b524a5f9dcf2e1b4ebda8274ae3008aa7845720a97083bef708f", size = 9305438, upload-time = "2026-07-17T16:21:30.017Z" },
+]
+
+[[package]]
+name = "mkdocs-material-extensions"
+version = "1.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/79/9b/9b4c96d6593b2a541e1cb8b34899a6d021d208bb357042823d4d2cabdbe7/mkdocs_material_extensions-1.3.1.tar.gz", hash = "sha256:10c9511cea88f568257f960358a467d12b970e1f7b2c0e5fb2bb48cab1928443", size = 11847, upload-time = "2023-11-22T19:09:45.208Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl", hash = "sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31", size = 8728, upload-time = "2023-11-22T19:09:43.465Z" },
+]
+
+[[package]]
+name = "mkdocstrings"
+version = "1.0.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jinja2" },
+ { name = "markdown" },
+ { name = "markupsafe" },
+ { name = "mkdocs" },
+ { name = "mkdocs-autorefs" },
+ { name = "pymdown-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/53/71/f85bdf13355073ae15a7375f09879375a830553552e58c1c4b7e0bbc5c8b/mkdocstrings-1.0.6.tar.gz", hash = "sha256:a0b8c2bdd29a6416c80d717aa369bbf7831946bd9f23c2a66db1b1dbe7693dbd", size = 100649, upload-time = "2026-07-11T19:38:05.732Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/5b/4c1902e8bdd5c4db63284e9d101dece4038d4025d6d88850ffe0a1578980/mkdocstrings-1.0.6-py3-none-any.whl", hash = "sha256:2703708697487d1b6d6d7b412e176fa436edf120c1bf81dc9e126b12d00893c7", size = 35787, upload-time = "2026-07-11T19:38:04.417Z" },
+]
+
+[[package]]
+name = "mkdocstrings-python"
+version = "2.0.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "griffelib" },
+ { name = "mkdocs-autorefs" },
+ { name = "mkdocstrings" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c5/65/0db919b7c91bf033b42d04f15ff16a56f4edd6afb26e84d448359660ece3/mkdocstrings_python-2.0.8.tar.gz", hash = "sha256:34545b54681cf0b212aab95394d37b0f5326a93b102f6ad1df3d965c79384c9d", size = 201771, upload-time = "2026-08-31T20:16:06.698Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/b7/a5a854d4e44853f26decc0c5e0d68df052399f3da344a313da4178f00816/mkdocstrings_python-2.0.8-py3-none-any.whl", hash = "sha256:e555d9ef53b0febdeae4dd603144d4f86ec90fef672276645ea3bb61232c3d24", size = 105443, upload-time = "2026-08-31T20:16:05.25Z" },
+]
+
[[package]]
name = "multidict"
version = "6.7.1"
@@ -1399,6 +1578,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" },
]
+[[package]]
+name = "paginate"
+version = "0.5.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ec/46/68dde5b6bc00c1296ec6466ab27dddede6aec9af1b99090e1107091b3b84/paginate-0.5.7.tar.gz", hash = "sha256:22bd083ab41e1a8b4f3690544afb2c60c25e5c9a63a30fa2f483f6c60c8e5945", size = 19252, upload-time = "2024-08-25T14:17:24.139Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl", hash = "sha256:b885e2af73abcf01d9559fd5216b57ef722f8c42affbb63942377668e35c7591", size = 13746, upload-time = "2024-08-25T14:17:22.55Z" },
+]
+
[[package]]
name = "pathspec"
version = "1.1.1"
@@ -1731,6 +1919,19 @@ crypto = [
{ name = "cryptography" },
]
+[[package]]
+name = "pymdown-extensions"
+version = "12.0.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown" },
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/7e/53144b512818c75d459523e3a91d9bf7b65cd6a1dfb964fe690adc8b3670/pymdown_extensions-12.0.1.tar.gz", hash = "sha256:88fe2c97fb9153b9f2cc1da59f1d7c7cb74971110daf33cccb2a5514e0f7e970", size = 866904, upload-time = "2026-09-15T23:20:33.187Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8c/30/c866b3ca8ad5cca79a3ee1a55c12d61d3e72381e5eadbfdaceb6006ebc84/pymdown_extensions-12.0.1-py3-none-any.whl", hash = "sha256:15d629e8035892b93d788a8424ccea754d4aa3cec247a64cbfe7b15089c9ad7f", size = 276957, upload-time = "2026-09-15T23:20:31.551Z" },
+]
+
[[package]]
name = "pyparsing"
version = "3.3.2"
@@ -1823,6 +2024,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
]
+[[package]]
+name = "pyyaml-env-tag"
+version = "1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/2e/79c822141bfd05a853236b504869ebc6b70159afc570e1d5a20641782eaa/pyyaml_env_tag-1.1.tar.gz", hash = "sha256:2eb38b75a2d21ee0475d6d97ec19c63287a7e140231e4214969d0eac923cd7ff", size = 5737, upload-time = "2025-05-13T15:24:01.64Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/04/11/432f32f8097b03e3cd5fe57e88efb685d964e2e5178a48ed61e841f7fdce/pyyaml_env_tag-1.1-py3-none-any.whl", hash = "sha256:17109e1a528561e32f026364712fee1264bc2ea6715120891174ed1b980d2e04", size = 4722, upload-time = "2025-05-13T15:23:59.629Z" },
+]
+
[[package]]
name = "questionary"
version = "2.1.1"
@@ -2460,6 +2673,30 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/45/ec/dbb7e5a6b91f86bfb9eb7d2988a2730907b6a729875b949c7f022e8b88fa/uvicorn-0.51.0-py3-none-any.whl", hash = "sha256:5d38af6cd620f2ae3849fb44fd4879e0890aa1febe8d47eb355fb45d93fe6a5b", size = 73219, upload-time = "2026-07-08T10:59:04.44Z" },
]
+[[package]]
+name = "watchdog"
+version = "6.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/db/7d/7f3d619e951c88ed75c6037b246ddcf2d322812ee8ea189be89511721d54/watchdog-6.0.0.tar.gz", hash = "sha256:9ddf7c82fda3ae8e24decda1338ede66e1c99883db93711d8fb941eaa2d8c282", size = 131220, upload-time = "2024-11-01T14:07:13.037Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/39/ea/3930d07dafc9e286ed356a679aa02d777c06e9bfd1164fa7c19c288a5483/watchdog-6.0.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:bdd4e6f14b8b18c334febb9c4425a878a2ac20efd1e0b231978e7b150f92a948", size = 96471, upload-time = "2024-11-01T14:06:37.745Z" },
+ { url = "https://files.pythonhosted.org/packages/12/87/48361531f70b1f87928b045df868a9fd4e253d9ae087fa4cf3f7113be363/watchdog-6.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c7c15dda13c4eb00d6fb6fc508b3c0ed88b9d5d374056b239c4ad1611125c860", size = 88449, upload-time = "2024-11-01T14:06:39.748Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/7e/8f322f5e600812e6f9a31b75d242631068ca8f4ef0582dd3ae6e72daecc8/watchdog-6.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6f10cb2d5902447c7d0da897e2c6768bca89174d0c6e1e30abec5421af97a5b0", size = 89054, upload-time = "2024-11-01T14:06:41.009Z" },
+ { url = "https://files.pythonhosted.org/packages/68/98/b0345cabdce2041a01293ba483333582891a3bd5769b08eceb0d406056ef/watchdog-6.0.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:490ab2ef84f11129844c23fb14ecf30ef3d8a6abafd3754a6f75ca1e6654136c", size = 96480, upload-time = "2024-11-01T14:06:42.952Z" },
+ { url = "https://files.pythonhosted.org/packages/85/83/cdf13902c626b28eedef7ec4f10745c52aad8a8fe7eb04ed7b1f111ca20e/watchdog-6.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:76aae96b00ae814b181bb25b1b98076d5fc84e8a53cd8885a318b42b6d3a5134", size = 88451, upload-time = "2024-11-01T14:06:45.084Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/c4/225c87bae08c8b9ec99030cd48ae9c4eca050a59bf5c2255853e18c87b50/watchdog-6.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a175f755fc2279e0b7312c0035d52e27211a5bc39719dd529625b1930917345b", size = 89057, upload-time = "2024-11-01T14:06:47.324Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/c7/ca4bf3e518cb57a686b2feb4f55a1892fd9a3dd13f470fca14e00f80ea36/watchdog-6.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:7607498efa04a3542ae3e05e64da8202e58159aa1fa4acddf7678d34a35d4f13", size = 79079, upload-time = "2024-11-01T14:06:59.472Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/51/d46dc9332f9a647593c947b4b88e2381c8dfc0942d15b8edc0310fa4abb1/watchdog-6.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:9041567ee8953024c83343288ccc458fd0a2d811d6a0fd68c4c22609e3490379", size = 79078, upload-time = "2024-11-01T14:07:01.431Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/57/04edbf5e169cd318d5f07b4766fee38e825d64b6913ca157ca32d1a42267/watchdog-6.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:82dc3e3143c7e38ec49d61af98d6558288c415eac98486a5c581726e0737c00e", size = 79076, upload-time = "2024-11-01T14:07:02.568Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/cc/da8422b300e13cb187d2203f20b9253e91058aaf7db65b74142013478e66/watchdog-6.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:212ac9b8bf1161dc91bd09c048048a95ca3a4c4f5e5d4a7d1b1a7d5752a7f96f", size = 79077, upload-time = "2024-11-01T14:07:03.893Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/3b/b8964e04ae1a025c44ba8e4291f86e97fac443bca31de8bd98d3263d2fcf/watchdog-6.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:e3df4cbb9a450c6d49318f6d14f4bbc80d763fa587ba46ec86f99f9e6876bb26", size = 79078, upload-time = "2024-11-01T14:07:05.189Z" },
+ { url = "https://files.pythonhosted.org/packages/62/ae/a696eb424bedff7407801c257d4b1afda455fe40821a2be430e173660e81/watchdog-6.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:2cce7cfc2008eb51feb6aab51251fd79b85d9894e98ba847408f662b3395ca3c", size = 79077, upload-time = "2024-11-01T14:07:06.376Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/e8/dbf020b4d98251a9860752a094d09a65e1b436ad181faf929983f697048f/watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2", size = 79078, upload-time = "2024-11-01T14:07:07.547Z" },
+ { url = "https://files.pythonhosted.org/packages/07/f6/d0e5b343768e8bcb4cda79f0f2f55051bf26177ecd5651f84c07567461cf/watchdog-6.0.0-py3-none-win32.whl", hash = "sha256:07df1fdd701c5d4c8e55ef6cf55b8f0120fe1aef7ef39a1c6fc6bc2e606d517a", size = 79065, upload-time = "2024-11-01T14:07:09.525Z" },
+ { url = "https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl", hash = "sha256:cbafb470cf848d93b5d013e2ecb245d4aa1c8fd0504e863ccefa32445359d680", size = 79070, upload-time = "2024-11-01T14:07:10.686Z" },
+ { url = "https://files.pythonhosted.org/packages/33/e8/e40370e6d74ddba47f002a32919d91310d6074130fe4e17dabcafc15cbf1/watchdog-6.0.0-py3-none-win_ia64.whl", hash = "sha256:a1914259fa9e1454315171103c6a30961236f508b9b623eae470268bbcc6a22f", size = 79067, upload-time = "2024-11-01T14:07:11.845Z" },
+]
+
[[package]]
name = "wcmatch"
version = "11.0.1"
]