diff --git a/.runpod/hub.json b/.runpod/hub.json index 279140b..daf2420 100644 --- a/.runpod/hub.json +++ b/.runpod/hub.json @@ -1,6 +1,6 @@ { - "title": "Holo4 (Holo4-27B-GGUF, Holo4-35B-A3B-GGUF)", - "description": "Serve Holo4 (Holo4-27B-GGUF, Holo4-35B-A3B-GGUF) on Runpod Serverless with vLLM. Computer use agents are a fast growing self-host workload, and teams want them on their own GPUs for privacy and cost control over long screenshot-heavy loops. Two sizes cover different rentals: the 27B dense fits a single 48GB to 80GB card, while the 35B-A3B MoE runs cheap per token and scales well for parallel agent sessions. GGUF weights mean day-one llama.cpp and Ollama deploys with the hai-agents harness on Runpod.", + "title": "Holo4-35B-A3B", + "description": "Serve Holo4-35B-A3B (H Company) on Runpod Serverless with vLLM.", "type": "serverless", "category": "image", "iconUrl": "https://www.runpod.io/favicon.ico", diff --git a/README.md b/README.md index ab73b7e..870e972 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -# Holo4 (Holo4-27B-GGUF, Holo4-35B-A3B-GGUF) on Runpod Serverless +# Holo4-35B-A3B on Runpod Serverless [![Runpod](https://api.runpod.io/badge/runpod-workers/worker-holo4)](https://console.runpod.io/hub/runpod-workers/worker-holo4) -Serve Holo4 (Holo4-27B-GGUF, Holo4-35B-A3B-GGUF) (H Company) on Runpod Serverless with vLLM. +Serve Holo4-35B-A3B (H Company) on Runpod Serverless with vLLM. License: apache-2.0. ## Recipe @@ -10,7 +10,7 @@ License: apache-2.0. | Setting | Value | |---|---| | Engine | vllm | -| Image | `runpod/worker-v1-vllm:v2.27.0` | +| Image | this repo's `Dockerfile`, on `vllm/vllm-openai:v0.29.0` | | GPU | 1x RTX 6000 Ada | | Precision | fp8 | | Max model length | 8192 | @@ -22,6 +22,7 @@ License: apache-2.0. | `GPU_MEMORY_UTILIZATION` | `0.90` | | `MAX_CONCURRENCY` | `30` | | `MAX_MODEL_LEN` | `8192` | +| `MAX_NUM_SEQS` | `32` | | `MODEL_NAME` | `Hcompany/Holo4-35B-A3B` | | `QUANTIZATION` | `fp8` | | `TENSOR_PARALLEL_SIZE` | `1` | @@ -33,26 +34,12 @@ License: apache-2.0. | 1x RTX 6000 Ada fp8 | $0.84 | 362.6 | 351.9 | 123.7 | $1.89 | Cost per 1M output tokens is the hourly rate divided by measured throughput. It -assumes one saturated worker and no idle time, so treat it as a floor. +assumes one saturated worker and no idle time, so treat it as a floor. Measured by Launch +Builder on `runpod/worker-v1-vllm:v2.27.0` with the same model, GPU, precision and context +length; this worker runs vLLM v0.29.0. ## Deploy links -- Deploy on Runpod: https://console.runpod.io/deploy?template=yye0eu6dtk&utm_source=hub&utm_medium=product&utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf&utm_content=readme -- Model page: https://www.runpod.io/models/h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf?utm_source=hub&utm_medium=product&utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf&utm_content=readme -- Docs: https://docs.runpod.io/public-endpoints/models/h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf?utm_source=hub&utm_medium=product&utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf&utm_content=readme +- Deploy on Runpod: https://console.runpod.io/hub/runpod-workers/worker-holo4?utm_source=hub&utm_medium=product&utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf&utm_content=readme -Every link carries `utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf`. Keep it intact when you copy a link anywhere else. - -## Before merging this bundle - -Run the suite against a live endpoint: - -```bash -node hub-test-suite.mjs --repo / --prefix h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf- --create -``` - -## Fast path while this is in review - -```bash -runpodctl serverless create --hub-id --model-reference hf://Hcompany/Holo4-35B-A3B -``` +The link carries `utm_campaign=202609_activation_ml-engineer_h-company-holo4-holo4-27b-gguf-holo4-35b-a3b-gguf`. Keep it intact when you copy a link anywhere else.