[realAlexBarge]

Problem

A startup moves an open-weight or fine-tuned model off a per-token API and starts vLLM or SGLang with defaults on whatever GPU it could get. The result is a KV cache exhausted under real concurrency, a GPU idle most of the day, or a cost per token above the API it left. The company has no committed GPU fleet to absorb an idle GPU, so every replica is new spend from a fixed cost ceiling, and one replica can be a material share of it.

Closes #346.

Solution

Add the self-hosted-llm-serving skill with a thin SKILL.md router (metadata.audience: startup) and six references. Each file opens with the decision it covers and what a committed GPU fleet would allow instead.

• SKILL.md: a table of six decisions with the committed-fleet answer, this skill's answer and the reason; scope; the four ordered decisions; required deliverables; upstream Skill() pointers and anti-patterns. Every number comes from a log, a benchmark or a price lookup, at on-demand list price before credits.
• self-host-or-managed.md: a break-even traffic number against per-token Bedrock, Custom Model Import and SageMaker inference components.
• fit-one-gpu.md: weights plus KV cache at measured context and concurrency, the startup log lines that confirm it, and the fixes to try before a second GPU.
• engine-configuration.md: pinned vLLM and SGLang versions, and each fix from fit-one-gpu.md mapped to its engine flag. GPU node setup routes to aws-containers and aws-compute.
• multi-lora.md: serve several fine-tunes as LoRA adapters on one base model instead of one endpoint each.
• idle-floor.md: one warm replica or scale to zero, priced against measured cold start, and how to shorten it.
• benchmark-on-target.md: a concurrency sweep on the target instance against a latency target.

Also adds the skill to the plugin README's skills table.

SageMaker endpoints and Custom Model Import route to aws-core, offline batch work routes to self-hosted-llm-batch-inference, and multi-node serving is out of scope.

Type of Change

• New plugin/power/tool
• Bug fix
• Enhancement to existing content
• Documentation update
• Guardrail/CI update

Team Folder

• advisor/
• migrate/
• solution-architecture/
• Other: ___

Validation

• mise run fmt, mise run lint:md and mise run build passed.
• Plugin validation passed.
• The Solution Architecture contribution gate passed, both over the whole folder and on the files this PR changes.
• All eight skills named in Skill() pointers exist (agent-toolkit-for-aws at 188af2f for aws-core), and all 25 cited URLs returned HTTP 200.
• With aws-core and aws-agents loaded, two runs per prompt on one model: serving prompts activated this skill, SageMaker or Custom Model Import prompts activated aws-ai-ml, and the batch prompt activated self-hosted-llm-batch-inference, in 18 of 18 runs. A prompt about an enterprise with a reserved GPU fleet activated no skill in 2 of 2 runs.
• Not done: an end-to-end run of the procedure on a GPU instance.

Detailed validation notes are available on request.

Checklist

• I have read the CONTRIBUTING.md guidelines
• My changes do not include hardcoded secrets, credentials, or internal-only content
• I have run mise run build locally and it passes
• I have updated documentation if needed
• My changes are scoped to my team's folder only

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[solution-architecture-reviewer]

solution-architecture-reviewer bot: 3 item(s) for a human

Not approving, not blocking. Something below is worth a look, but none of it is the solution-architecture-reviewer bot's call to settle: a human decides.

Reviewed 7 changed file(s).

Axis assessment

• 🟡 PR hygiene (advisory) — The PR body states validation was done on one model only and that an end-to-end run of the procedure on a GPU instance was not done; this is disclosed, which is what this axis asks for, but a human should decide whether that disclosed gap is acceptable for a new skill of this scope before merge.
• 🟡 Upstream duplication (advisory) — SKILL.md names vLLM/SGLang serving mechanics, GPU sizing, and benchmarking in detail; the supplied aws-compute text covers EC2/ASG/instance selection generally but not GPU memory sizing, KV cache, or engine-specific serving, so no direct overlap is checkable against the text given. aws-ai-ml's and aws-agents' full bodies were not supplied, so I cannot confirm or rule out overlap with SageMaker inference optimization or AgentCore serving guidance at concern-level confidence; a human should check whether aws-ai-ml's benchmarking/optimization content already covers the generic engine-tuning material here.
• 🟡 Scope (startup-specific) (advisory) — The skill is framed tightly around the fixed-cost-ceiling constraint (Every GPU replica is paid from a fixed cost ceiling with no committed fleet to absorb an idle GPU) and most content (break-even math, idle floor, one-GPU-first ordering) is genuinely reshaped by that constraint, which looks like a pass on criterion 1. However, large stretches of engine-configuration.md and fit-one-gpu.md (pinned vLLM/SGLang flags, GPU memory tables, ALB idle timeout settings, EKS/ECS GPU driver prerequisites) read identically for an enterprise with a committed GPU fleet, since the mechanics of fitting weights plus KV cache on a GPU do not change with the cost ceiling, only the threshold for when a second GPU is tolerable does. A human should weigh whether these generic-engine-mechanics sections tip criterion 1 toward Agent Toolkit for AWS scope despite the startup-specific framing wrapped around them.
• ✅ Contradiction — No two files in this PR give guidance that cannot both be followed; the skill is internally consistent and its routing to aws-core skills and to self-hosted-llm-batch-inference does not conflict with anything shown.
• ✅ Staleness — The skill pins specific dated versions (vLLM v0.31.0 released 2026-10-05, SGLang v0.5.21 released 2026-10-02) and explicitly instructs re-verifying against current docs rather than trusting the date, which is the opposite of staleness. The one sunset-service mention (Text Generation Inference (TGI) in engine-configuration.md:15) warns against it and recommends vLLM/SGLang instead, which is the warn framing criterion 3 wants, not a recommend framing.
• ✅ Manifest drift — The new skill directory, its six references/ files, and the SKILL.md frontmatter (audience: startup) match what the PR description claims to add; no counts or cross-file version claims disagree with the tree shown.
• ✅ Activation quality (advisory) — The SKILL.md description is long but specific: it names the engines (vLLM, SGLang), the platforms (EKS, ECS, EC2), the cost-ceiling constraint, and explicit negative scope (Not for running a model on SageMaker endpoints or Bedrock Custom Model Import, offline batch inference, training, or serving one model across several nodes), which should make activation against aws-ai-ml and self-hosted-llm-batch-inference reasonably unambiguous.
• ✅ Injection posture (advisory) — This skill does not consume untrusted repo contents, user files, or web pages at runtime; it is a planning/decision skill over measured infrastructure numbers, so an untrusted-content rule is not applicable here.
• ✅ Authoring conventions (advisory) — The SKILL.md is a thin router linking to six references/ files as the authoring conventions prescribe, body text is in imperative/third-person style consistent with solution-architecture/CONTRIBUTING.md, and reference files use the prescribed **references/x.md**: Read when... link format, e.g. references/self-host-or-managed.md: Read first, before choosing any GPU.`.

🔴 concern · 🟡 borderline · ✅ clear. Advisory axes are reported but never block, since the contribution gate reserves those criteria for human reviewers.

_{Reviewed 7 file(s). Model: us.anthropic.claude-sonnet-5. Findings below 0.6 confidence are surfaced as borderline rather than hedged. Judgment-only: formatting, schema, and security scanning are covered by existing CI.}

[realAlexBarge]

Replies to the items marked for a human, checked at d56c1ed:

• Scope: CONTRIBUTING.md line 98 sets the criterion 1 test: "Naming the startup constraint is not the same as the constraint changing the answer". The SKILL.md decisions carry the constraint: stay managed until measured traffic crosses the break-even against the idle GPU floor (line 35), one GPU before two (line 39), adapters instead of an endpoint per fine-tune (line 43), one warm replica or scale to zero (line 47). The reference mechanics implement them.
• aws-ai-ml overlap: its model-deployment references cover deployment to SageMaker endpoints and Bedrock, benchmarking an existing SageMaker endpoint, and AI Recommendation Jobs that deploy a model on candidate instance types and rank them (overview.md lines 21-30, recommendation-workflow.md line 3). None of the 15 model-deployment files mentions EKS, ECS, SGLang, KV cache or self-hosting, and vLLM appears only as a serving framework for SageMaker endpoints. This skill routes SageMaker and Custom Model Import there (SKILL.md line 66, benchmark-on-target.md line 5).
• GPU run: not done, as the PR states. It needs a GPU instance; I can run one on request and post the startup log and benchmark here.

[realAlexBarge]

@dgeske, I applied your #347 standard here too, in 6b56d92 and 8cca887 (which removes self-hosted serving from the open gap list):

• Startup-specific throughout: SKILL.md opens with a table of six decisions, each with what a committed GPU fleet allows, this skill's answer and why, and every reference opens with that contrast. The break-even now uses on-demand list price before credits.
• Trigger: the description rests on the company having no committed GPU fleet, so one idle GPU is a material share of a fixed ceiling. "No platform team" is gone, because a small platform team in a large company has the same gap.
• README: the skill is in the skills table. CODEOWNERS routes SKILL.md to @awslabs/startups-admins, so this also needs an admin approval.

Bot items:

• Scope (engine and platform sections): fixed. GPU node setup on EKS, ECS and EC2 points to aws-core:aws-containers and aws-core:aws-compute; the ALB idle timeout, per-GPU memory list, TGI note and two flag rows are cut. The vLLM and SGLang flag table stays, mapped to the fit-one-GPU order, because no upstream skill covers self-managed engines.
• Upstream duplication: re-checked at agent-toolkit-for-aws 188af2f. No aws-core or aws-agents file mentions SGLang or KV cache, and vLLM appears only in aws-ai-ml model-deployment files for SageMaker endpoints.
• PR hygiene: the GPU run is still not done. Trigger rerun on this commit: the 2026-10-05 prompts route as before in 18 of 18 runs, and an enterprise with a reserved GPU fleet loads no skill in 2 of 2.

[solution-architecture-reviewer]

solution-architecture-reviewer bot: 4 item(s) for a human

Not approving, not blocking. Something below is worth a look, but none of it is the solution-architecture-reviewer bot's call to settle: a human decides.

Reviewed 8 changed file(s).

Axis assessment

• 🟡 Staleness — references/engine-configuration.md:17 and nearby lines pin vLLM v0.31.0 and SGLang v0.5.21 with release dates in October 2026, which reads as either a forward-dated placeholder or a typo; a human should confirm these are real, current versions before merge since downstream flag defaults are cited from them.
• 🟡 Upstream duplication (advisory) — Per the supplied aws-core/aws-compute and aws-core/amazon-bedrock text, no upstream skill covers vLLM/SGLang self-managed serving, KV cache sizing, or multi-LoRA adapters, so the core of this skill is not duplicative; fit-one-gpu.md's GPU pricing/memory tables border on aws-compute's instance-selection remit but the skill routes node setup upstream, so this is a judgment call left to a human reviewer rather than a clear violation.
• 🟡 Scope (startup-specific) (advisory) — The skill's decision table and references consistently frame answers around the no-committed-GPU-fleet constraint (break-even traffic, one-GPU-before-two, adapters over per-fine-tune endpoints, warm-replica-vs-zero), which is the kind of constraint-reshapes-the-answer content criterion 1 asks for; a human should still confirm the GPU pricing/memory-ratio tables in fit-one-gpu.md are substantively startup-specific rather than general GPU-selection advice restated with a startup framing, since that file is dense with general AWS pricing mechanics alongside the startup-specific break-even framing.
• ✅ Contradiction — No cross-file contradiction found; the new skill's guidance (stay managed until break-even, prefer one GPU, adapters over endpoints, warm replica or scale-to-zero) is internally consistent and does not conflict with sibling skills like multi-tenant-isolation, agentcore-patterns, or cost-regression-check, nor with the README.md skills table.
• ✅ Manifest drift — The README.md skills table addition for self-hosted-llm-serving matches the new skill directory and files added in this PR, and the removal of the 'Self-hosted inference serving' bullet from the open-gap list (README.md) correctly reflects that the gap is now filled by this PR.
• ✅ Activation quality (advisory) — The self-hosted-llm-serving description in SKILL.md is detailed, names concrete triggers (KV cache exhaustion, idle GPU, cost-per-token regression) and explicit exclusions (SageMaker endpoints, Bedrock Custom Model Import, batch, multi-node), and does not obviously overlap with sibling skills multi-tenant-isolation, agentcore-patterns, or cost-regression-check in this plugin.
• ✅ Injection posture (advisory) — This skill does not consume untrusted repo contents, user files, or web pages as input in a way that requires an untrusted-content rule; it operates on measured logs, benchmarks, and price lookups, which is the same posture as sibling skills in this plugin.
• ✅ PR hygiene (advisory) — The PR description and prior review thread disclose the RFC linkage (Closes #346), the deliberate scope narrowing (engine/platform sections routed to aws-core), and the undone GPU end-to-end run, all explicitly flagged rather than hidden; these are sequencing/disclosure matters already settled by the human thread.
• ✅ Authoring conventions (advisory) — SKILL.md frontmatter uses only name, description, license, and metadata.audience, consistent with the folder's documented allowed fields, and the body follows the thin-router pattern with bolded backticked reference links, matching the agentcore-patterns exemplar and the supplied skill-development conventions.

🔴 concern · 🟡 borderline · ✅ clear. Advisory axes are reported but never block, since the contribution gate reserves those criteria for human reviewers.

Borderline, for human judgement

Below the reporting threshold, so not stated as findings. A suspicion a reviewer can check beats one silently dropped.

• 🟡 staleness solution-architecture/plugins/aws-startups-solution-architecture/skills/self-hosted-llm-serving/references/engine-configuration.md:17 — The skill pins vLLM v0.31.0 and SGLang v0.5.21 as current with release dates in October 2026, which is a future date relative to any plausible review window and cannot be verified as a real, current release. engine-configuration.md:17 states 'vLLM v0.31.0 (released 2026-10-05)' and the SGLang line gives a 2026-10-02 release date. These look like placeholder or speculative version/date combinations rather than verified current releases; if they are fabricated, every downstream flag name and default cited from these versions (--gpu-memory-utilization defaulting to 0.92, etc.) is unverifiable. A human should confirm these versions and dates are real before merge.

_{Reviewed 8 file(s). Model: us.anthropic.claude-sonnet-5. Findings below 0.6 confidence are surfaced as borderline rather than hedged. Judgment-only: formatting, schema, and security scanning are covered by existing CI.}

[rsmets]

@realAlexBarge please resolve the conflicts that have come up

[solution-architecture-reviewer]

solution-architecture-reviewer bot: 2 item(s) for a human

Not approving, not blocking. Something below is worth a look, but none of it is the solution-architecture-reviewer bot's call to settle: a human decides.

Reviewed 8 changed file(s).

Axis assessment

• 🟡 Upstream duplication (advisory) — SKILL.md's engine-configuration table maps vLLM/SGLang flags to memory fixes, which is self-managed-engine mechanics not covered by the supplied aws-core/aws-compute or aws-ai-ml description text (per the maintainer's own re-check, no aws-core/aws-agents file mentions SGLang or KV cache). Without the full aws-ai-ml body supplied, I can't fully rule out overlap on model-deployment benchmarking/sizing mechanics, so this stays borderline rather than a concern, consistent with the human thread's own conclusion that it was checked and found non-overlapping.
• 🟡 Scope (startup-specific) (advisory) — The skill's core break-even/one-GPU/adapters/idle-floor decisions are plausibly startup-shaped per criterion 1, as the human thread argues, but large stretches of fit-one-gpu.md, engine-configuration.md, and benchmark-on-target.md are detailed vLLM/SGLang/EC2-GPU mechanics (flag tables, log line formats, benchmark CLI flags) that would read identically for an enterprise with a committed fleet once the floor arithmetic is set aside — criterion 1 explicitly reserves this judgment to human review, and the maintainer thread has already weighed in, so this is left open rather than asserted as a defect.
• ✅ Contradiction — No cross-file contradiction found; the new skill's guidance (stay managed until break-even, bound before adding a GPU, adapters over one-endpoint-per-fine-tune) is consistent with the rest of the plugin and does not conflict with aws-core or advisor/ guidance supplied.
• ✅ Staleness — All dated facts (2026-10-05 price lookups, pinned vLLM v0.31.0 and SGLang v0.5.21) are presented as measured-on-this-date values with explicit re-check instructions, not as evergreen claims; no sunset service is recommended anywhere in the diff — Bedrock Custom Model Import, SageMaker endpoints and Spot are all discussed factually or as comparators, never recommended as the final answer over a non-deprecated alternative.
• ✅ Manifest drift — The README.md skills table and the 'What to contribute' gap list are updated consistently: the new self-hosted-llm-serving row is added and the old 'Self-hosted inference serving' gap bullet is removed in the same diff, so the manifest and the gap list agree with the new skill's existence.
• ✅ Activation quality (advisory) — The self-hosted-llm-serving description is detailed and scoped (online self-managed serving on vLLM/SGLang with no committed GPU fleet) and explicitly excludes SageMaker endpoints, Custom Model Import, batch, training, and multi-node serving, with disclosed validation runs showing clean routing against sibling skills and aws-ai-ml; no overlap with a sibling in this plugin's table is apparent.
• ✅ Injection posture (advisory) — The skill consumes production logs, benchmark output, and prompt samples, but these are operator-supplied measurement artifacts rather than untrusted third-party content (e.g. web pages or arbitrary repo contents), and benchmark-on-target.md:9 even instructs removing personal/customer data before use, which is the closest sibling convention to an untrusted-content rule for this kind of input.
• ✅ PR hygiene (advisory) — The PR discloses the one undone item (no end-to-end GPU run) plainly in its description and the discussion thread, and the human reviewer thread already resolved the scope/overlap/PR-hygiene questions that were flagged; per instructions this sequencing is left to the people who own the process.
• ✅ Authoring conventions (advisory) — SKILL.md frontmatter uses only name, description, license, and metadata.audience (no banned version or when_to_use), the description opens with the required third-person This skill should be used when, and the file follows the thin-router-plus-references/ layout mandated by solution-architecture/CONTRIBUTING.md, consistent with the exemplar agentcore-patterns skill cited in the README.

🔴 concern · 🟡 borderline · ✅ clear. Advisory axes are reported but never block, since the contribution gate reserves those criteria for human reviewers.

_{Reviewed 8 file(s). Model: us.anthropic.claude-sonnet-5. Findings below 0.6 confidence are surfaced as borderline rather than hedged. Judgment-only: formatting, schema, and security scanning are covered by existing CI.}