[Reference Architecture] Secure AI Agent Sandboxes with Kata Containers at Ant Group

[xiaoya-yaya]

Name

Secure AI Agent Sandboxes with Kata Containers at Ant Group

Organization

Ant Group (CNCF Gold End User Member)

Team(s)

Ant Group engineers who maintain and operate Kata Containers and related cloud-native infrastructure. The Xolis reference implementation was built by Xu Wang, with Yu Hu from Japan AI, for their KubeCon + CloudNativeCon Japan 2026 session.

Overview and Goals

AI agents sometimes need to run code they have just generated and work with user-provided files. Xolis gives each task a short-lived environment with a clear tenant boundary and its own guest kernel. That environment is cleaned up whether the task finishes normally or fails halfway through.

This submission joins two bodies of work:

• Ant Group's production foundation: Kubernetes, containerd, Kata Containers, Dragonfly, and Nydus have been used at scale in Ant's infrastructure. Ant has publicly described a Kubernetes cluster of up to 15,000 nodes and long-running production use of Kata's runtime-rs and Dragonball stack.
• Xolis: an Apache-2.0 reference implementation that makes the agent-sandbox lifecycle reproducible on AWS. It was demonstrated at KubeCon Japan 2026 and is available in the kubecon-japan-2026 tag. A later demo-with-pvm tag adds an experimental PVM path.

Xolis is currently a development and architecture-validation environment. Its value here is that it turns production experience with secure containers and artifact delivery into a concrete design that the CNCF end-user community can inspect, reproduce, and improve.

Architecture

The runtime path is deliberately conventional. Kubernetes Agent Sandbox manages SandboxTemplate, SandboxWarmPool, and SandboxClaim. A claim lands on a dedicated, labelled and tainted node; RuntimeClass selects Kata's containerd handler; runtime-rs and its built-in Dragonball VMM create the guest. The Xolis API exposes tenant-scoped command, file, streaming, and interactive PTY operations against a bounded runtime inside that guest.

The image path is independent of runtime selection. Ordinary OCI is the validated default. Nydus is an opt-in containerd snapshotter path for lazy image reads, while Dragonfly is the production-proven distribution layer that the reference architecture intends to evaluate at cluster scale. Keeping these paths separate avoids treating Nydus as a Kubernetes runtime.

The AWS lab uses one EKS system node and a dedicated sandbox-node Auto Scaling group. Its test tool explicitly moves sandbox capacity through 0 -> 1 -> 0; this is a reproducible lifecycle test, not yet workload-driven autoscaling. Cleanup runs after success and failure, recording command logs and Kubernetes resource snapshots before deleting claims and returning disposable capacity to zero. These snapshots are diagnostic evidence, not VM checkpoints.

KVM and PVM paths

The KubeCon version uses Kata 4.0.0 on supported M8i instances with EC2 nested virtualization enabled. The later experimental PVM path runs Kata on an x86 EC2 VM without vmx or svm, using pinned PVM host and guest kernels and a dedicated xolis-kata-pvm handler. On the tested EKS node, Kata guests boot, follow CNI and NetworkPolicy, serve cold and warm claims, and recover through cleanup after node loss. Interactive PTY also passes. PVM is still an experimental qualification release; a model-backed Hermes workflow and broader failure injection remain open.

What has been measured

The repository includes machine-readable lifecycle reports and a bounded cold-versus-warm comparison. Five sequential samples on one already-Ready c7i.xlarge PVM node measured mean claim-to-Ready time of 8.806 seconds with a zero-replica pool and 1.385 seconds with one Ready sandbox. This result checks that the warm-pool path behaves as expected. It does not establish tail latency, concurrency, density, or production performance.

Adoption indicators

• Established: Kubernetes, containerd, Kata Containers, Dragonfly, and Nydus in Ant Group's production infrastructure.
• Developing: the Xolis lifecycle on AWS, including Kubernetes Agent Sandbox resources, Kata isolation, tenant-scoped APIs, bounded execution, network policy, TTL cleanup, interactive sessions, and the experimental PVM path.
• Emerging: workload-driven capacity management, high availability, multi-cloud support, model-backed Hermes validation on PVM, large-cluster Dragonfly distribution, and ModelPack-based model delivery.

Public references: Xolis project blog, Xolis status and measurements, KubeCon Japan 2026 session, Ant Group and Kata Containers, Ant Group's CNCF end-user story, and Dragonfly and Nydus at Ant Group.

Projects

Project or service	Role and current use
Kubernetes	Control plane, scheduling, Secrets, lifecycle resources, dedicated node pools, and network policy. Established at Ant; used in Xolis through EKS.
Kubernetes Agent Sandbox	SandboxTemplate, SandboxWarmPool, and SandboxClaim APIs used by Xolis. Developing.
containerd	CRI runtime configuration and snapshotter integration. Established at Ant and used in Xolis.
Kata Containers	VM-backed isolation through runtime-rs, Dragonball, and a guest kernel. Established at Ant and used in Xolis.
Dragonfly	P2P distribution used at scale by Ant. Large-cluster integration in Xolis remains emerging.
Nydus	Production image acceleration at Ant; an opt-in lazy-loading path in Xolis. Ordinary OCI remains the default and fallback.
PVM	Experimental KVM backend for x86 hosts without hardware virtualization extensions. Qualified in the bounded Xolis AWS lab, not a production support claim.
Xolis	Rust service API, tenant boundary, lifecycle tooling, evidence capture, and bounded Python runtime. Developing.
OpenTofu	Reproducible AWS lab provisioning, remote state, and explicit plan/apply/destroy workflow. Developing.
Hermes Agent and ZenMux	Interactive agent and external inference path used in the demo. The PVM path has only validated Hermes CLI bootstrap so far.
ModelPack	Candidate format for future OCI model artifacts and model-aware delivery with Dragonfly. Emerging.

Planned Evolution

• Separate the provider-neutral API, lifecycle, profile policy, and cleanup rules from the current AWS adapter before adding another cloud.
• Replace operator-driven capacity changes with a production capacity controller, then test concurrency, node loss, image-cache misses, and cleanup under failure.
• Add production access controls and a highly available service topology. The current lab uses header-based tenant identity, one system node, and at most one sandbox node.
• Complete the PVM qualification gates, including a model-backed Hermes workflow, broader failure injection, and kernel operations policy.
• Evaluate Dragonfly distribution at cluster scale. Add ModelPack when local model artifacts become part of the sandbox workload.
• Share reusable integration work and measured findings with Kubernetes Agent Sandbox, Kata Containers, Dragonfly, Nydus, PVM, and ModelPack communities.