[Reference Architecture] One Heterogeneous Compute Pool, Two Workload Types: How China Merchants Bank Unified Training and Inference on Kubernetes

[tpx818]

Name

Unified Training and Inference on Kubernetes for Heterogeneous AI Compute at China Merchants Bank

Organization

China Merchants Bank; Alibaba Cloud

Team(s)

Architecture Management Team, Information Technology Department, China Merchants Bank

Overview and Goals

Context and problem space

China Merchants Bank operates an enterprise AI platform with nearly 10,000 heterogeneous accelerator cards. The platform supports large-model training, fine-tuning, online inference, and other AI workloads that share the same expensive compute pool but have fundamentally different operating requirements.

Distributed training jobs are long-running and throughput-oriented. They require stable capacity and should begin only when all required workers and accelerator resources are available. Online inference is request-driven and latency-sensitive, so it must scale with changing traffic while avoiding the cost of keeping excess capacity continuously active. Multi-tenant LoRA fine-tuning adds another challenge: tenant state must remain isolated without loading a separate copy of the same large base model for every tenant.

The bank's goal was not simply to add more accelerator cards. It was to use existing capacity more efficiently while preserving workload isolation, operational control, and the ability for training and inference technologies to evolve independently.

Architecture goals

• Provide one Kubernetes-based control plane while preserving separate execution policies for training and inference.
• Manage training admission and quotas before compute and data resources are provisioned.
• Scale inference services from live demand signals rather than static peak-capacity reservations.
• Share heterogeneous accelerator capacity at fine granularity without allowing smaller workloads to monopolize a full card.
• Reuse one base-model runtime across multiple LoRA tenants while keeping adapters, configurations, and tenant state separate.
• Provide shared access to datasets, model weights, checkpoints, the model registry, and operational metrics.
• Allow training frameworks, inference engines, and models to evolve without tightly coupling the entire platform to one runtime or vendor stack.

Architecture overview

The architecture uses one shared Kubernetes foundation with two decoupled workload paths.

Training path: Training requests enter through the Training API. Kueue applies queue-, quota-, and admission-based control before the workload starts. After admission, Kubernetes Scheduler selects suitable nodes, HAMi assigns accelerator capacity, and Fluid provides access to datasets and checkpoints. Twinkle runs multi-tenant training on Ray and passes resulting checkpoints into the downstream model-management process.

Inference path: Online requests enter through the Inference Gateway and are served by vLLM or SGLang on Ray. Prometheus collects service and demand metrics, and KEDA uses those signals to add or remove replicas. Kubernetes Scheduler and HAMi place new Pods and allocate accelerator resources, while Fluid supplies cached model weights so newly created replicas can become ready more quickly.

Both paths share the Kubernetes control plane, accelerator node pool, data and model access layer, model registry, and observability foundation. They remain separate at the workload-policy and runtime levels so each can use the behavior best suited to its demand pattern.

Key design decisions and trade-offs

• Share infrastructure without forcing one operating policy. Training uses queue- and quota-based admission through Kueue, while inference uses metric-driven elasticity through Prometheus and KEDA.
• Separate admission from placement and allocation. Kueue decides when a training workload may enter the cluster; Kubernetes Scheduler and HAMi determine where Pods run and how accelerator capacity is allocated.
• Treat compute sharing and model sharing as different concerns. HAMi allocates physical accelerator capacity, while Twinkle enables multiple LoRA tenants to reuse one base-model instance. Neither mechanism replaces the other.
• Treat model readiness as part of inference elasticity. Creating a replica is not sufficient if it must still wait for model weights. Fluid's compute-side cache reduces this model-loading delay.
• Use unit cost to assess efficiency at growing scale. Cost per one million input and output tokens is more meaningful than total spend when business volume is changing.

Production outcomes

Metric	Result
Average accelerator compute utilization	Increased from 35% to more than 60%
Inference unit cost	Decreased by more than 60% per 1M tokens, counting input and output combined
Accelerator resource usage for the default five-tenant configuration	Reduced by 80% by reducing base-model replicas from five to one
Multi-tenant training density	Increased fivefold; eight concurrent tenants have been validated and five are used by default in production

These results use the bank's internally consistent before-and-after methodology and comparable model and service conditions.

Reusable lessons

• A shared compute pool does not require training and inference to use the same operating policy.
• Inference elasticity depends on model readiness, not only on replica scheduling.
• Physical accelerator sharing and base-model runtime reuse solve different sources of waste.
• Kubernetes-native interfaces allow the control plane, resource layer, data path, monitoring system, and workload runtimes to evolve with limited coupling.
• Accelerator utilization, inference cost per token, and training density measure different problems and should be defined and evaluated separately.

Projects

CNCF-hosted and Kubernetes ecosystem projects

• Kubernetes: Provides the shared control plane, Pod orchestration, Deployments, scheduling foundation, and common resource interfaces for both workload paths.
• Kueue: Manages training queues, quotas, and workload admission so distributed jobs do not partially start and hold capacity while waiting for the remaining resources.
• KEDA: Converts service and demand signals into scaling decisions for online inference workloads.
• Prometheus: Collects QPS, request-queue depth, inference latency, and other operational metrics used for observability and demand-driven scaling.
• HAMi: Pools heterogeneous accelerator resources and allocates accelerator capacity at fine granularity for workloads that can safely share it.
• Fluid: Provides Kubernetes-native access to datasets, model weights, and checkpoints, and caches model data near compute to reduce repeated remote reads and model-loading delays.

Other projects and technologies

• Ray: Provides the distributed runtime used by both the training and inference paths.
• Twinkle: Runs multi-tenant LoRA training on Ray, enabling tenants to share one base-model instance while retaining separate adapters, configurations, and state.
• vLLM / SGLang: Serve online inference workloads on Ray behind the Inference Gateway.

The value of the architecture comes from the interaction of admission, scheduling, accelerator allocation, data access, observability, elasticity, and model-runtime reuse rather than from any single component.

Planned Evolution

• Dynamic multi-tenant concurrency: Adjust Twinkle tenant capacity according to model size, LoRA configuration, memory pressure, and service objectives instead of fixing concurrency at five tenants.
• Unit-cost-based capacity management: Combine accelerator utilization, queue pressure, latency, and token cost when assigning capacity to training and inference.
• Broader hardware and runtime support: Integrate additional heterogeneous accelerators and training and inference backends, and contribute production experience upstream.

[jrsapi]

What I really liked about this reference architecture is that it explains why training and inference share infrastructure but do not use the same operating policy. The separation between admission, placement, accelerator allocation, and data readiness is also useful. Making model readiness part of elasticity is a particularly valuable lesson for readers trying to run inference at scale.
What would make this more useful to an end-user audience is a little more clarity on what is reusable, what CMB built, and the operating context behind the results:

• Could you distinguish the open source components from the CMB-built pieces in the diagram and narrative? For example, is the Training API an internal service, and what responsibility does it own relative to Twinkle, Kueue, and the other components?
• Alongside the percentage improvements, could you add enough operating context for readers to understand where the results apply: the deployment scale, representative model sizes, the inference SLO being protected, and an example of replica readiness with and without Fluid caching?
• Could you add a short section on what still needs improvement? The strongest reference architectures show the tradeoffs, not just the results. You point to dynamic tenant concurrency, broader hardware support, and unit-cost-based capacity management as future work. What is not working well enough today, what is the current workaround, and what would need to change?
• Were there other approaches you considered for this problem? If so, it would be helpful to include one example in the key design decisions and tradeoffs section, along with why it did not fit your operating conditions.
  These additions would help readers understand both the architecture and what it would take to apply the lessons in their own environment.

[tpx818]

What I really liked about this reference architecture is that it explains why training and inference share infrastructure but do not use the same operating policy. The separation between admission, placement, accelerator allocation, and data readiness is also useful. Making model readiness part of elasticity is a particularly valuable lesson for readers trying to run inference at scale. What would make this more useful to an end-user audience is a little more clarity on what is reusable, what CMB built, and the operating context behind the results:

• Could you distinguish the open source components from the CMB-built pieces in the diagram and narrative? For example, is the Training API an internal service, and what responsibility does it own relative to Twinkle, Kueue, and the other components?
• Alongside the percentage improvements, could you add enough operating context for readers to understand where the results apply: the deployment scale, representative model sizes, the inference SLO being protected, and an example of replica readiness with and without Fluid caching?
• Could you add a short section on what still needs improvement? The strongest reference architectures show the tradeoffs, not just the results. You point to dynamic tenant concurrency, broader hardware support, and unit-cost-based capacity management as future work. What is not working well enough today, what is the current workaround, and what would need to change?
• Were there other approaches you considered for this problem? If so, it would be helpful to include one example in the key design decisions and tradeoffs section, along with why it did not fit your operating conditions.
  These additions would help readers understand both the architecture and what it would take to apply the lessons in their own environment.

Thank you for the detailed and constructive suggestions. We agree that these additions would help readers better understand both the architecture and the conditions under which its lessons can be applied. We plan to address the four points as follows:

1. Open source and CMB-built components
   Yes, we can update both the architecture diagram and the narrative to distinguish open source components from CMB-built components and integration services. We will also clarify whether the Training API is an internal service and describe its responsibilities and boundaries relative to Twinkle, Kueue, Kubernetes Scheduler, HAMi, Fluid, and the other components.
2. Operating context for the reported improvements
   We can provide additional operating context for the percentage improvements Some detailed production information may not be shareable because of the bank’s internal security and disclosure policies, but we will provide as much anonymized or aggregated data as those policies allow.
3. Current limitations and areas for improvement
   Yes, we can add a short section describing the current limitations, operational workarounds, and planned improvements. In particular, we will expand on dynamic tenant concurrency, broader hardware and runtime support, and unit-cost-based capacity management by explaining what is not yet working as well as we would like, how these limitations are handled today, and what technical or operational changes would be required to reach the intended target state.
4. Alternative approaches considered
   We also evaluated solutions from several other vendors. In comparison, those solutions were not able to adapt quickly enough to the rapid changes in the large-model ecosystem or fully support the pace of iteration required in our internal environment. We therefore did not proceed with them.
   However, we are not sure whether a vendor-oriented comparison would be appropriate for this reference architecture, particularly if it cannot include sufficiently detailed or attributable information. One possible approach would be to describe the trade-off in general terms—for example, comparing a tightly integrated vendor solution with the more modular, Kubernetes-native approach we selected—without naming specific vendors. We would appreciate your opinion on whether such a generalized comparison would be useful in the “Key design decisions and trade-offs” section.

Thank you again for the feedback. We will incorporate the first three additions and, based on your guidance, determine the most appropriate way to present the alternative approaches considered.