Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Amazon ECS GPU sharing

Amazon ECS can share one physical NVIDIA GPU across multiple containers with NVIDIA Multi-Process Service (MPS). ECS sets up MPS, places containers onto GPUs, and enforces per-container limits. To request it, add a `sharingStrategy` with an `nvidiaMps` object to the container's `GPU` resource requirement instead of a whole-GPU `value`. The object takes a GPU `memory` limit in MiB and an optional `maxComputePercent`. ECS places tasks by GPU memory only. GPU sharing runs on ECS Managed Instances and ECS on EC2. Read the documentation for the supported instance families and the AMI that ECS on EC2 needs.

Use GPU sharing for inference and smaller models that do not need a whole GPU. Use a whole GPU (`value`) when a container runs more than one CUDA process or when you need per-task GPU metrics. GPU sharing does not work on AWS Fargate, ECS Express Mode, ECS managed daemon tasks, or G6f fractional GPU instances. On G6f fractional GPU instances, request a fractional GPU `value` instead, and read the GPU task definition documentation for the supported values. Prefer GPU sharing to the older manual method, which removes the GPU resource requirements and makes the NVIDIA runtime the default for all containers on the instance.

## Security considerations

Containers that share a GPU also share its failure domain. Recommend these defaults and confirm the details in the documentation:

- A fatal GPU error in one container, such as an illegal memory access, restarts every container on that physical GPU. Give a workload that must not be affected by its neighbors a whole GPU with `value`.
- Set `memory` for each container. It is a hard ceiling, so one container cannot take GPU memory from the others. `maxComputePercent` is a ceiling, not a reservation, so it does not guarantee compute to a container.
- Do not use the manual sharing method on new workloads. It lets every container on the instance use the GPUs.

## Documentation

Before advising or acting, the agent MUST read and cite [Share GPUs on Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu-sharing.html) at response time and answer from its published content. For GPU task definitions in general, read [Amazon ECS task definitions for GPU workloads](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu.html). Do not copy validation rules, task definition examples, or the supported instance list into this reference.
1 change: 1 addition & 0 deletions plugins/aws-core/skills/aws-containers/references/ecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Load the appropriate reference file based on what the user is working on:
- **task definitions**, **task/execution IAM roles**, **secrets injection**, **volumes**, **managed daemons**, **daemon tasks**, or **agents on instances** -> see [ecs-workloads.md](ecs-workloads.md)
- **Action Logs**, **control-plane troubleshooting**, **deployment rollback / circuit-breaker**, **task placement failures**, **scaling / task replacement visibility**, or **enabling or reading ECS Action Logs** -> see [action-logs.md](action-logs.md)
- **ECS Gateway**, **service ingress**, **shared HTTP/HTTPS endpoints for services**, or **choosing between ECS Gateway, Express Mode, and a load balancer** -> see [ecs-gateway.md](ecs-gateway.md)
- **GPU sharing**, **NVIDIA MPS**, **sharing one GPU across containers or tasks**, or **`sharingStrategy` in a GPU resource requirement** -> see [ecs-gpu-sharing.md](ecs-gpu-sharing.md)

## Resources

Expand Down
17 changes: 17 additions & 0 deletions skills/core-skills/aws-containers/references/ecs-gpu-sharing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Amazon ECS GPU sharing

Amazon ECS can share one physical NVIDIA GPU across multiple containers with NVIDIA Multi-Process Service (MPS). ECS sets up MPS, places containers onto GPUs, and enforces per-container limits. To request it, add a `sharingStrategy` with an `nvidiaMps` object to the container's `GPU` resource requirement instead of a whole-GPU `value`. The object takes a GPU `memory` limit in MiB and an optional `maxComputePercent`. ECS places tasks by GPU memory only. GPU sharing runs on ECS Managed Instances and ECS on EC2. Read the documentation for the supported instance families and the AMI that ECS on EC2 needs.

Use GPU sharing for inference and smaller models that do not need a whole GPU. Use a whole GPU (`value`) when a container runs more than one CUDA process or when you need per-task GPU metrics. GPU sharing does not work on AWS Fargate, ECS Express Mode, ECS managed daemon tasks, or G6f fractional GPU instances. On G6f fractional GPU instances, request a fractional GPU `value` instead, and read the GPU task definition documentation for the supported values. Prefer GPU sharing to the older manual method, which removes the GPU resource requirements and makes the NVIDIA runtime the default for all containers on the instance.

## Security considerations

Containers that share a GPU also share its failure domain. Recommend these defaults and confirm the details in the documentation:

- A fatal GPU error in one container, such as an illegal memory access, restarts every container on that physical GPU. Give a workload that must not be affected by its neighbors a whole GPU with `value`.
- Set `memory` for each container. It is a hard ceiling, so one container cannot take GPU memory from the others. `maxComputePercent` is a ceiling, not a reservation, so it does not guarantee compute to a container.
- Do not use the manual sharing method on new workloads. It lets every container on the instance use the GPUs.

## Documentation

Before advising or acting, the agent MUST read and cite [Share GPUs on Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu-sharing.html) at response time and answer from its published content. For GPU task definitions in general, read [Amazon ECS task definitions for GPU workloads](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu.html). Do not copy validation rules, task definition examples, or the supported instance list into this reference.
1 change: 1 addition & 0 deletions skills/core-skills/aws-containers/references/ecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Load the appropriate reference file based on what the user is working on:
- **task definitions**, **task/execution IAM roles**, **secrets injection**, **volumes**, **managed daemons**, **daemon tasks**, or **agents on instances** -> see [ecs-workloads.md](ecs-workloads.md)
- **Action Logs**, **control-plane troubleshooting**, **deployment rollback / circuit-breaker**, **task placement failures**, **scaling / task replacement visibility**, or **enabling or reading ECS Action Logs** -> see [action-logs.md](action-logs.md)
- **ECS Gateway**, **service ingress**, **shared HTTP/HTTPS endpoints for services**, or **choosing between ECS Gateway, Express Mode, and a load balancer** -> see [ecs-gateway.md](ecs-gateway.md)
- **GPU sharing**, **NVIDIA MPS**, **sharing one GPU across containers or tasks**, or **`sharingStrategy` in a GPU resource requirement** -> see [ecs-gpu-sharing.md](ecs-gpu-sharing.md)

## Resources

Expand Down
Loading