Skip to content

Fix nvidia fallback services failing to start #475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vigh-m wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

Fix nvidia fallback services failing to start #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8997da6
kmod-nvidia: add %build section
vigh-m Jun 26, 2026
137ebf4
kmod-nvidia: fix devid for g4dn instances
vigh-m Jun 26, 2026
97a504d
kmod-nvidia: fix *-fallback.service to install on all variants
vigh-m Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions packages/kmod-6.1-nvidia-r580/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service

@arnaldo2792 arnaldo2792 Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the commit message, or even in this file, explain that this unit doesn't need an Install section because the service will be started by a timer.

But also, you must guarantee that the service will start as soon as we reach multi-user.target. If you validated this, call it out in the PR description. And in the commit message, call out why the wanted ordering is preserved (e.g. timers.target run before, in parallel, after multi-user.target whichever is the case we want).

@vigh-m vigh-m Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the commit message with more info about this.
We don't need to wait for multiuser.target for this service. It was already setup to run after nvidia-gridd.service which is installed in preconfigured.target. This change shouldn't affect the ordering or install since timers.target was always going to install this service.

4 changes: 3 additions & 1 deletion packages/kmod-6.1-nvidia-r580/kmod-6.1-nvidia-r580.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ tar -xf %{_cross_datadir}/bottlerocket/kernel-devel.tar.xz
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

@arnaldo2792 arnaldo2792 Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two nits:

  • I would have kept this commit out of the series, because you are fixing three different problems
  • You could have used kmod-nvidia- in the commit message

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -256,7 +258,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"

@arnaldo2792 arnaldo2792 Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In your PR description, can you add the testing you did here to validate that g4dn.* are using the correct driver. And add in the commit message that we are following the steps of the EKS Optimized AMI.

and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.1-nvidia-r580/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

@arnaldo2792 arnaldo2792 Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit message is not well formatted and confusing. Could you please expand on how your change fixes the problem with non-k8s variants?

Also, should the service run After=nvidia-gridd.service? WantedBy doesn't guarantee ordering.

@vigh-m vigh-m Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the commit message with more details.

The services can start at the same time as nvidia-gridd.service. They no-op if they are on an unsupported instance and will exit without failing the boot.


[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.1-nvidia-r580/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
3 changes: 0 additions & 3 deletions packages/kmod-6.1-nvidia-r595/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service
4 changes: 3 additions & 1 deletion packages/kmod-6.1-nvidia-r595/kmod-6.1-nvidia-r595.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ tar -xf %{_cross_datadir}/bottlerocket/kernel-devel.tar.xz
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -256,7 +258,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"
and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.1-nvidia-r595/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.1-nvidia-r595/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
3 changes: 0 additions & 3 deletions packages/kmod-6.12-nvidia-r580/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service
4 changes: 3 additions & 1 deletion packages/kmod-6.12-nvidia-r580/kmod-6.12-nvidia-r580.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,8 @@ rpm2cpio %{_sourcedir}/nvidia-imex-%{tesla_ver}-1.amzn2023.%{_cross_arch}.rpm |
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -284,7 +286,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"
and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.12-nvidia-r580/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.12-nvidia-r580/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
3 changes: 0 additions & 3 deletions packages/kmod-6.12-nvidia-r595/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service
4 changes: 3 additions & 1 deletion packages/kmod-6.12-nvidia-r595/kmod-6.12-nvidia-r595.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,8 @@ rpm2cpio %{_sourcedir}/nvidia-imex-%{tesla_ver}-1.amzn2023.%{_cross_arch}.rpm |
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -273,7 +275,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"
and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.12-nvidia-r595/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.12-nvidia-r595/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
3 changes: 0 additions & 3 deletions packages/kmod-6.18-nvidia-r580/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service
4 changes: 3 additions & 1 deletion packages/kmod-6.18-nvidia-r580/kmod-6.18-nvidia-r580.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,8 @@ rpm2cpio %{_sourcedir}/nvidia-imex-%{tesla_ver}-1.amzn2023.%{_cross_arch}.rpm |
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -284,7 +286,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"
and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.18-nvidia-r580/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.18-nvidia-r580/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
3 changes: 0 additions & 3 deletions packages/kmod-6.18-nvidia-r595/grid-license-check.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,3 @@ ExecStart=/usr/bin/grep -q "License Status.*: Licensed" /tmp/.nvidia-gridd-licen
ExecStart=/usr/bin/touch /etc/drivers/.grid-licensed
ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
StandardOutput=append:/tmp/.nvidia-gridd-license

[Install]
WantedBy=nvidia-k8s-device-plugin.service
4 changes: 3 additions & 1 deletion packages/kmod-6.18-nvidia-r595/kmod-6.18-nvidia-r595.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,8 @@ rpm2cpio %{_sourcedir}/nvidia-imex-%{tesla_ver}-1.amzn2023.%{_cross_arch}.rpm |
# This recipe was based in the NVIDIA yum/dnf specs:
# https://github.com/NVIDIA/yum-packaging-precompiled-kmod

%build

# Begin open driver build
pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/kernel-open

Expand Down Expand Up @@ -273,7 +275,7 @@ pushd NVIDIA-Linux-%{_cross_arch}-%{tesla_ver}/supported-gpus
jq -r '.chips[] | select(.features[] | contains("kernelopen")) |
select(.devid != "0x1DB1"
and .devid != "0x1DB5"
and .devid != "0x1DEB8"
and .devid != "0x1EB8"
and .devid != "0x1EB4"
and .devid != "0x2237")' supported-gpus.json | jq -s '{"open-gpu": .}' > open-gpu-supported-devices.json
# confirm "NVIDIA H100" is in the resulting file to catch shape changes
Expand Down
2 changes: 1 addition & 1 deletion packages/kmod-6.18-nvidia-r595/open-gpu-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
2 changes: 1 addition & 1 deletion packages/kmod-6.18-nvidia-r595/tesla-license-fallback.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ ExecStart=/usr/bin/systemctl stop grid-license-check.timer --no-block
RemainAfterExit=true

[Install]
WantedBy=nvidia-k8s-device-plugin.service
WantedBy=nvidia-gridd.service
Loading