Skip to content

DRA claim handling - #806

Draft
bart0sh wants to merge 3 commits into
stacked/bart0sh/DRA/001-policy-boundaryfrom
stacked/bart0sh/DRA/002-claim-handling
Draft

bart0sh wants to merge 3 commits into
stacked/bart0sh/DRA/001-policy-boundaryfrom
stacked/bart0sh/DRA/002-claim-handling

Conversation

@bart0sh

@bart0sh bart0sh commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

DRA claim handling

Prepare and unprepare claims, and grant them with CDI

Third PR of the DRA series, on top of #803. It fills in the two kubelet handlers
the skeleton left as no-ops: a claim is allocated by the active policy through
the interface #803 added, and what it got is granted to the containers using it
by a CDI spec of its own.

Still inert on a cluster — every backend answers AllocateClaim with
policy.ErrNoDRAClaims, so a claim naming this driver fails and the pod stays
pending. The next PR brings the first policy that really allocates, and with it
the first look at this path outside its tests.

Three commits, in order: the library bump to Kubernetes v0.37 and the one
source change it forces; the CDI spec store — file mechanics and device
naming, where the device for result i of claim uid is claim-<uid>-<i>, so
the positional correspondence #803 defined survives into the CDI IDs; then
claim dispatch.

The ordering is the design

Prepare is: allocate through the policy, commit, write the spec. Unprepare
is: release, commit, remove the spec. Committing is a cache save plus the NRI
container updates the policy made, which is what lets an allocation outlive us
and reach the containers that give resources up for it.

Spec last is what keeps the accounting honest across a crash: the worst case is a
spec no claim accounts for, never resources granted to a container and accounted
to nobody. Two things follow from it.

A spec proves a prepare completed, so a failure partway through a
re-prepare must not roll back: AllocateClaim is idempotent and ReleaseClaim
releases the whole claim, so releasing here would take resources from a pod
already using them. prepareClaim rolls back only while the claim has no spec.

Claims are refused until the runtime synchronization has completed

stub.Start() returns once the runtime has answered Configure; the containers
already running only reach the policy with the Synchronize that follows — the
one NRI handler that runs without the resource manager's lock and with cache
saves blocked. A claim served in that window would be allocated beside
policy.Sync and would report a commit that never happened. Taking the lock in
Synchronize is not the fix, since committing a claim calls out to the runtime
with that lock held. Instead the plugin refuses claims until Synchronize calls
AllowClaims(), and the kubelet retries. The flag is never cleared: losing the
runtime connection exits the process.

Those refusals are per request, not per claim — nothing is wrong with the
individual claims, and a request with no answer is what the kubelet retries.
Everything else is answered per claim, because the kubelet records the answers
per claim and retries only the ones that failed.

The library bump

v0.37 is what Device.ShareID and
DeviceRequestAllocationResult.ConsumedCapacity need, the two fields a claim
against a device with consumable capacity is answered with. The plugin interface
grew WatchHealthStatus, which we decline: our devices are the node's own CPUs
and memory. Importing CDI is also the first thing in the tree to pull in
runtime-tools, whose replace pin — inherited, with no importer ever since —
does not compile against the runtime-spec already here, so it goes.

Testing

go test ./... and go test -race ./pkg/resmgr/... pass, and each commit builds
and tests green on its own with go.mod tidy. The spec store and both handlers
are covered: the devices a claim is answered with, other drivers' results left
alone, per-claim isolation, rollback at each failure point, a re-prepare and a
failed re-prepare, both commit-failure paths, and the gate in both directions
including the Synchronize line that opens it. Every assertion written for the
ordering and the gates was checked against the mutation it exists to catch.

v0.37 grew the kubeletplugin.DRAPlugin interface with WatchHealthStatus,
which we decline: our devices are the node's own CPUs and memory, and
the kubelet already knows whether the node is healthy. Declining is what
makes it stop asking.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
A claim's devices are granted to the containers using it by a CDI spec of
its own, named after the claim so that unpreparing it can find the spec
again. What a device grants is the policy's business: the container edits
it hands us are written out verbatim, and the spec version follows from
them, so a policy reaching for a newer kind of edit needs nothing here.

Importing the CDI library is the first thing in the tree to pull in
runtime-tools, whose stale replace does not compile against the
runtime-spec we already have. It has had no importer since it was
inherited, so it goes.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
A claim is allocated by the policy, committed, and only then granted by a
CDI spec. That order is what keeps the accounting honest: a crash between
the two leaves a spec nothing claims, which can be found and cleaned up,
rather than resources granted to a container and accounted to nobody.
Releasing follows the same order, so what may be left behind is always a
spec and never an allocation.

Every claim is answered on its own, because the kubelet records the
answers per claim and retries only the ones which failed. A claim the
policy allocated but we could not prepare is handed back, unless an
earlier prepare of it had already succeeded: its devices may be in use,
and the kubelet releases the claim when its last pod goes away.

Claims are refused until the runtime synchronization has completed. The
driver registers as soon as we start, but Synchronize is the one NRI
request which runs without the resource manager's lock and with cache
saves blocked, so a claim served beside it would be allocated
concurrently with the policy sync and committed to nothing.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
@bart0sh
bart0sh added this pull request to stack #807 September 21, 2026 15:59
@bart0sh bart0sh changed the title stacked/bart0sh/DRA/002 claim handling DRA claim handling Sep 21, 2026
@bart0sh
bart0sh requested a balanced review from Copilot September 21, 2026 16:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Claim synchronization can race, failed NRI updates are not safely retryable, and required orphan-spec cleanup is missing.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 High severity · 1 Medium severity

Open (3)
What changed in this PR

Adds DRA claim preparation and release through policies, with CDI-based resource grants and synchronization gating.

Changes:

  • Implements per-claim allocation, rollback, commit, and release handling.
  • Adds CDI spec storage and comprehensive tests.
  • Upgrades Kubernetes dependencies to v0.37.
File Description
pkg/​resmgr/​resource-manager.go Starts NRI before DRA.
pkg/​resmgr/​nri.go Enables claims after synchronization.
pkg/​resmgr/​dra/​plugin.go Adds CDI storage, gating, and health handling.
pkg/​resmgr/​dra/​plugin_test.go Extends plugin test infrastructure.
pkg/​resmgr/​dra/​claim.go Implements claim preparation and release.
pkg/​resmgr/​dra/​claim_test.go Tests claim lifecycle and failures.
pkg/​resmgr/​dra/​cdi.go Implements per-claim CDI specs.
pkg/​resmgr/​dra/​cdi_test.go Tests CDI storage behavior.
pkg/​resmgr/​dra.go Commits policy and container updates.
pkg/​resmgr/​dra_test.go Tests synchronization gating.
go.mod Updates Kubernetes and CDI dependencies.
go.sum Refreshes dependency checksums.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/resmgr/dra.go
Comment thread pkg/resmgr/nri.go
Comment on lines +341 to 343
m.dra.AllowClaims()

return p.getPendingUpdates(nil), nil
Comment thread pkg/resmgr/dra/plugin.go
Comment on lines +129 to +132
store, err := newCDIStore(driverName, opts.CDIDir)
if err != nil {
return nil, err
}
@klihub
klihub requested review from askervin, kad and klihub September 21, 2026 16:51
@bart0sh
bart0sh marked this pull request as ready for review September 21, 2026 18:17
@bart0sh
bart0sh marked this pull request as draft September 21, 2026 18:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants