Repository navigation
PERF: Keep the sensitivity work on the GPU - #38
Merged
Merged
Conversation
In the 4-rank MI300A run at 512^3, after the solves got faster, the element-wise sensitivity work had become the largest item after them: "hv sensitivity" alone 207 s of 842 (1.7 s per load case and product), plus the objective's "sensitivity" and the product's "(other)". It ran in NumPy on the host: per load case, four device-to-host copies of the kernel fields and several full-size host temporaries per accumulation. The element density, the SIMP derivatives, the gradient and Hessian-product accumulators and the per-case Hessian cache now live in the fields' array module (cupy on a GPU) in double precision. Only the design comes in and the gradient or product goes out through the host. set_density accepts device arrays, NodalElementMap.gather_mean can return one, and scatter_mean takes one. The regularization stays on the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Once the solves got faster (muGrid #251/#252, #37), the element-wise sensitivity work became the largest item after them. In the 4-rank MI300A run at 512³ (optimization 842 s):
hv sensitivityhessian_vector_product (other)sensitivity(objective)All of it ran in NumPy on the host. Per load case, that meant four device-to-host copies of the sensitivity fields, and several full-size host temporaries per accumulation, on ~33 M elements per rank.
What changes
These now live in the fields' array module (CuPy on a GPU), in double precision:
g_shear,g_vol,rho_e).Only the design comes in through the host, and the gradient or Hessian product goes out.
Homogenization.xp: the fields' array module.set_densityaccepts device arrays and interpolates on the device.NodalElementMap.gather_mean(..., on_device=True)returns the element average on the device, andscatter_meanaccepts device arrays.Memory: the per-case Hessian cache (2 fields per load case) moves from host to device memory. That's the same physical memory on an APU, and on a discrete GPU it's 12 extra scalar fields.
Tests
np.asarrayon device arrays; the material derivatives use NumPy ufuncs, which dispatch to CuPy; the nodal-map weights are NumPy scalars times device arrays, as before.hv sensitivityto drop from ~1.7 s to tens of ms per load case and product, andhessian_vector_product (other)to shrink as well.🤖 Generated with Claude Code