Skip to content

PERF: Keep the sensitivity work on the GPU - #38

Merged
pastewka merged 1 commit into
mainfrom
perf/sensitivity-on-device
Oct 30, 2026
Merged

pastewka merged 1 commit into
mainfrom
perf/sensitivity-on-device

Conversation

@pastewka

Copy link
Copy Markdown
Contributor

Why

Once the solves got faster (muGrid #251/#252, #37), the element-wise sensitivity work became the largest item after them. In the 4-rank MI300A run at 512³ (optimization 842 s):

time
hv sensitivity 207 s 1.7 s per load case and Hessian product
hessian_vector_product (other) 56 s nodal gather/scatter, material derivatives
sensitivity (objective) 20 s

All of it ran in NumPy on the host. Per load case, that meant four device-to-host copies of the sensitivity fields, and several full-size host temporaries per accumulation, on ~33 M elements per rank.

What changes

These now live in the fields' array module (CuPy on a GPU), in double precision:

  • the element density;
  • the SIMP derivatives;
  • the gradient and Hessian-product accumulators;
  • the per-case Hessian cache (g_shear, g_vol, rho_e).

Only the design comes in through the host, and the gradient or Hessian product goes out.

  • Homogenization.xp: the fields' array module. set_density accepts device arrays and interpolates on the device.
  • NodalElementMap.gather_mean(..., on_device=True) returns the element average on the device, and scatter_mean accepts device arrays.
  • The accumulations are now float64 also for single-precision fields; for the nodal design they used to follow the field precision.
  • The regularization (~2% of the run) stays on the host.

Memory: the per-case Hessian cache (2 fields per load case) moves from host to device memory. That's the same physical memory on an APU, and on a discrete GPU it's 12 extra scalar fields.

Tests

  • The full suite passes on the CPU (249), including the finite-difference checks of gradient and Hessian. There the array module is NumPy, so this checks the arithmetic but not CuPy-specific behaviour, and muTopOpt has no GPU CI.
  • I went through the GPU path by hand: no np.asarray on device arrays; the material derivatives use NumPy ufuncs, which dispatch to CuPy; the nodal-map weights are NumPy scalars times device arrays, as before.
  • Please check on MI300A: a timing run like the last one. I expect hv sensitivity to drop from ~1.7 s to tens of ms per load case and product, and hessian_vector_product (other) to shrink as well.

🤖 Generated with Claude Code

In the 4-rank MI300A run at 512^3, after the solves got faster, the
element-wise sensitivity work had become the largest item after them:
"hv sensitivity" alone 207 s of 842 (1.7 s per load case and product), plus
the objective's "sensitivity" and the product's "(other)". It ran in NumPy
on the host: per load case, four device-to-host copies of the kernel fields
and several full-size host temporaries per accumulation.

The element density, the SIMP derivatives, the gradient and Hessian-product
accumulators and the per-case Hessian cache now live in the fields' array
module (cupy on a GPU) in double precision. Only the design comes in and the
gradient or product goes out through the host. set_density accepts device
arrays, NodalElementMap.gather_mean can return one, and scatter_mean takes
one. The regularization stays on the host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pastewka
pastewka merged commit d56cc61 into main Oct 30, 2026
8 checks passed
@pastewka
pastewka deleted the perf/sensitivity-on-device branch October 30, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant