feat(compute): Hyper-Connection forward pass (DSV4 Phase 2)

[ezutfen]

Summary

DeepSeek-V4-Flash Phase 2: Hyper-Connections (HC) — the structural backbone.

HC replaces standard residual connections with a 4-copy hidden state [seq, hc_mult, d] that flows through every layer. Each block has hc_pre (reduce hc_mult→1) before attention/FFN and hc_post (expand 1→hc_mult) after. The final layer uses hc_head (sigmoid-based, no Sinkhorn).

New module: forward/hyper_connection.rs

• hc_expand(h, hc_mult): Initialize HC dimension from [seq, d] → [seq, hc, d]
• hc_pre(x_hc, params, iters, eps): Reduce [seq, hc, d] → [seq, d]
  1. RMSNorm on flattened [seq, hc*d]
  2. Linear projection → mixes [seq, mix_hc] where mix_hc = (2+hc)*hc = 24
  3. Split into pre[hc], post[hc], comb[hc*hc]
  4. pre = sigmoid(mixes * scale[0] + base) + eps
  5. post = 2 * sigmoid(mixes * scale[1] + base)
  6. comb = softmax(mixes * scale[2]) → Sinkhorn-normalized (alternating row/col, 20 iters)
  7. y = sum(pre * x, dim=hc) — weighted reduction
• hc_post(x, residual, post, comb): Expand [seq, d] → [seq, hc, d]
  • y = post * x + sum(comb * residual, dim=hc) — weighted expansion + residual mixing
• hc_head(x_hc, params, eps): Final-layer reduction [seq, hc, d] → [seq, d]
  • Uses [1] scale (not [3]), sigmoid-based (no Sinkhorn)

HC parameter shapes (hc_mult=4, hidden=4096)

Parameter	Shape	Notes
hc_fn (fn_weight)	[24, 16384]	mix_hc × (hc × d)
hc_base	[24]	Per-mix-component bias
hc_scale	[3] (per-layer) or [1] (head)	pre/post/comb scales

Tests (22 total)

• Shape verification for all functions
• Finiteness checks (no NaN/Inf on realistic inputs)
• Sinkhorn doubly-stochastic property (rows and cols sum to ~1.0 after 50 iters)
• Round-trip: expand → hc_pre → hc_post shape consistency
• Zero-input produces zero-output (with eps-positive pre values)
• V4-Flash realistic dimensions (hidden=4096, hc_mult=4, seq=1 decode)
• Sigmoid/softmax/RMSNorm helper correctness

Test results (Valinor RTX 3060, Rust 1.97.0)

• 784 lib tests: all pass (22 new + 762 existing)
• cargo fmt + clippy: clean

Phase context

• Phase 0 (config + arch module): MERGED PR feat(models): DeepSeek-V4-Flash architecture module + config fields (DSV4 Phase 0) #65
• Phase 1 (FP8/FP4 dequant): MERGED PR feat(models): FP8 E4M3 block-scale dequant + shared expert FP4 support (DSV4 Phase 1) #66
• Phase 2 (this PR): HC forward pass
• Phase 3 (attention forward): next — depends on Phase 0+1+2