diff --git a/docs/own-net-auditor.md b/docs/own-net-auditor.md index bd8dbc6..05ccd2b 100644 --- a/docs/own-net-auditor.md +++ b/docs/own-net-auditor.md @@ -243,6 +243,22 @@ dotnet-gcdump) гоняет сценарий N× на стенде и выдаё Часть уже покрыта own-check (subscription/region-escape) и WpfAnalyzers (3 902 wpf-freezable) — пак расширяет, не дублирует. +**Карта покрытия — `docs/wpf-audit-coverage.md`:** каждый смелл × техника детекта +(static-C# / static-XAML / runtime-heap / runtime-trace) × статус (have / small-rule / +gap) × маппинг на Avalonia-оракул. Два недоспеценных слоя зафиксированы там же: +**XAML-анализатор** (binding-path, virtualization-off, nested ScrollViewer, heavy +DataTemplate, ElementName/RelativeSource) и **runtime-trace коллектор** (binding-error +log, notification storms). Freezable — чисто-WPF, на оракуле не воспроизводится. + +**Data-duplication / строки — `docs/string-canonicalization.md`:** дублированный +`System.String` как доминирующий случай оси data-duplication, под net472. Тот же каркас, +что и везде: **анализатор** (heap-отчёт wasted bytes + Roslyn birth-sites — кладётся в +фазу 5) → **точечный мутатор** (AdaptiveStringCanonicalizer + Cecil/ReJIT/Harmony только +на approved hot sites, только новые строки) → пере-замер. Переписывание всего heap по +ссылкам **отвергнуто** (ломает `ReferenceEquals`/`lock(string)`/identity-кэши/unsafe — +ровно поэтому GC-string-dedup так и не въехал в dotnet/runtime). Анализатор — +framework-агностичен, тестируется на оракуле здесь. + ## 5. Что брать готовое | Слой | Инструмент | diff --git a/docs/string-canonicalization.md b/docs/string-canonicalization.md new file mode 100644 index 0000000..72b9655 --- /dev/null +++ b/docs/string-canonicalization.md @@ -0,0 +1,108 @@ +# String canonicalization under .NET 4.7.2 — design note + +The data-duplication axis (`docs/wpf-audit-coverage.md`), taken to its most valuable special +case: **duplicated `System.String`**. In a long-running WPF/DevExpress LOB app, bounded-vocabulary +strings — status codes, column names, enum-ish labels, dictionary values rehydrated per row/DTO — +get allocated thousands of times. Same bytes, different references, all retained. This note +captures the whole "CLR string surgery" branch so nothing is lost, and frames it in the project's +own terms: **analyzer first, targeted mutator second, never a heap rewrite.** + +This is the same methodology as the rest of Own.NET — *measure → confirm → targeted fix → re-measure* +(the phase-2 ratchet + phase-5 runtime-confirm loop), just pointed at strings. The mutator half is +a fix-arm-style tiered intervention, kept strictly opt-in. + +--- + +## Phase A — Analyzer (folds into Own.NET, build-free to consume) + +Objective measurements before anyone touches a pointer. Two sources: + +**heap (primary).** ClrMD / dotnet-gcdump heap walk after a scenario → enumerate every +`System.String`, group by content, and report per value: +- `count` — live instances, +- `total_bytes` — Σ object size, +- `wasted_bytes` — `(count − 1) × size` if canonicalized to one, +- `retention` — the dominators / GC-root paths keeping them alive, +- `long_lived_only` — filter to gen2 / pinned, ignore transients. + +This is the phase-5 collector extended with a string lens — it emits the same kind of artifact +(`strings.json`, shape mirrors `runtime.json`'s `retained[]`) that the Python side correlates and +the dashboard renders. Top-N by `wasted_bytes` is the actionable output. + +**C# (birth sites).** A Roslyn pass that flags *where* repeated long-lived strings are born — +`StringBuilder.ToString()`, `string.Concat`, DTO ctors/factories, parsers/deserialization, +setters/hydration code — so the report points at call sites, not just values. + +The analyzer alone is the 80% win: it replaces intuition with **top-20 values by wasted bytes + +their birth sites + retention roots**, which is exactly the report you want before deciding whether +canonicalization is even worth it. + +## Phase B — Mutator (separate, gated, net472-specific track) + +Only for **approved hot sites**, **bounded-vocabulary** values, **new strings only**. Never the +existing heap. The menu, cheapest→heaviest: + +1. **`AdaptiveStringCanonicalizer`** (the policy core, not a mechanism) — *not* `String.Intern`. + Own dictionary with evictable / weak semantics where possible, a `count` threshold and an + `estimated-saved-bytes` threshold, metrics built in. `String.Intern` is rejected as the + default: its pool is process-lifetime and never collected, so it trades duplication for an + unreclaimable leak. +2. **Build-time IL weaving** (Mono.Cecil / Fody) — insert `value = Canonicalizer.Canonicalize(value)` + after the approved string-producing calls. Pros: no GC, no object-graph rewrite, trivially + revertible, works on Framework. The pragmatic default for the mutator. +3. **CLR Profiling API** (`ICorProfilerCallback`) — the grown-up runtime path for net472, in two + modes: + - *observe*: `ObjectAllocated` for allocation stats, `ObjectReferences`/`RootReferences2` to + build the retention graph (overlaps Phase A's heap source), + - *scalpel*: `SetILFunctionBody` + `RequestReJIT` (ReJIT is available on 4.5+) to rewrite + specific methods so new strings route through the canonicalizer. + Most powerful, heaviest to operate. +4. **Harmony** (`Lib.Harmony`) prefix/postfix/transpiler on managed string factories — dirty but + fast for a one-evening PoC of a single pipeline. Coverage is poor and it's easy to introduce + behavioural schizophrenia; not for broad surgery. + +## Explicitly rejected — whole-heap reference rewrite + +The seductive version ("build the object graph, pick a canonical instance per value, mass-rebind +every reference, sync with the GC") is **out of scope, on purpose.** It breaks the world: +- `ReferenceEquals(a, b)` and any identity-keyed cache (`ConditionalWeakTable`, identity hash); +- `lock(someString)` (monitor uses object identity / sync block); +- `unsafe` code that mutates or pins strings on the assumption they're private; +- compatibility becomes sticky and miserable. + +This is precisely why the runtime's own GC string-dedup design (dedup thread, old generations, +weak handles, CAS on reference swap, no stack refs) **never shipped** in dotnet/runtime — it works +in environments where strings don't expose identity the way .NET's do. For net472 this is GC/runtime +territory, not a library, and "truly patch the CLR without it later being agony in prod" is the +opening line of a bad on-call legend. We don't go there. + +## Staged PoC plan + +1. Small profiler/ClrMD report of duplicated strings (Phase A heap source). +2. Top-20 values by wasted bytes + their top call sites / retention roots (Phase A C# source). +3. Proof-of-concept `AdaptiveStringCanonicalizer` with metrics. +4. IL weaving (Cecil) **or** Harmony at 2–3 approved sites only. +5. Re-measure — wasted bytes before/after. Feedback loop, not a one-shot. + +That simulates runtime string dedupe exactly where it pays, without becoming a garbage collector +for an afternoon. + +## net472 specifics + +- **Profiling API**: `ICorProfilerCallback2/3`; enable via `COR_ENABLE_PROFILING` + profiler GUID. + Heavyweight, but the only first-class retention-graph source in-process. +- **ReJIT**: `RequestReJIT` / `SetILFunctionBody` are available since 4.5 — usable on 4.7.2. +- **Cecil/Fody**: build-time IL rewrite works on Framework assemblies; easiest to ship and revert. +- **ClrMD** (`Microsoft.Diagnostics.Runtime`): attach to a live process or open a dump, enumerate + the heap, group strings — the analyzer's backbone. +- **`String.Intern`**: exists, but its pool is never collected → memory can't be reclaimed; use the + adaptive canonicalizer instead. + +## Where this is testable *here* (Avalonia oracle) + +String duplication is pure CLR/heap, so the **entire Phase A analyzer is framework-agnostic and +testable on the Avalonia oracle** (or any .NET app we build in this container) — group strings, +report wasted bytes, find birth sites. The build-time Cecil weaving path is cross-platform too, so +even a slice of Phase B can be exercised here. The Profiling-API/ReJIT scalpel is the one piece that +stays a stand exercise. So this branch isn't STS-only — most of it validates on the oracle, same as +phases 3–5. diff --git a/docs/wpf-audit-coverage.md b/docs/wpf-audit-coverage.md new file mode 100644 index 0000000..40cace3 --- /dev/null +++ b/docs/wpf-audit-coverage.md @@ -0,0 +1,93 @@ +# WPF audit pack — coverage matrix (Own.NET Auditor §4) + +What the auditor can detect for a WPF/DevExpress LOB app like STS, *by detection technique*, and +how honestly each maps onto an **Avalonia** oracle (the cross-platform stand-in we can build and +run here — see `docs/own-net-auditor.md`). The point of this doc is to draw the line between +"we've got this", "small rule away", and "whole surface we haven't built yet" — so nothing on +the wishlist quietly falls through. + +## Detection techniques + +| Tag | Technique | Where it runs | +|---|---|---| +| **C#** | Roslyn syntax/semantic analyzer over `.cs` | stand (graph/analyzer extractor) | +| **XAML** | analyzer over `.xaml`/`.axaml` (it's just XML — parseable even in Python) | stand or CI | +| **heap** | ClrMD / dotnet-gcdump heap walk after a scenario (phase 5 `runtime.json`) | stand | +| **trace** | runtime listeners: WPF binding-error trace, Dispatcher/notification counters | stand | + +The governing principle is the phase-5 one, extended from leaks to perf/correctness: +**suspect statically, confirm at runtime.** A disabled-virtualization attribute is a *suspicion* +(XAML); 100k live containers for 100k rows is the *confirmation* (heap). + +## Matrix + +| Smell | Technique | Status | Avalonia | +|---|---|---|---| +| **Event leaks** | | | | +| `+=` without `-=` | C# + **heap** | ✅ core (own-check OWN001) + phase-5 confirm | ✅ 1:1 | +| static-event subscription from instance | C# + **heap** | ✅ + phase-5 confirm | ✅ 1:1 | +| `DependencyPropertyDescriptor.AddValueChanged` (roots in a static table forever) | C# | ✅ small rule | ❌ WPF-only (Avalonia: `GetObservable().Subscribe()` un-disposed — same category, diff API) | +| `CollectionChanged` / `PropertyChanged` subscription leak | C# + **heap** | ✅ | ✅ 1:1 | +| `CommandManager.RequerySuggested` static-event | C# | ✅ small rule | ❌ WPF-only | +| **Timers** | | | | +| `DispatcherTimer` / `Timers.Timer` / `Threading.Timer` no Stop/Dispose | C# + **heap** | ✅ small rule + confirm | ✅ 1:1 (`Avalonia.Threading.DispatcherTimer`) | +| **WPF binding** | | | | +| binding path doesn't exist | **trace** (reliable) / XAML (heuristic) | ⚠️ **no XAML analyzer, no trace collector** | ✅ Avalonia logs `LogArea.Binding` errors → trace maps | +| binding to an expensive property | XAML + heuristic | ⚠️ no XAML analyzer | ~ | +| binding errors via runtime log | **trace** | ⚠️ **no collector** (cheap + high value) | ✅ | +| `ElementName` / `RelativeSource` hell (complexity) | XAML | ⚠️ no XAML analyzer | ✅ (same XAML dialect) | +| **PropertyChanged hell** | | | | +| fat setter (one set → pile of `OnPropertyChanged`) | C# | ✅ Roslyn rule | ✅ 1:1 | +| cascading notifications | C# (call graph) + **trace** | ⚠️ partial; storms need trace | ✅ | +| duplicate / repeated notifications | **trace** | ⚠️ no collector | ✅ | +| UI-thread notification storms | **trace** (Dispatcher counters) | ⚠️ no collector | ✅ | +| **Virtualization** | | | | +| `IsVirtualizing=False` / `VirtualizationMode=Standard` (no recycling) | XAML + **heap** | ⚠️ XAML gap; **heap confirm falls out of phase 5** | ✅ (`VirtualizingStackPanel`/`ItemsRepeater`) | +| non-virtualizing `ItemsPanel` (StackPanel swap) | XAML + **heap** | ⚠️ XAML gap; heap confirm ✅ | ✅ | +| nested `ScrollViewer` / `CanContentScroll=False` kills virtualization | XAML | ⚠️ XAML gap | ✅ | +| heavy `DataTemplate` (element/nesting count) | XAML | ⚠️ XAML gap | ✅ | +| containers not recycled (live count ≫ visible rows) | **heap** | ✅ phase-5 native | ✅ 1:1 | +| DevExpress `GridControl` virtualization config | XAML (vendor) | ❌ vendor-only | ❌ no DevExpress | +| **Freezable** | | | | +| `Brush`/`Geometry`/`ImageSource` not `Freeze()`d | C#/XAML + **heap** | ⚠️ small rule | ❌ **Freezable is a WPF-only concept** | +| repeated immutable resources instead of static/shared | XAML + **heap** (dup count) | ⚠️ | ~ (heap-dup maps; Freeze doesn't) | +| **Data duplication** | | | | +| immutable reference data multiplied across VM/DTO | **heap** (N copies of a should-be-singleton) | ✅ phase-5 native | ✅ 1:1 | +| large collections copied needlessly | **heap** + C# (`.ToList()` in hot paths) | ⚠️ partial | ✅ | +| **duplicated strings** (the dominant case) | **heap** + C# birth-sites | → see [`string-canonicalization.md`](string-canonicalization.md) | ✅ 1:1 (pure CLR) | + +## The two surfaces we haven't built (the honest gaps) + +1. **XAML analyzer.** A large slice of the wishlist lives in `.xaml`, not `.cs`: binding paths, + `ElementName`/`RelativeSource` complexity, **virtualization-disabled patterns**, nested + `ScrollViewer`, heavy `DataTemplate`, Freezable-in-resources. We have a Roslyn (C#) extractor + but nothing reads XAML. Biggest gap — and technically cheap: XAML is XML, rules are tree + patterns. Avalonia `.axaml` is the same dialect, so it maps. +2. **Runtime-trace collector.** Beyond the heap snapshot there's a second runtime signal class — + the WPF binding-error trace (`System.Windows.Data Error: 40 …`) and UI-thread notification + counters. The heap collector doesn't see these; a small ETW/trace listener (or, on Avalonia, + a `Logger` sink) would. Cheap, high value for binding correctness + storms. + +One concept — **Freezable** — is WPF-only and won't exercise on the Avalonia oracle at all; that +tail is validated only on STS. + +## Avalonia mappability — summary + +What the oracle exercises splits by what's already built — to keep this honest against the gaps +above: + +- **Today** (framework-agnostic core, on techniques we have — Roslyn graph + CLR heap): all event + leaks, timers, fat-setter, virtualization *heap-confirmed* (container count ≫ visible rows), + data duplication, string duplication. +- **Once the two gap surfaces exist** (XAML analyzer + runtime-trace collector): heavy templates, + virtualization-off *patterns*, binding-path / `ElementName` complexity, and binding errors — + Avalonia's `.axaml` is the same XAML dialect and it logs binding errors too, so the oracle will + validate these the moment those surfaces land. (Planned, not current.) +- **Never** on the oracle (WPF/vendor-only tail, STS only): Freezable, DevExpress controls, + `CommandManager.RequerySuggested`, `DependencyPropertyDescriptor.AddValueChanged`. + +So ~70–80% of the surface is oracle-reachable in total; the "today" slice is the +framework-agnostic core, the rest unlocks with the XAML analyzer and the trace collector. + +Crucially, the framework-agnostic core — Roslyn graph, CLR heap — behaves identically, so phases +3–5 are validated on *real* data the moment we run them against a leaking Avalonia app.