Repository navigation
Introduce the concept of a 'near miss' intersection. - #2204
brownd1978 wants to merge 2 commits into
Conversation
…rajectory covariance in-bound fraction at the surface intersection. Add near-miss intersections to the KinKal payload. Refactor TrkCrvMatchToo to use 'near miss' instead of the by-hand edge testing. Coded by Claude.
|
Hi @brownd1978,
which require these tests: build. @Mu2e/fnalbuild-users, @Mu2e/write have access to CI actions on main. ⌛ The following tests have been triggered for c4c53db: build (Build queue - API unavailable) |
|
oksuzian
left a comment
There was a problem hiding this comment.
PR Review Summary — #2204
Reviewed at head 49929928b238762be6b4bc0b80a1a64a44753895. First pass.
Decision
- 🔴 request changes. Dave, the PR does not link with muse.
libmu2e_CRVReco.sonow calls KinKal trajectory code but does not linkKinKal_Trajectory(finding 1). Three added lines fix it, and with them the merged tree builds with no warnings and runs. - Taking KinKal#239 also drops calorimeter material from a few tracks, through
KKShellXing. This PR is the one that moves Offline to that KinKal, so this needs a fix, in KinKal or here, before it merges (finding 2). - The near-miss idea works. With both fractions at 0 the output is identical to
main. With the defaults, fits do not change; KalSeeds gain many out-of-bounds IPA and stopping-target intersections, and the LoopHelix fits cost about 15% more CPU (finding 4). The TrkCrvMatcher change loses some matches, most of all on reconstruction files made before this PR (finding 3). - FNALbuild cannot build this yet: the envset KinKal does not have KinKal#239. Merge order: KinKal#239, then a KinKal tag at
cb8107dcor later and an envset that carries it, then this PR. A tag atd426035cis not enough. This PR'sExtrapolateSweepTestchecks that a near miss does not split the trajectory, and that needs theMaterial::extrapolatechange incb8107dc. I read this from the code; I did not build this PR againstd426035c. - The head moved during my review, from
c4c53dbfto49929928("Merge main, resolve conflicts"). I had made the same merge myself, and the two trees are identical. Everything below was built and run on that tree.
Scope understood
- A "near miss" is an out-of-bounds crossing where at least a set fraction of the projected position covariance lies inside the surface (
inbfrac_from KinKal#239). The sweep accepts it (ExtrapolateSweep.hh:459, test at:348-360), andKKExtrapadds it as an inactive material crossing (KKExtrap.hh:475). It does not use up its target or group (ExtrapolateSweep.hh:189-195). NearMissFractionsets the fraction for IPA, stopping-target foils, DS shells, plane shells and calorimeter panels (KKExtrap.hh:197,:216,:261,:276,:309).CRVNearMissFractionsets it for CRV sectors (:290). Both default to 0 in C++ and are set to 0.1 and 1e-3 in all four extrapolation configurations inprolog.fcl:382-448.- TrkCrvMatcher drops its own projection of line fits with no CRV crossing. It uses the near-miss CRV intersections instead.
TrkCrvXinggainsfloat inbFrac_(TrkCrvMatch.hh:56), andinBounds_ == falsenow means a near miss, not a projection (:55). - The covariance helper
TrkCrvPlaneError.hhis deleted.TrkCrvGeom.hh:42-58now uses KinKal'splaneCovarianceand the intersection'spcov_. KKShellXing::updateReferenceasks for the projected covariance when the crossing has one (KKShellXing.hh:89).- Validation histograms are renamed from "Proj" to "Near".
Findings
- 🔴 [S0] The PR does not link with muse.
- Evidence:
muse buildagainst KinKalcb8107dcstops atlibmu2e_CRVReco.so:The link line hasCRVReco/inc/TrkCrvGeom.hh:44: undefined reference to `KinKal::KinematicLine::planeCovariance(double, KinKal::Plane const&, double) const' RecoDataProducts/inc/KalIntersection.hh:42: undefined reference to `KinKal::KinematicLine::KinematicLine(KinKal::ParticleStateEstimate const&, ...)'-lKinKal_Geometry -lKinKal_Generaland no-lKinKal_Trajectory. muse links shared libraries with-Wl,--no-undefined, so this is an error. Same failure atc4c53dbfand on the merged tree. - Cause:
TrkCrvGeom.hh:44callski.kinematicLine().planeCovariance(...).CRVReco/src/SConscript:15-38(library) and:40-(plugins) list onlyKinKal_GeometryandKinKal_General.TrkCrvMatchMCMaker_module.cc:463-464makes the same call, andCRVResponse/src/SConscript:33-45has the same list. - Fix: add
'KinKal_Trajectory',to both lists inCRVReco/src/SConscriptand to the plugin list inCRVResponse/src/SConscript. For the CMake build, addKinKal::Trajectorynext toKinKal::GeometryinCRVReco/CMakeLists.txt:15and:52andCRVResponse/CMakeLists.txt:158. With the three SConscript lines, the merged tree builds with no warnings. My patch:w_merge_cb8/local_link_patch.diff.
- Evidence:
- 🟠 [S1] With KinKal#239,
KKShellXing::updateReferenceswitches off calorimeter material in some downstream tracks.- Code: the piece search at
KKShellXing.hh:90can now return an out-of-bounds crossing, and:93,:96and:99accept it ononsurface_ && inrange_.updateStatethen needsgood()(:109), so the material is off. Before KinKal#239 that search returned nothing in the cases I reproduced with a toy, and the code went on to search the neighbouring piece. For the 5 tracks below I did not confirm the exact path. - Measured on Offline
mainwith KinKal3a6860b(this PR's KinKal base) againstcb8107dc, 2,000 OnSpill events withSaveMaterialXings : true. 6EMC_Disk_1_InnerRingcrossings in 5 tracks go out of bounds, and their energy loss (1.2 to 9.3 MeV/c each) is dropped. That is 4 of 1,991 KKDe tracks and 1 of 1,955 KKDmu tracks. Momentum atEMC_Disk_1_Backmoves by +1.5 to +10.5 MeV/c (event 3979: 86.93 to 97.42 MeV/c). The fits themselves do not change, because the ring is past the last hit. Details are in my KinKal#239 review. - I do not have a fix. Two local patches to
updateReferencedid not work. Refusing to replace an in-bounds crossing with an out-of-bounds one keeps stale crossings and changes 30 fits. Retrying the ±1 ns trajectory search and preferring an in-bounds result leaves the same 5 tracks without the ring. Details and patches are in my KinKal#239 review. - Any fix here must apply only to crossings that were in bounds, because near misses are out of bounds by design.
- Code: the piece search at
- 🟡 [S2] TrkCrvMatcher loses some geometric matches, and loses all of the old projections on existing reconstruction files.
-
Measured on 10,000 CosmicCRYExtracted events (6,730 KKLine tracks), TrkCrvMatcher plus TrkCrvMatchMC as in my job (
ProbeAllPulses : false,KalSeedCollection : "KKLine"):tracks with a geometric best ... that is the true cluster true clusters that are geometric candidates out-of-bounds crossings main1,744 1,735 2,719 65 (projections) this PR, defaults 1,718 1,711 2,701 2,346 (near misses) this PR, both fractions 0 1,681 1,675 2,667 0 this PR's matcher on a reco file made by main1,681 1,675 2,667 0 -
With the defaults, 26 tracks lose their geometric best and none gain one. 24 of the 26 were true matches. Most are in
CRV_EX: 49 true clusters were reached only through a projection inmain, and 32 are reached through a near miss here. The TrkCrvMatchMC summary gives an average chi2/ndof of 19.3 for the 49 projection matches inmain, and 5.3 for the 32 near-miss matches here. I did not check the chi2 of the 17 that are lost, so I cannot say how many would pass a chi2 cut. -
Old files: a reco file made before this PR has no near-miss intersections. Running this matcher on it gives the "fractions 0" row: all 65 projections are gone, and 63 tracks lose their geometric best. EventNtuple runs TrkCrvMatcher when it makes ntuples from existing reco files (EventNtuple
fcl/prolog.fcl:203-235). So ntuples from MDC2025 reco made with this Offline lose those matches. -
Please decide whether this is wanted, and say it in the PR description. If the old files matter, one option is to keep the projection for line fits whose KalSeed has no CRV intersection at all.
-
- 🟡 [S2] Default production reconstruction gains many out-of-bounds intersections in every LoopHelix KalSeed, and costs more CPU.
- Measured on 2,000 OnSpill events, merged tree with defaults against the same tree with both fractions 0. Fits are identical in every collection: status, active hits, chi2, t0, momentum. With both fractions 0 the output is identical to
mainbuilt against the same KinKal. - New out-of-bounds intersections (
inbounds_false,dMom0): KKDe +395 IPA and +8,233 stopping target, KKDmu +431 and +7,168, KKUe +745 and +5,552, KKUmu +652 and +6,280. For KKDe, the in-bounds stopping-target intersections are 21,725, so this adds 38%. Tracks with at least one out-of-bounds intersection go from 153 to 1,750 of 1,991 KKDe tracks. - Readers that do not check
inbounds_now count these. EventNtuple counts every IPA and stopping-target intersection innipaup,nipadown,nstupandnstdown(src/InfoStructHelper.cc:110-121), and writes each intersection as a track segment.rooutil/inc/common_cuts.hh:80-90(stopping_target_foils) selects segments by surface only. OfflineValidation/src/ValKalSeed.cc:187-194counts IPA and stopping-target intersections.SimpleKalSeedSelectortakes the momentum fromintersections().front()(TrkReco/src/SimpleKalSeedSelector_tool.cc:8), andKKFit::sampleFitputs IPA crossings first with no time sort (KKFit.hh:961,:1020). So a near miss can now be the intersection it reads. - CPU, same machine, the two jobs run side by side: the four LoopHelix fits go from 0.0905 to 0.1046 s/event (+15.6%; KKDe 0.0228 to 0.0264). The full event goes from 0.1548 to 0.1688 s (+9.0%). One run each. I did not split the cost. The new work is the covariance and bounds fraction for near-miss candidates and winners (
ExtrapolateSweep.hh:348-360,:507), and bounded cylinders are no longer cut by z inwithinReachwhen near misses are on (:179), as the comment there says. The KKLine fit in the cosmic job goes from 3.24 to 3.38 ms/event (+4.1%). - Output size:
mcs.art+10,524,722 bytes for 2,000 OnSpill events (+1.8%, 5.3 kB/event). Cosmic job:mcs.art+0.27%, TrkCrvMatch file +0.84%. - Please say whether near misses should be on by default in production. If they are, the EventNtuple counters and the segment tables need an
inboundscheck, or a flag for near misses.
- Measured on 2,000 OnSpill events, merged tree with defaults against the same tree with both fractions 0. Fits are identical in every collection: status, active hits, chi2, t0, momentum. With both fractions 0 the output is identical to
- 🟡 [S2]
TrkCrvXingreads back fine both ways, but old and new data mean different things.- New code reading a TrkCrvMatch file made by
main: no warnings, all fields match.inbFrac_reads as 0 for all 3,293 in-bounds crossings, while the comment atTrkCrvMatch.hh:56says it is 1 when not computed. The 65 old projections read as near misses (inBounds_false). - Old code reading a file made by this PR: no warnings, all fields match,
inbFrac_is ignored. The 2,346 near misses look like projections. EventNtuple documentsinboundsfalse as a projection (doc/branches.md:74,:79;utils/trkcrv_example.py:6-7,:84-86) and does not copyinbFrac_(inc/TrkCrvXingInfo.hh). - Suggested: a
ClassVersionbump with an I/O read rule that setsinbFrac_frominBounds_for old data, or change the comment to say old files give 0. An EventNtuple follow-up should update the docs and addinbfrac.
- New code reading a TrkCrvMatch file made by
- ⚪ [S3] One of 7,822 tracks changes its in-bounds stopping-target crossings. Event 107, KKDmu: foil 15 is no longer crossed and foil 23 is, and the momentum at the foils moves by about 0.2 MeV/c. A possible cause: a near miss counts as a crossing in the
KKExtraploop (KKExtrap.hh:417-418), and each call to KinKal'sextrapolatestarts a newMaxDtwindow and new field domains from the current end. I did not trace this event. - ⚪ [S3]
NearMissFractionandCRVNearMissFractioninLHSEEDXTRAP(prolog.fcl:413-414) do nothing: that configuration hasUpstream,ToCRV,ToOPAand the calorimeter targets off, so it has no target that uses them. - ⚪ [S3] Validation histograms are renamed (
NXingProjtoNXingNear,chi2ndofProj,XingProjShield,RecoXingProj,RecoXingProjTrue,TrueXingProj,XingDistProj). valCompare between a release before and after this PR will show these as missing. Nothing in Offline, Production or EventNtuple reads them by name.
Checked, no action needed
- 🟢 The merge conflict was mechanical. The only conflict was
CRVReco/inc/TrkCrvMatchTool.hh:maincommit4914d1de3(#2147) addedvoid checkResolutions() const;next toerrorModel, whose argument type this PR changes.49929928keeps both, as I did. Git auto-mergedprolog.fcl(#2152 addsTS5_FacetoSampleSurfaces),ExtrapolateSweep.hh(#2111 comment),KKExtrap.hh(#2111 range-out truncation, #2152) andTrkCrvMatchTool.cc(#2147). These do not interact with near misses: a near miss skipsrangesOut(KKExtrap.hh!near && rangesOut), andTS5_Faceis a sampled surface, not a sweep target. - 🟢 The covariance rotation in
trkCrvXingCovariance(TrkCrvGeom.hh:50-58) is right. Withrot(i,j) = axes[i]·idirs[j],Similarity(rot, pcov_)gives R C Rᵀ, the covariance in the sector axes, when the intersection plane is the sector plane. For CRV sectors the KinKal tangent plane is the rectangle itself, andTrkCrvGeom.cc:28-30takesu,v,wfrom the same rectangle. The fallback buildsPlane(g.w, g.u, point), whose V direction is w × u =g.v, so the sign of the U-V term is kept. It is the same straight-line projection as the deletedTrkCrvPlaneError.hh. Measured: with near misses off, all 1,681 tracks with a geometric best have the same sector and chi2 asmain. - 🟢
KKShellXingand near misses: a near miss has no material at construction (KKShellXing.hh:75needsgood()), becomes active only if a refit moves it in bounds (:109), and does not split the trajectory (KinKalcb8107dcMaterial.hh:106-108).truncateignores inactive effects, so near misses do not block the range-out truncation from #2111. In the OnSpill sample the in-bounds intersections are identical tomainin every track except event 107 (finding 6). - 🟢
ExtrapolateSweepTestpasses on the merged tree: "ExtrapolateSweepTest: all checks passed", 0.14 s, run against KinKalcb8107dc. It includes the new near-miss cases.
Validation check
- Build/tests run: yes, with local changes, described here.
- Merged tree (
49929928) andc4c53dbf, muse p111, against KinKalcb8107dcinstalled locally: link failure (finding 1). With my three SConscript lines: the merged tree builds with no warnings. - Offline
main0f6b4ad0against KinKal3a6860b,d426035candcb8107dc: all build with no warnings. ExtrapolateSweepTest: passes.- Jobs: production
recoMC/OnSpill.fclon 2,000 events ofdig.mu2e.CeMLeadingLogMix1BB.MDC2025au_best_v1_3.001430_00001612.art, andrecoMC/Extracted.fclplus TrkCrvMatcher, TrkCrvMatchMC and Validation on 10,000 events ofdig.mu2e.CosmicCRYExtracted.MDC2025bb_best_v1_5.001400_00017143.art. Inputs read in place from dCache. Production atf1624445, mu2e-trig-config at169df7db. All jobs exit 0 with no exceptions.
- Merged tree (
- CI: read at 2026-11-21 23:25Z. At
49929928:mu2e/buildtestpending, "This test has not been triggered yet". Atc4c53dbf: FNALbuild could not prepare the workspace because of the merge conflict. When it runs, it will fail to compile: the p111 envset KinKal hasKinematicLine::planeCovariance(double, Plane const&)with nodirtol, and noinbfrac_. - Sourced numbers:
NearMissFraction : 0.1andCRVNearMissFraction : 1e-3have no stated source. The PR description does not say how they were chosen. - Cross-repo consistency: needs a KinKal tag with KinKal#239 at
cb8107dcor later, and an envset. EventNtuple needs follow-ups (findings 4 and 5).
Residual risk
- One OnSpill file and one extracted cosmic file. I did not run upstream or off-spill cosmic samples, where CRV near misses on LoopHelix and CentralHelix fits would matter.
- CPU numbers are from one run each on a shared machine.
- I did not check the CentralHelix (
CHXTRAP) fits or EventNtuple's own TrkCrvMatch configurations, which use other inputs. - I did not run Offline's validation suite or valCompare.
Author follow-ups
- Add
KinKal_Trajectoryto the CRVReco and CRVResponse link lists (finding 1). - Resolve the calorimeter-material loss with KinKal#239 (finding 2), in KinKal or in
KKShellXing, before this merges. - Decide on the TrkCrvMatcher change for old files, and say it in the description (finding 3).
- Say whether near misses are on by default in production, and open EventNtuple issues for the counters, segment tables and
inbFrac_(findings 4 and 5). - Fix the
inbFrac_comment or add a read rule (finding 5). - After the KinKal tag and envset bump, update
.museand re-run FNALbuild.
A near miss is defined using new functionality in KinKal, to calculate the trajectory covariance in-bounds fraction at a surface intersection. This new functionality gives a floating-point value for (effectively) the probability that the extrapolated trajectory actually passed through a material, even if that material wasn't included in the fit, which could be useful in analysis. It also allows refactoring the Crv plane extrapolation. This PR depends on KinKal KFTrack/KinKal#239
Summary of changes in #2204
This PR adds "near miss" intersections to the KinKal extrapolation. A near miss is a material crossing that is out of bounds, but where a configurable fraction of the trajectory's projected position covariance lies inside the surface boundary. It is kept in the track as an inactive material crossing. It is then used to replace the hand-built line projections in
TrkCrvMatchTool.The PR depends on KFTrack/KinKal#239, which provides
Intersection::pcov_,inbfrac_andhascov_,fillPlaneCovariance, the newplaneCovariance(time, plane, dirtol)signature, and theMaterial::extrapolatechange for inactive crossings.1. Near-miss intersections in the extrapolation (Mu2eKinKal)
ExtrapolateSweep):SweepTargethas a newnearfrac_setting (default 0, disabled).inbfrac_is at leastnearfrac_.inbfrac_is the fraction of the projected position covariance inside the surface boundary.ExtrapolateSweep::nearMiss(target, inter)identifies one.KKShellXingfor a near miss has no material crossings, so it is inactive. With KinKal#239,Material::extrapolatethen leaves the trajectory unsplit and adds no energy loss or scattering.boundsFractionruns only for candidates that pass.withinReachis skipped, because a near miss can lie past the cylinder's ends. This may add some cost. Tuning it is left for later optimization studies.KKShellXing::updateReference: it asks for the covariance projection when the crossing already has one, sopcov_andinbfrac_stay current through refits.KKExtrap:nearfrac_on the IPA, stopping-target foils, cylinder shells, plane shells and calorimeter panels, and uses a separate value for the CRV sectors. It is not set on calorimeter-support or record-only targets.rangesOut).KKExtrapConfiggainsNearMissFractionandCRVNearMissFraction, both defaulting to 0.0 (disabled).Mu2eKinKal/fcl/prolog.fcl, LHDRIFTXTRAP, LHSEEDXTRAP, KLXTRAP and CHXTRAP setNearMissFraction : 0.1andCRVNearMissFraction : 1e-3. These are starting values, pending optimization studies.KalIntersectionwithinbounds_ = false. Itskkinter_carriespcov_andinbfrac_, which estimate the probability that the track really passed through that material, for use in analysis.2. CRV matching refactor (CRVReco, RecoDataProducts)
TrkCrvMatchTool:projectXing, the hand-built line projections onto the extended plane of a sector, andTrkCrvPlaneError.hh. The CRV crossings now come directly from the CRVKalIntersections: in-bounds crossings and near misses.TrkCrvGeom.hh:trkCrvXingCovariance(ki, sector)takes the intersection'spcov_and rotates it to the sector (u, v) axes. If the intersection has no covariance, it falls back to the plane covariance of the intersection'sKinematicLine.trkCrvPlaneCovariance(ki, sector, point, time)gives the line's covariance on a sector plane at a candidate point.trkCrvMinCosToNormal= 1e-3.errorModelnow takes aKinKal::PMAT.TrkCrvXinggainsfloat inbFrac_. It holds the intersection'sinbfrac_when that was computed, and otherwise 1 for in bounds and 0 for out of bounds.inBounds_ == falsenow means a near miss rather than a projection, and the field comments are updated. There is no ClassVersion, so no dictionary edit is needed. Older files should be analysed with an older release.3. Downstream renames (CRVResponse, Validation)
TrkCrvMatchMCMaker:trkCrvPlaneCovariance.trueCluProj,chi2ndProj) keep their names, since that step still exists.ValTrkCrvMatchandValTrkCrvMatchMChistograms are renamed from "Proj" to "Near" (for exampleRecoXingProj→RecoXingNear,NXingProj→NXingNear,chi2ndofProj→chi2ndofNear), with matching titles. Comparisons against reference files made before this PR will not match these histograms.4. Tests
ExtrapolateSweepTest: a new near-miss section on aKinematicLinecrossing a 50x50 mm rectangle (1 mm position sigma) checks:pcov_is filled only when near misses are enabled;inbfrac_(for example 0.31 at 0.5 sigma outside the boundary);Production/JobConfig/recoMC/Extracted.fcl(field-off extracted cosmics, KKLine) ran on 200 events with no errors.inbfrac_values all respect the configured thresholds: 0.0018–0.49 on the CRV sectors and 0.11–0.46 on the strongback.TrkCrvMatcher, so the rewritten matching code has been compiled but not yet run on events.physics.producers.CosmicShowerFilter.ComboHitCollection : "makeSH".CosmicShowerFilterstill readsmakePHwhileSimpleTimeClusterusesmakeSH. This is unrelated to this PR and will be fixed in a separate one.— Claude (Opus 5.5, Anthropic), via Claude Code