Repository navigation
Increased speed for CRV simulation - #2150
Conversation
|
Hi @ehrlich-uva,
which require these tests: build. @Mu2e/fnalbuild-users, @Mu2e/write have access to CI actions on main. ⌛ The following tests have been triggered for 98d1d1d: build (Build queue - API unavailable) |
|
☀️ The build tests passed at 98d1d1d.
N.B. These results were obtained from a build of this Pull Request at 98d1d1d after being merged into the base branch at 7024300. For more information, please check the job page here. |
oksuzian
left a comment
There was a problem hiding this comment.
PR Review Summary — #2150
Reviewed at head 98d1d1d5cfb5364be2217187f775f76b5c916cb0. First pass.
Decision
- 🟢 approve. The SiPM charge generator gives exactly the same output as
main, and the photon generator gives the same distributions asmainwithin statistics, on the same input events. The speed-up is real: about 16 times forCrvPhotonGeneratorand about 3 times forCrvSiPMChargeGeneratorincosmicSimReco. FNALbuild build 3607 passed at this head. Itstriggerrow fails the same way on every currentmainbuild. The findings below are one S2 about very large channels and some S3 items; none of them blocks. This PR and Offline#2145 both replace the same photon loop, and this PR also overlaps Offline#2142. Please agree on a merge order; see the section after the findings.
Scope understood
MakeCrvPhotons: the 0.1 mm sub-steps of a CrvStep are still looked up one by one. A run of sub-steps that share all three lookup-table bins is then handled as one group: one binomial draw gives the number of arriving photons for the whole group, and all of them start at the group's centre time.findBinbecomes a binary search.MakeCrvSiPMCharges: the pixel map becomes a vector that is kept acrossSimulate()calls. A per-call counter marks pixels discharged in an earlier call as fully charged, and each pixel carries its own inactive flag. The multiset of scheduled charges becomes a vector that is sorted once. Cross-talk and after-pulse charges are inserted at their time-ordered position, and an index walks the vector instead of erasing processed charges.CrvSiPMChargeGenerator: the photons of each channel are found through a per-event index by channel number, instead of a search over the whole collection for every SiPM.
Findings
-
🟡 [S2] Inserting into the sorted vector costs time proportional to the queue length, so
Simulate()becomes quadratic in the number of photons on one SiPM.-
Evidence: cross-talk charges are inserted at
begin()+_currentChargeIndex(MakeCrvSiPMCharges.cc:76,:81,:86,:91), and after-pulses atupper_bound(:57-59,:66-68). Each insertion moves every charge still queued behind it. I timed oneSimulate()call with theprolog_v12SiPM constants and photons spread over 50 ns,mainagainst this PR, with the same seed; the number of output charges was the same in every case:photons on one SiPM mainthis PR 10,000 5.4 ms 4.2 ms 50,000 55 ms 46 ms 100,000 86 ms 118 ms 1,000,000 1.6 s 4.8 s 100,000 over 1 µs, after-pulses on 93 ms 1.8 s After-pulses are off in
prolog_v12(TrapType0ProbandTrapType1Probare 0), so the last row is not reached by current configurations. -
Impact: none in the samples I ran. The busiest channel held 7,639 photons in 10,000 off-spill cosmic events and 1,475 in the 565
cosmicSimRecoevents that reached the CRV, far below the crossover at 50,000 to 100,000 photons per channel. The risk is a long tail in wall time for rare very large deposits, and a large slowdown if after-pulses are ever switched on. -
Suggested fix: a cross-talk charge always carries the current time and is processed next, last in first out. A small stack that is popped before
_currentChargeIndexadvances gives the same order, so the output stays identical tomain, with constant-time insertion. After-pulses could go into a min-heap that is merged with the vector as it is read.
-
-
⚪ [S3] All photons of a merged group start at the group's centre time (
MakeCrvPhotons.cc:397-399). The mean time is kept, but the structure below 1 ns changes for steps that last nanoseconds. I droveMakePhotonsdirectly with fixed steps through both libraries. For a 60 mm step along the counter that lasts 5 ns, and a 19 mm crossing that lasts 3 ns, the 0.1 ns arrival-time histograms differ clearly (KS D 0.005 to 0.008, p < 0.001), while the mean agrees within 1.2σ and the rms within 0.03 ns. In the full simulation the time of each photon after the start of its step, in 0.1 ns bins, gives chi2/ndf 859/613 againstmain, where a reseededmaingives 681/621. In 1 ns bins the two numbers are 154/149 and 158/149. Pulse times do not change (KS p 0.30). The CRV is sampled every 12.55 ns, so I see no physics effect. If you wantmain's time distribution back exactly, give each arriving photon a random sub-step inside its group:timeStart + (timeEnd-timeStart)*(firstStep + floor(repeatedBins*u) + 0.5)/nSteps, wherefirstStepis the group's first sub-step andua uniform draw. That costs one extra draw per arriving photon. Otherwise, a comment that states the approximation would do. -
⚪ [S3] The comment at
MakeCrvPhotons.cc:427-428saysCLHEP::RandBinomialis low only up to n·p of about 0.03. In the test done for #2145 it was also 3.4σ low at (n, p) = (40, 0.2), where n·p is 8 (4M draws). The switch at 15.0 (:429) avoids that region. I ran this PR's sampler, copied verbatim, against the exact binomial at 15 (n, p) points from n·p = 0.001 to 1000, including p = 0.6 to 0.95, with 4M draws each. Every point matches (smallest chi2 p 0.06). The code is right. Please correct the comment so nobody lowers the switch on its advice, and give 15.0 a name. -
⚪ [S3] Smaller items:
MakeCrvSiPMCharges.hh:95: the new_crossTalkProbabilitySinglePixelhas no initializer, while the new_counter{0}and_currentChargeIndex{0}(:77,:82) do.MakeCrvSiPMCharges.hh:26: thePixelcomment says old counter values "count as dischared". They count as fully charged (GetVoltage,MakeCrvSiPMCharges.cc:109).CrvSiPMChargeGenerator_module.cc:206: a secondCrvPhotonsentry for one channel only getsmf::LogError, and the job goes on.CrvPhotonGeneratorwrites one entry per (bar, SiPM) from astd::map(CrvPhotonGenerator_module.cc:378), so a duplicate means the input is broken; please throw acet::exceptionthere. The new throw atMakeCrvSiPMCharges.cc:156should also be acet::exceptionrather thanstd::logic_error.MakeCrvPhotons.hh:100: the default yield goes from 39400 to 0. Dropping the unsourced number is right. But aMakeCrvPhotonswhoseSetScintillationYieldis never called now makes no light and says nothing. The module always calls it (CrvPhotonGenerator_module.cc:228), so nothing changes today. Passing the yield to the constructor would rule this out.MakeCrvPhotons.hh:116:ProcessPhotonsis public, but onlyMakePhotonscalls it; it can be private.MakeCrvPhotons.ccnow usesstd::lower_boundandstd::log1pwithout including<algorithm>and<cmath>.
Overlap with Offline#2145 and Offline#2142 (merge order)
- Both overlaps are textual conflicts (
git merge-treeof this head against each): #2145 (67228d53) conflicts inMakeCrvPhotons.hhandMakeCrvPhotons.cc, and #2142 (0a9a6084) conflicts inCrvSiPMChargeGenerator_module.cc. Each of the three merges cleanly intomainon its own. - #2145 and this PR are two implementations of the same idea, so only one should land as written.
CrvPhotonGenerator_module.ccmerges without a conflict marker, but the result cannot work. It keeps #2145'sbinomialArrivalsparameter and itsSetBinomialArrivalscall next to this PR's four-argument constructor. If this PR's version is the one kept,binomialArrivals : truemust also leaveprolog_v12.fcl, or FHiCL validation will reject the key. - What the two compute:
-
They agree on:
- the per-photon survival probability, the bin's
arrivalProbabilitytimes_photonYieldDeviation[SiPM]; probabilities of 1 or more let every photon arrive; - the number of photons created per sub-step;
- the bin lookup at each sub-step centre;
- the
findBinbinary search, which is the same apart from one comment line.
So the photon counts have the same distribution: the sum of k binomials with one p is one binomial.
- the per-photon survival probability, the bin's
-
They differ in three ways:
- Time: #2145 keeps each sub-step's own time, so its arrival times follow
main's distribution exactly. This PR uses the group centre (finding 2). - Random numbers: #2145 draws once per sub-step and photon type, by inversion, unless P(0) underflows (n·log q < −700). This PR draws once per group and lookup table, by inversion up to n·p = 15 and with
RandBinomialabove. Neither reproducesmainor the other event by event. - Old configurations: #2145 keeps the per-photon loop behind
binomialArrivals, which is false in C++ and true only inprolog_v12, soprolog_v03tov11reproduce older productions bit for bit. This PR has no switch, so every prolog version changes.
- Time: #2145 keeps each sub-step's own time, so its arrival times follow
-
Speed for
CrvPhotonGeneratoron the same off-spill cosmic input: #2145 measured 14.1 to 1.76 ms in its review; this PR measures 12.6 to 1.02 ms here.
-
- With #2142 both PRs build the same per-channel index. Even outside the conflict hunk, the merged file declares
photonsOfChanneltwice in one scope, so it would not compile. Whoever resolves it should keep one index and drop the other. #2142 skips an out-of-range SiPM number without a message. This PR fails loudly through.at(), which I prefer. #2142 also changesCaloDigiMaker, which does not overlap this PR.
Verified
-
🟢 SiPM charges are identical to
main. I builtmain70243009with only this PR's three SiPM files (MakeCrvSiPMCharges.hh,MakeCrvSiPMCharges.cc,CrvSiPMChargeGenerator_module.cc) and compared its output withmain's. Every charge's time, charge, PE and step reference matched, event by event, in three samples:- 10,000 off-spill cosmics in the production configuration (dark noise off), 5.1M charges;
- 2,000 with dark noise on, 106.5M charges;
- 1,000 with dark noise and after-pulses on (
TrapType0Prob0.14,TrapType1Prob0.06), 59.5M charges.
CrvDigis,CrvRecoPulsesandCrvCoincidenceClustersare identical too. This covers the per-call counter (no discharge carries over between channels or events), the inactive flags, the time-ordered insertion of cross-talk and after-pulses, and the processing index (no charge processed twice or skipped). It also shows that random numbers are drawn in the same order as before. The code agrees:stable_sortkeeps equal-time charges in insertion order, as the multiset did.- Cross-talk inserted at the current index runs last in, first out, as
emplace_hint(begin())did onmain. - The current charge is copied out before
GenerateAvalanchecan grow the vector (MakeCrvSiPMCharges.cc:199-207).
-
🟢 Threading. Both modules are legacy
art::EDProducers (CrvPhotonGenerator_module.cc:54,CrvSiPMChargeGenerator_module.cc:42), which art runs on one event at a time.mainalready kept the pixel map and the charge queue as members, so the state kept across calls does not change the threading picture. -
🟢 The photon generator output agrees with
mainwithin statistics. I ran 10,000 off-spill cosmic events (CosmicCRYAll MDC2025ap, Productiondigitize/OffSpill.fclplus CRV reconstruction) throughmain, this PR, andmainwith another seed as a control. Same input steps; paired per step, channel or event where marked:quantity mainPR PR vs maincontrol photons per CrvStep (35,183 steps) 358.55 358.57 z +0.17, KS p 0.87 z −0.11, KS p 0.89 photons per SiPM per event 102.05 102.05 z +0.17, chi2 p 0.98 z −0.11, chi2 p 0.95 photon time after its step's start, 1 ns bins (12.6M photons) 38.88 ns 38.90 ns chi2/ndf 154/149 158/149 SiPM PE per channel 41.02 41.03 z +0.38, KS p 0.95 z 0.00, KS p 0.77 CrvRecoPulsePE32.27 32.21 chi2 p 0.89, KS p 0.67 KS p 0.95 CrvRecoPulsetime after the event's first pulse24.78 ns 24.86 ns KS p 0.30 KS p 0.61 pulses per event 15.44 15.47 z +1.49 z +0.72 coincidence clusters per event 0.5727 0.5738 z +1.57 z +0.88 cluster PE 678.9 677.9 KS p 0.99 KS p 0.97 events with a coincidence 5,715 5,726 z +1.57 (19 lost, 30 gained) z +0.75 The KS and chi2 tests on individual photon times treat photons as independent, which they are not within one step. The reseeded control shows how large the scatter is.
- Two more samples agree. On 565
cosmicSimRecoevents with dark noise on, photons per step give z −0.43 and events with a coincidence 562 against 561. On 2,000 off-spill events with dark noise on, pulses per event give z +1.47 and clusters per event z −0.24. - I also split the steps of each sample into six subsets by deposit and by duration. The largest pulls are +2.0 (steps lasting 0.1 to 1 ns, 10,000-event sample) and −2.9 (0.5 to 2 MeV steps,
cosmicSimRecosample, 1,320 steps). The same 0.5 to 2 MeV subset gives +0.37 in the 10,000-event sample. The next item checks that regime directly. - I used "events with a coincidence" as the efficiency measure. It is not a truth-matched cosmic-muon efficiency.
- Two more samples agree. On 565
-
🟢 No bias in photon yield near bin edges, fibres or counter ends. The grouping only joins neighbouring sub-steps whose three bin numbers are all equal (
MakeCrvPhotons.cc:390-395). Each sub-step is still looked up at its own centre, as onmain, and a group's photon count is the per-sub-step count times the group length. I droveMakePhotonsdirectly through both libraries with fixed steps, 4,000 to 250,000 repetitions each:- a crossing at mid-length, a crossing 2 mm from the counter end, and a crossing through a fibre;
- a 60 mm step running into the counter end, a 0.05 mm step, and a slow 3 ns step;
- two 1 MeV electron-like steps, 4 and 5 mm long.
For five of these steps I also computed the exact expected number of arriving photons per SiPM from the lookup table, with no random numbers (the 60 mm, 0.05 mm and 3 ns steps were not computed). This PR's 28 SiPM means agree with it: chi2 17.5/28, every mean within 0.11%, and the mean pull is +0.14.
maingives chi2 37.3/28. Againstmain, the count per SiPM agrees in variance (ratios 0.96 to 1.04) and in shape (smallest chi2 p 0.015 over 40 comparisons). -
🟢 Speed. Per-event
TimeTrackeraverages,mainagainst this PR, both run at the same time on the same machine (a shared node, so the absolute numbers are higher than the author's):job CrvPhotonGeneratorCrvSiPMChargeGeneratorfull event cosmicSimReco, 4,000 events (565 reach the CRV)22.2 → 1.35 ms (16×) 53.7 → 16.7 ms (3.2×) 64.2 → 54.1 ms off-spill digitization, 10,000 cosmics, dark noise off 12.6 → 1.02 ms (12×) 1.09 → 0.50 ms 23.8 → 11.3 ms the same with dark noise on, 2,000 events 13.8 → 1.74 ms 56.0 → 17.7 ms (3.2×) 149 → 98 ms The ratios match the description, which gives 15 → 0.8 ms (19×) and 33 → 11 ms (3×).
-
🟢
findBinis the same code as in #2145, apart from one comment line. That version was compared with the old linear scan on 19.6M probes of all 14 v6_0 tables, with no mismatch.
Validation check
- Build/tests run: yes.
- Builds: sparse muse builds of
CRVResponseat this head, atmain70243009(this PR's base30088f39has the sameCRVResponse), and atmainplus this PR's three SiPM files, all on CI buildmain/70243009_dc19af05_1dc8ee40. All built with-Werrorand no warnings. No other package includes the two changed headers. - Jobs: Production
digitize/OffSpill.fclwith CRV reconstruction on six CosmicCRYAll MDC2025ap files, read in place from dCache; 10,000 events, plus 2,000 with dark noise and 1,000 with dark noise and after-pulses.Production/Validation/cosmicSimReco.fcl, 4,000 events. - Standalone tests of
MakePhotons,Simulate()and the binomial sampler, and an exact calculation of the expected photon counts from the lookup table.
- Builds: sparse muse builds of
- FNALbuild build 3607 at this head, merged on
main70243009, finished 21:25:09 UTC. Every row passed excepttrigger, which returned code 2. Its 991TrigReportcount lines and its comparison messages match build 3605 (Offline#2148, the samemain) line for line, except the timing line. The messages are theapr_protonrename (three new paths, one missing) and count changes on 32 paths, the same as in build 3605. The trigger job runs no CRV module at all, so nothing in that row comes from this PR. The whitespace check passed. The two clang-tidy errors are in cetlib'sLibraryManager.h. - Config contract check: pass. No FHiCL parameter changed.
- Cross-repo consistency: no Production change needed. Merge order with #2145 and #2142 needs coordinating, as described above.
Residual risk
- Every CRV simulation output changes event by event, including jobs that rerun an older production with its matching prolog. The distributions are the same.
- The quadratic insertion cost (finding 1) was not reached in these samples. I did not run on-spill mixed events, where the busiest channels may hold more photons.
Author follow-ups
oksuzian
left a comment
There was a problem hiding this comment.
PR Review Summary — #2150
Reviewed at head 045c1cbef30eae512498694586d5b1c0aea6bb99. Re-review of 98d1d1d.
Decision
- 🟡 comment now, approve when green. The new commit fixes most of the small items from my review and changes nothing in what the code computes. FNALbuild has not finished a build at this head yet (
mu2e/buildtestpending, checked 23:20 UTC); I will approve once it passes.
What changed since 98d1d1d
- One commit,
045c1cb"small changes in comments and throw messages" (5 files, +8 −5). Not a merge ofmain.
Prior findings (from my review at 98d1d1d5)
- 🟡 [S2, open, optional] Insertion into the sorted charge vector is quadratic in the photons on one SiPM. Unchanged.
- ⚪ [S3, open, optional] All photons of a merged group start at the group's centre time. Unchanged.
- 🟢 [was ⚪ S3] The
RandBinomialcomment (MakeCrvPhotons.cc:427) no longer gives a range: it now says the generator is too low for small n·p. ADDRESSED. The 15.0 switch is still a bare number. - Smaller items:
- 🟢 FIXED: a duplicate
CrvPhotonsentry for one channel now throws acet::exception(CrvSiPMChargeGenerator_module.cc:206). - 🟢 FIXED: the pixel-outside-SiPM check throws a
cet::exceptioninstead ofstd::logic_error(MakeCrvSiPMCharges.cc:158-159). - 🟢 FIXED: the
Pixelcomment says old counter values count as fully recharged. - 🟢 FIXED:
ProcessPhotonsis private. - ⚪ Open:
_crossTalkProbabilitySinglePixelstill has no initializer (MakeCrvSiPMCharges.hh:95). - ⚪ Open: the default yield of 0 (
MakeCrvPhotons.hh:100). - ⚪ Open: the missing
<algorithm>and<cmath>includes inMakeCrvPhotons.cc.
- 🟢 FIXED: a duplicate
- Merge order with Offline#2145 and #2142: unchanged.
git merge-treeof this head still conflicts with #2145 (67228d53) inMakeCrvPhotons.hhand.cc, and with #2142 (0a9a6084) inCrvSiPMChargeGenerator_module.cc. Neither of those PRs has moved, and I see no reply on the order yet.
Validation check
- Build/tests run: no new runs. I read the increment and checked the conflicts with
git merge-tree. The increment changes comments, one access specifier and two exception types, so the distributions, the bit-identical SiPM charges and the timing measured at98d1d1d5still apply. - Config contract check: pass.
- Cross-repo consistency: pass; merge order as above.
Residual risk
- As in my review at
98d1d1d5.
|
@FNALbuild run build test |
|
⌛ The following tests have been triggered for 045c1cb: build (Build queue - API unavailable) |
|
☀️ The build tests passed at 045c1cb.
N.B. These results were obtained from a build of this Pull Request at 045c1cb after being merged into the base branch at 7024300. For more information, please check the job page here. |
oksuzian
left a comment
There was a problem hiding this comment.
PR Review Summary — #2150
Reviewed at head 045c1cbef30eae512498694586d5b1c0aea6bb99. Follow-up to my comment review at the same head.
Decision
- 🟢 approve. FNALbuild build 3609 has now built this head, merged on
main70243009, and every row passed: the prof build, the validation jobs,rootOverlaps,g4surfaceCheckandtrigger. Both clang-tidy errors are in cetlib'sLibraryManager.h. This is the approval my comment review promised once the build was green. The optional findings stay as listed there. The merge order with Offline#2145 and #2142 still needs to be agreed.
|
Ready to be merged. |
Mu2e#2150 already reworked CRVResponse SiPM charge generation. Keep only the CaloDigiMaker once-per-event CaloShowerRO index from this PR.
Speed increase when running cosmicSimReco
Changes in photon generator
Changes in SiPM charge generator