Re-skim fixes: luminosity and missing data, MET filters, 2022-2023 pileup and jet ID, VBF jets, trigger-SF uncertainty, job image option, softdrop mass by zichunhao · Pull Request #368 · LPC-HH/HH4b

zichunhao · 2026-10-27T07:23:28Z

Follows #367. One commit per topic; like #367, most of them change the skims and go with the next re-skim.

Commits

Samples: the 2024 $\mathrm{W}\to\mathrm{qq}$ PTQQ-600 v2-v3 production replaces the invalid v2-v2 one (index, file lists, its GenXSecAnalyzer cross section 1.217 pb), back in the V+jets stitch; a W-600 skim is used only if its totals per LHE V $p_\mathrm{T}$ show no event below 600 GeV. GenV1CS/GenV2CS accept both daughter orders of $\mathrm{W}\to\mathrm{cs}$ (both flags were always False in the 2024 WW sample).
Index: the missing 2024I MINIv6NANOv15_v2 and 2025C/F PromptReco-v1 JetMET data (the skims missed 49% of the 2024I and 61% and 72% of the 2025C and 2025F golden luminosity).
Luminosity:
- The data events outside the pinned Run 3 golden JSONs are dropped at load time (about 2% of the 2024/2025 data events were in uncertified lumisections).
- LUMI is the brilcalc luminosity of these golden JSONs (2022-2025: $282484.9\ \mathrm{pb}^{-1}$).
- Data jobs open every file first, with retries, leave out and record the files that never open, and record the entries of each file and the entries read; src/condor/check_data_files.py lists the lost files and resubmits their jobs.
- The first load of a year writes <tag>/lumi_<year>.json: the lumisections the skims lost are dropped from the data and the MC is scaled to the luminosity the data hold (the current skims hold 94.8% of the 2024 and 70.5% of the 2025 golden luminosity; the re-skim with the indexed data restores it).
- The FOM cache key gets __lumi<hash>, after main's __dibosonexact.
MET filters: era-dependent lists (each Run 3 filter once, ecalBadCalibFilter for 2024; the ttSkimmer hfNoisyHitsFilter typo); 2022-2023 data get the JME jet-based veto of the bad EB crystal; data skims save Flag_ecalBadCalibFilter, which utils.load_samples requires for 2025 data while their MC is the 2024 MC (MC_SHARING_DATA_FILTERS).
Corrections: the 2022-2023 pileup weights are evaluated at Pileup_nTrueInt, the variable they are a function of (they were evaluated at Pileup_nPU; 2018 keeps nPU); the 2022-2023 AK4 jet ID comes from the JME JetID JSON (corrections/data/jetid.json.gz; jetid_v12 kept 46-57% of the jets beyond $\lvert\eta\rvert = 3$ instead of 89-93%).
VBF jets: the two highest-$p_\mathrm{T}$ candidates.
Trigger SF: per-event trigger-SF uncertainty (relative efficiency errors in quadrature), with the jets outside the efficiency maps at the edge bins instead of SF = 1 (FOM cache key __trigedge); the unit tests of calculate_trigger_weights expect the per-event uncertainty.
Condor job image: submit.py --image (also in submit_from_yaml.py) chooses the image, since coffea-dask:latest-py3.8 is no longer on /cvmfs/unpacked.cern.ch:
- zichun/hh4b-docker (default): gitlab-registry.cern.ch/zichun/hh4b-docker, coffea-base-almalinux8 with a Python 3.11 environment that holds the pinned requirements; the jobs run HH4b from the clone with PYTHONPATH=src, without pip install.
- coffeateam/coffea-base-almalinux8 (Python 3.8): the jobs pip install HH4b and requirements.txt, as before.
requirements.txt keeps its pins, which the Python 3.8 image needs, with a note on the versions Python 3.11 needs (correctionlib 2.9.0, numba 0.60.0); pytest ignores the Python 3.11 DeprecationWarnings of sre_parse and sre_constants (lark parser of awkward 1.10).
JMR placeholder: the GloParT-v3 JMR placeholder is one-sided, up 1.05 (was 1.15), down 1; CreateDatacard --jmr-symmetrize (default) gives combine the up variation mirrored about the nominal as the down template.
Softdrop mass: the skimmers recompute it from the subjets re-corrected with the AK4 JEC of our AK4 jets, as PocketCoffea's msoftdrop_correction (no JER; NanoAOD's value kept as MsdNano; the $m_\mathrm{SD}$ cuts and the JMS/JMR variations use the new one). NanoAOD msoftdrop carries the subjet JEC of the MiniAOD production: +0.6% (2024 MC) and -1.6% (2024C data) away from V5; in the 2023 PFNano data it is 7% below the mass of its own subjets. On the Correctionlib JECs for 2022-2023, JES/JER of the AK4 BDT inputs, Type-1 PUPPI MET with the re-applied JECs #367 comparison file it equals the V5 softdrop mass that approach 1 and PocketCoffea reproduce.

Checks

Rebased onto main after Correctionlib JECs for 2022-2023, JES/JER of the AK4 BDT inputs, Type-1 PUPPI MET with the re-applied JECs #367: the corrections commit keeps only its pileup and jet-ID parts, since Correctionlib JECs for 2022-2023, JES/JER of the AK4 BDT inputs, Type-1 PUPPI MET with the re-applied JECs #367 already merged the correction tables; the tree equals the pre-rebase branch plus Correctionlib JECs for 2022-2023, JES/JER of the AK4 BDT inputs, Type-1 PUPPI MET with the re-applied JECs #367's $\phi$ clip and README links.
Unit tests: 286 passed at every commit, after the rebase.
The trigger SF and its uncertainty equal an independent recomputation (2022EE and 2023BPix ggF HH, 2024 TTto4Q).
One-chunk skims of 2024 and 2022 TTto4Q, 2025C JetMET and 2023 Run C v4 JetMET run.
The hh4b-docker environment gives the same one-chunk skims of 2024 TTto4Q and 2025C JetMET as the current one (up to $6\times10^{-17}$ in two MET $\phi$ columns), and the unit tests pass in it (Python 3.11) except test_package.py::test_version, which needs a pip-installed package.
Softdrop mass, one-chunk skims before and after (2024 and 2022 TTto4Q, 2025C and 2023 Run C v4 JetMET): only the softdrop columns change, MsdNano equals the previous Msd; median new / NanoAOD 0.992, 1.000, 1.04 and 1.11, with 1 and 12 more data events passing the $m_\mathrm{SD}$ cuts.
22 one-file condor jobs (TTto4Q and one JetMET era per year, 2022-2025) ran with the coffea-base image.
Condor, --image: one-file jobs of 2024 TTto4Q and 2025C JetMET with each image give the same skims (all columns identical except the MET ones, at $10^{-11}$) and cutflows. The two coffea-base 2025C jobs failed once in coffea's read of the file through the global redirector and ran when resubmitted.
Condor, tip: one-file jobs of 2024 TTto4Q and 2023 Run C v4 JetMET with the default image ran; softdrop mass / NanoAOD 0.992 and 1.11, as in the local skims.
Like Correctionlib JECs for 2022-2023, JES/JER of the AK4 BDT inputs, Type-1 PUPPI MET with the re-applied JECs #367, these change the skims (MET filters, data-file records, jet ID, VBF jets, the saved flag, the softdrop mass), so they go with the next re-skim.

…he dibosons The indexed Wto2Q-2Jets_Bin-PTQQ-600 v2-v2 production is INVALID in DAS (generated with the LHE V pT filter at 400 GeV), so it was left out of the V+jets stitch. The index and make_filelists_v15.py now point to the v2-v3 production (filter at 600 GeV, 72 files on T1_US_FNAL_Disk), with its GenXSecAnalyzer cross section (1.217 pb, 18 MiniAOD files), and the stitch selectors include the sample again. The static guard that raised for W-600 is replaced by a check of the skim: load_samples uses a W-600 skim only if its per-LHE-V-pT-bin totals show no generated event below 600 GeV, so skims of the old production (or without those totals) are left out with a warning, as before. The PTQQ stitch comments in utils.py are shortened. GenV1CS and GenV2CS tested the (c, s) daughter order twice. In the 2024 WW sample every hard-process W -> cs has its daughters as (s, c) (32,008 of 32,008 in the first 50k events), so both flags were always False. They now accept (c, s) and (s, c).

The data index lacked part of three eras: 2024I MINIv6NANOv15_v2 (runs 386694-386951) and 2025C/F PromptReco-v1 (runs 392174-393087 and 396629-397596). Each holds the runs of its era that the indexed dataset does not (the prompt processing changed version mid-era), so the skims missed 49% of the 2024I and 61% and 72% of the 2025C and 2025F golden luminosity. They are added to make_filelists_v15.py and to those three entries of nanoindex_v15.json; JetMET0 2025C-v1 and JetMET1 2025F-v1 are not on FNAL disk and are read through the global redirector.

…misections The Run 3 golden JSONs are pinned in corrections/data/golden_json/ (fetch_golden_json.py updates them from the CMS CAT area). utils.load_samples now drops the data events outside the year's golden JSON (hh_vars.GOLDEN_JSON); run and luminosityBlock are read for it and dropped again unless requested. No lumi mask was applied before: about 2% of the 2024/2025 data events (up to 5.6% per era) and up to 0.5% in 2022-2023 were in uncertified lumisections. LUMI is the recorded luminosity of these golden JSONs in the analyzed eras, now covered by the data index, from brilcalc (corrections/golden_lumi.py, which also reports how much of it the data skims of a tag cover; install_brilcalc.sh installs brilcalc for offsite use): normtag_PHYSICS, and for 2025, which normtag_PHYSICS does not cover yet, normtag_BRIL (preliminary). 2025 now includes era G. 2022-2025: 282484.9 pb^-1 instead of 261808.6. MC skims keep the luminosity they were made with until the next re-skim; the 1x/2x check of the 2022-2023 W cross sections now tolerates a LUMI update since the skim. The FOM slim cache key gets "__lumi<hash>" (content of the golden JSON, LUMI of the year and of its MC source year), so caches built before are not read. coffea skips a data file that raises FileNotFoundError at once, and a chunk whose reads keep timing out after its retries, with only a warning in the job log, so lost data went unnoticed (run.py meant to stop skipping for data, but compared the fileset keys with the sample names, which never match). Data jobs now open every file first, three times one minute apart (an authentication error still fails the job), leave out the files that never open, and save in the job pickle these files with their error, the entries of the others, and the entries the skimmer read from each file. A job whose processor fails exits non-zero without copying any output. src/condor/check_data_files.py (logic in HH4b/data_files.py) lists per data sample the lost files (not opened, read incompletely, in no job's output) and the files read twice, with their jobs, and resubmits those jobs with --submit-missing. The first load of a year now writes <tag>/lumi_<year>.json (HH4b/data_lumi.py): {} if the data skims hold every golden lumisection of LUMI[year], else the lumisections they lost and lumi_pb, the luminosity they hold. The lost lumisections come from the job records (those of the lost files from DAS, or from the file), or, for skims made before the records, are the golden lumisections without any data event in the skims. lumi_pb = LUMI x (1 - f), f the lost fraction of the golden luminosity from the per-lumisection luminosity of the DC pileup JSON. utils.load_samples drops the lost lumisections from the data (whole lumisections: they are split between JetMET0 and JetMET1 and over several files) and scales the MC used for the year by lumi_pb / LUMI; the FOM cache key includes the file and golden_lumi.py --tag leaves the lost lumisections out. The era run ranges move to hh_vars.DATA_ERAS. On the current skims, which have no records, this matches the brilcalc coverage within 0.2%; their 2024 and 2025 MC are scaled by 0.9475 and 0.7046.

…alBadCalibFilter at load time bbbbSkimmer listed eeBadScFilter twice. The base list now has each Run 3 flag once, and Flag_ecalBadCalibFilter is added for 2024 only: in 2022-2023 NanoAOD it is still the Run 2 filter of 35 EE crystals and misses the bad EB crystal of those years, for which the JME recipe is a data-only jet veto instead. A listed flag that is missing from the input is now logged instead of skipped silently. ttSkimmer listed "hfNoisyHitsFilt", which the silent check dropped, so hfNoisyHitsFilter was never applied there. In 2022-2023 NanoAOD Flag_ecalBadCalibFilter is still the Run 2 filter of 35 EE crystals and misses the bad EB crystal of runs 362433-367144, so data get the JME jet-based veto: an event of those runs is rejected if PuppiMET pT > 100 GeV and a jet has pT > 50 GeV, -0.5 < eta < -0.1, -2.1 < phi < -1.8, a neutral or charged EM fraction > 0.9 and |dphi(jet, PuppiMET)| > 2.9. It removes a few ppm of the 2022G JetMET events. The skims save Flag_ecalBadCalibFilter for data, and utils.load_samples requires it for the 2025 data (MC_SHARING_DATA_FILTERS, TODO(mc-sharing)): their MC is the 2024 MC, whose MET filter list has it, while the 2025 recommendation omits it, so the skimmer keeps the per-year lists. Data skims without the column are loaded as before, with a warning. The flag passes 99.93% of the 2025C JetMET events.

The 2022-2023 inputs are the PFNano 25v2, NanoAODv14 files skimmed through the v15 path, where their jets went through jetid_v12, the fix for the wrong NanoAODv12 jetId bits: beyond |eta| = 3 it kept 46-57% of the jets instead of the 89-93% the JetID criteria keep, so 2022-2023 had fewer VBF jets (SM VBF HH events with two VBF jets: 49.9% in 2022EE, 57.6% in 2024). Their Tight and TightLeptonVeto now come from the JME JetID JSON, corrections/data/jetid.json.gz (the file of the 2022-2025 JME folders of the CAT area), with its int inputs read as real, since correctionlib 2.2.2 cannot bin on int inputs; a TODO falls back to jetid_v15 once the 2022-2023 NanoAODv15 MCs are ready. 2024-2025 keep jetid_v15, which the JSON reproduces except at |eta| = 3.0 exactly. Below |eta| = 3 the 2022-2023 ID changes for 0.2% of the jets; a 2022EE SM VBF HH chunk now has two VBF jets in 56.5% of the events. The 2022-2023 pileup weights are a function of the true number of interactions but were evaluated at Pileup_nPU, kept there so that new skims of those years matched the existing ones, which the re-skim of every year no longer needs; they are now evaluated at Pileup_nTrueInt (2018 keeps nPU). The 2018, 2024 and 2025 weights, the JECs and the jet veto decisions are unchanged; in 50k TTto4Q events of each 2022-2023 era the mean pileup weight is now 1.000 within 0.2% (it was 0.955-1.058) and its RMS drops from 0.53-0.78 to 0.48-0.62. The sample normalization, divided by the sum of these weights, does not change.

objects.vbf_jets returned the first two jets passing its selection, and the jets keep the NanoAOD order, the pT order with the production's JECs; with the re-applied JECs and JER the second VBF jet had the higher pT in 2-3.5% of the 2024 events. It now takes the two candidates with the highest corrected pT; the jet collections and the other selections are unchanged. In a 2024 TTto4Q chunk no VBF pair is out of order.

…ficiency maps at the edge bins The trigger-SF uncertainty multiplied the absolute errors of the (pT, mSD) and TXbb efficiencies of each jet instead of adding the relative errors in quadrature, and the templates then took a flat per-sample factor of 1 ± 1e-5-1e-3 instead of the per-event uncertainty that trigger_SF already computed. Jets outside the maps (in 2022-2023 mostly second jets with mSD below 50 GeV, in 53% of the ggF HH events) got efficiency and error 1.0 from the map flow, i.e. SF = 1; they now take the edge bins. weight_triggerUp/Down use the per-event uncertainty, with the OR of the two jets written without dividing by 1 - efficiency. On the ggF HH skims the uncertainty is now 1.5% (2022EE), 3.7% (2023BPix) and 1.9% (2024) per event, and the nominal SF moves by -0.13%, -0.43% and 0; the FOM cache key gets "__trigedge". The unit tests of calculate_trigger_weights expect the per-event uncertainty.

The jobs asked for coffeateam/coffea-dask:latest-py3.8, which is no longer on /cvmfs/unpacked.cern.ch, so every job failed at the container. submit.py --image now chooses the image of the jobs: - zichun/hh4b-docker (default): gitlab-registry.cern.ch/zichun/hh4b-docker, the coffea-base-almalinux8:0.7.22-py3.8 image with a Python 3.11 environment that holds the pinned requirements. The job runs HH4b from the clone with PYTHONPATH=src, without pip install. - coffeateam/coffea-base-almalinux8: coffea-base-almalinux8:0.7.22-py3.8 (Python 3.8 and coffea); the job pip installs HH4b and requirements.txt, as before. requirements.txt keeps its pins, which the pip install on Python 3.8 needs, with a note on the versions Python 3.11 needs, which the image has: correctionlib 2.9.0 (2.2.2 has no Python 3.11 wheel) and numba 0.60.0 (newer numba breaks the numba code of awkward 1.10). pytest ignores the DeprecationWarnings of sre_parse and sre_constants, which the lark parser of awkward 1.10 imports on Python 3.11. One-chunk skims of 2024 TTto4Q and 2025C JetMET in the hh4b-docker environment equal those of the current environment, up to 6e-17 in two MET phi columns; a test of 22 one-file jobs (TTto4Q and one JetMET era per year, 2022-2025) ran with the coffea-base image.

The GloParT-v3 JMR placeholder goes from up 1.15 to up 1.05, down 1 (no extra smearing). JMR can only widen the mass, so CreateDatacard --jmr-symmetrize (on by default) ignores the JMR-down templates of the GloParT-v3 placeholder and gives combine the up variation mirrored about the nominal, Down = max(2 x Nominal - Up, 0) per bin, instead of a one-sided morph; rhalphalib's own symmetrization is wrong for shape effects. The measured GloParT-v2 and PNet JMR keep their down templates.

NanoAOD msoftdrop sums the subjets with the JEC of the MiniAOD production, while the jets carry the re-applied JECs. In 2024 that subjet JEC (Winter24 in MC, a pre-Summer24 payload in the 2024C data) puts msoftdrop +0.6% (MC) and -1.6% (data) away from Summer24Prompt24_V5, and in the 2023 PFNano data msoftdrop is 7% below the mass of its own subjets. As in PocketCoffea's msoftdrop_correction, each raw subjet is now corrected with the AK4 PUPPI JEC of the AK4 jets (L1L2L3 in MC, with the residuals in data), without JER, and the softdrop mass is the mass of their sum; jets without subjets or with msoftdrop <= 0, and NaN masses, keep NanoAOD's value, which is saved as MsdNano. The cuts and the JMS/JMR variations use the new mass. On the 2024 ggHH file of the #367 ntuple comparison it equals the V5 softdrop mass of that comparison to 6e-8, which approach 1 and PocketCoffea reproduce. In one-chunk skims only the softdrop columns change: by 0.992 in 2024 MC, 1.000 in 2022 MC, 1.04 in 2025C data and 1.11 in 2023C data (median ratio to NanoAOD), with 1 and 12 more data events passing the msd cuts.

cmantill

LGTM, minor comment here

cmantill · 2026-10-27T15:11:59Z

@@ -420,7 +420,8 @@
            "Wto2Q-2Jets_Bin-PTQQ-100": "/Wto2Q-2Jets_Bin-PTQQ-100_TuneCP5_13p6TeV_amcatnloFXFX-pythia8/RunIII2024Summer24NanoAODv15-150X_mcRun3_2024_realistic_v2-v3/NANOAODSIM",


Question: I recall that you found a bug where these samples (or similar ones) had only an upper cut > 100 GeV, so you had to save the LHEGenPt to stitch them, is that still the case?

Discussed in the meeting, but for bookkeeping: Yes, we have to save the LHE VPt and use the variable to stitch the samples.

zichunhao added 10 commits October 26, 2026 19:22

zichunhao requested a review from cmantill October 27, 2026 07:29

cmantill reviewed Oct 27, 2026

View reviewed changes

cmantill merged commit 1294a15 into main Oct 27, 2026
5 checks passed

zichunhao deleted the general-fixes-run3 branch October 28, 2026 05:04

Navigation Menu

Repository navigation

Re-skim fixes: luminosity and missing data, MET filters, 2022-2023 pileup and jet ID, VBF jets, trigger-SF uncertainty, job image option, softdrop mass - #368

Re-skim fixes: luminosity and missing data, MET filters, 2022-2023 pileup and jet ID, VBF jets, trigger-SF uncertainty, job image option, softdrop mass#368
cmantill merged 10 commits into
mainfrom
general-fixes-run3

zichunhao commented Oct 27, 2026 •

edited

Loading

Uh oh!

cmantill left a comment

Uh oh!

cmantill Oct 27, 2026

Uh oh!

zichunhao Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -420,7 +420,8 @@
		"Wto2Q-2Jets_Bin-PTQQ-100": "/Wto2Q-2Jets_Bin-PTQQ-100_TuneCP5_13p6TeV_amcatnloFXFX-pythia8/RunIII2024Summer24NanoAODv15-150X_mcRun3_2024_realistic_v2-v3/NANOAODSIM",

Conversation

zichunhao commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commits

Checks

Uh oh!

cmantill left a comment

Choose a reason for hiding this comment

Uh oh!

cmantill Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

zichunhao Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zichunhao commented Oct 27, 2026 •

edited

Loading