Spurious comparison differences in workflows 136.X, 141.044, 4.54, ...

[perrotta]

Since a few IB's the workflows in the title show spurious comparison differences in DQM tests.
There are 6 failing comparisons in ParticleFlow_JetResponse_slimmedJets and 6 failing comparisons in ParticleFlow_JetResponse_slimmedJetsPuppi, both in the JEC subfolder.

Examples in:

• Replace CPP11_auto_ptr with std::unique_ptr  #43444 , baseline CMSSW_14_0_X_2023-11-30-1100
• Update ROCm tools #43451 , baseline CMSSW_14_0_X_2023-11-30-1100
• Run3-gex175 Remove std::cout in favour of edm::LogVerbatim in Fireworks/Geometry #43447 , baseline CMSSW_14_0_X_2023-11-30-1100
• Phase2-hcx359 Define modules for HFNose following the structure of updated HGCal (as in V17 version) #43445 , baseline CMSSW_14_0_X_2023-11-30-0800
• Added ActivityRegistry signals for EDProducer transforms #43442 , baseline CMSSW_14_0_X_2023-11-29-1100
• ...

Well, all PRs I can see that were run with the latest IB's show this "feature", while I don't see it for PRs tested with CMSSW_14_0_X_2023-11-28-1100 (e.g.)

The PRs that were merged between CMSSW_14_0_X_2023-11-28-1100 and CMSSW_14_0_X_2023-11-29-1100 are:
#43404 from smuzaffar: [SIMULATION] [DEVEL] Fix warnings on deprecated copy-constructors
#43338 from fwyzard: Extend the GenericConsumer to consume individual products
#43403 from civanch: [14_0_X SIM] Update FTFP_BERT_EMN physics
#43339 from bouchamaouihichem: Enabling the JetCore+DeepCore2.2.1 hybrid with MaxCand=30 (JC30DC)
#43242 from francescobrivio: Read VertexSmearing from GT in Run-1/2/3 MC
#42936 from fwyzard: Fix logic error in FinalPath checks
#43164 from RSalvatico: Change EGM code to be able to read HCAL PF thresholds from DB
#43386 from mmusich: [TkAl] remove "old" all-in-one metatool from cmssw
#43385 from mmusich: fix HLT track collection for SiStrip at HLT monitoring
#43348 from ggonzr: Update hash_name function for RelMon utilities
#43176 from swagata87: Migrate RecHit-based rho producer to use HCal noise cuts from GT
#43410 from aandvalenzuela: [ALCA][DB] Fix copy-constructor warnings reported in DEVEL IBs
#43304 from peteryouBuffalo: UPDATED: Adding of jet purity and efficiency plots to PFValidation package

The last PR (#43304) at a first glance could be the responsible...,

[perrotta]

assign pf, dqm

[cmsbuild]

New categories assigned: pf,dqm

@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini,@bellan,@kdlong,@swagata87 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @perrotta Andrea Perrotta.

@Dr15Jones, @smuzaffar, @antoniovilela, @sextonkennedy, @rappoccio, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[civanch]

assign simulation

[cmsbuild]

New categories assigned: simulation

@civanch,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks

[swagata87]

FYI @cms-sw/jetmet-pog-l2

[makortel]

@civanch Why assign to simulation? On a quick look only(? or mainly?) data workflows are affected.

[civanch]

@makortel , in the list of PRs SIM present, likely has nothing to do with the problem. I mainly add "simulation" for fast access of the issue.

[mmusich]

The last PR (#43304) at a first glance could be the responsible...,

What about #43304 (comment) ?

[perrotta]

@peteryouBuffalo in #43304 (review) was pointed out that there is an issue with the case if nEvents == 0.
In that thread you wrote "I will try to reach out and discuss with my colleagues at Buffalo who originally started and helped with the original PR (#39902), so we can figure out resolving this specific case. My assumption is that it should be fine to just continue if nEvents == 0, but I will check with others to confirm." (which I report here for completeness).

This issue is becoming annoying, and it can make difficult to check and review new PR. Did you verify whether it is correct to just continue after L126 if nEvents = 0? At the first order, I would just verify that it is not disrupting anything, as it make sense not continuing there if there are no events.

Could you please let us know if a PR is coming? Otherwise we can just try that suggestion in a fix PR and see what happens...

[peteryouBuffalo]

@perrotta We have a PR coming to fix this issue. @laurenhay is going to make a PR to add a statement to fix the nEvents == 0 case. We will have the PR taken care of as soon as possible.

[perrotta]

#43530 fixed the issue of the weird behaviour when nEvents = 0

It looks like it also fixed what pointed out here, that is the spurious differences in the test outputs of several workflows.

In fact, the PRs tested after #43530 was merged (since CMSSW_14_0_X_2023-12-11-2300) do not show those differences in the test output any more, see for example #43544 or #42633

Therefore, I think that this issue can get closed now