[DBG_X] Multiple relvals failing due to "looking for SiStripNoises for a strip out of range: strip 0"

[iarspider]

In CMSSW_14_2_DBG_X_2024-10-03-2300, multiple RelVals failed with

----- Begin Fatal Exception 04-Oct-2024 06:21:16 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module SiStripDigiToRawModule/'SiStripDigiToRaw'
   [5] Calling method for module MixingModule/'mix'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 0
----- End Fatal Exception -------------------------------------------------

Full log: link

[iarspider]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 0

this in all likelihood is coming from the GT updates in #46184

[makortel]

assign simulation, alca

[makortel]

unassign reconstruction

MixingModule is not part of reconstruction

[cmsbuild]

New categories assigned: simulation,alca

@atpathak,@civanch,@consuegs,@kpedro88,@mdhildreth,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[iarspider]

@cms-sw/alca-l2 @cms-sw/simulation-l2 This is still happening - see e.g. 11601.0

----- Begin Fatal Exception 11-Oct-2024 08:39:03 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module SiStripDigiToRawModule/'SiStripDigiToRaw'
   [5] Calling method for module MixingModule/'mix'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 0
----- End Fatal Exception -------------------------------------------------

[perrotta]

[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 0

this in all likelihood is coming from the GT updates in #46184

More likely #46124, which included the latest GT updates for 2022MC used in 11601.0 (even though the SiStripNoises, tag was untouched in it)

@iarspider , maybe a naive question: the fact that it only shows up in the DBG_X builds couldn't point out to some wrong memory management in some Debug part of the code?

[mmusich]

the fact that it only shows up in the DBG_X builds couldn't point out to some wrong memory management in some Debug part of the code?

Nope:

cmssw/CondFormats/SiStripObjects/interface/SiStripNoises.h

Lines 72 to 77 in 6da1102

	static float getNoise(uint16_t strip, const Range& range) {
	#ifdef EDM_ML_DEBUG
	verify(strip, range);
	#endif
	return getNoiseFast(strip, range);
	}

cmssw/CondFormats/SiStripObjects/src/SiStripNoises.cc

Lines 61 to 65 in 6da1102

	void SiStripNoises::verify(uint16_t strip, const Range& range) {
	if (9 * strip >= (range.second - range.first) * 8)
	throw cms::Exception("CorruptedData")
	<< "[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip " << strip;
	}

verify - which is throwing now, is executed only in debug mode (because it's very time consuming)

[mmusich]

(even though the SiStripNoises, tag was untouched in it)

comparing the two versions of the GTs updated in PR #46124, I see:

conddb diff 140X_mcRun3_2022_realistic_v9 140X_mcRun3_2022_realistic_v3 | grep SiStrip
[2024-10-11 13:15:56,066] INFO: Connecting to pro [frontier://PromptProd/cms_conditions]
SiStripBadChannelRcd                 -               SiStripBadComponents_realisticMC_for2022_pre_EE_v0_mc            SiStripBadComponents_realisticMC_for2022_v2_mc            

IF SiStripBadComponents_realisticMC_for2022_pre_EE_v0_mc unmasked a strip that was otherwise masked before, it might have uncovered an underlying issue with the existing SiStripNoise tag.

[perrotta]

Thank you @mmusich

Then the message points out to some range where range.first is supposedly larger than range.second (because strip is 0 in the log message). That range is computed in CondTools/SiStrip/plugins/SiStripNoisesAndBadCompsChecker.cc

If we want to search the origin of it in the GT update, then the tag for SiStripBadChannel was updated in them, see the diff here: probably @cms-sw/trk-dpg-l2 can further check if those updated conditions may have originated the wrong range.

[mmusich]

That range is computed in CondTools/SiStrip/plugins/SiStripNoisesAndBadCompsChecker.cc

again no. This is just a utility to fix the payloads exactly for these use cases (making sure that the noise and bad components are covering the same sets of strips).

[mmusich]

If we want to search the origin of it in the GT update, then the tag for SiStripBadChannel was updated in them

this is exactly the content of #46250 (comment) I guess messages crossing...

[mmusich]

Something looks very wrong with SiStripBadComponents_realisticMC_for2022_pre_EE_v0_mc.
It's not masking a lot of channel that should be masked.

see difference w.r.t the previous tag SiStripBadComponents_realisticMC_for2022_v2_mc

@perrotta @cms-sw/alca-l2 why was this payload admitted to the queue?