Sequence-based simplified HLT phase2 menu. #44025

rovere · 2024-02-20T09:27:31Z

PR description:

This (large) PR is a rewrite of the Simplified Phase2 HLT Menu using sequences and abandoning the cms.ConditionalTasks almost completely. The notable exceptions are the Muons paths for which more expert help is needed to perform this transition.

The timing variant of the menu has been updated to include the recent move to L1P2GT and converted to using sequences as well.

Some of the pset configurations have been included in the PR even if they contain 0 changes. This is due to how the tool edmConfigSplit internally dumps those psets. Rather than chasing all of them one by one, I decided to let it do its job.

Unused modules have been removed. If needed, they could be extracted from git.

PR validation:

Run before/after this PR on some 100 events and the results are identical.

All paths have been tested one-by-one to verify that they could run alone, w/o any other paths present.

cmsbuild · 2024-02-20T09:28:38Z

cms-bot internal usage

cmsbuild · 2024-02-20T09:33:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44025/38955

This PR adds an extra 400KB to repository
Found files with invalid states:
- HLTrigger/Configuration/python/HLT_75e33/modules/simSiStripDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/fullDumpPhase2_HLT_OriginalNoTimingGeneric.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simSiPixelDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simEcalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHFNoseUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/options_cfi.py:
  - Deleted: 1a2126e
  - Added: ae63efe
- 24834.0_TTbar_14TeV+2026D98/genericPhase2SinglePath.py:
  - Added: f9f77b8
  - Modified: 2f14caf, 68959fd, 4d0efca, 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/maxEvents_cfi.py:
  - Deleted: 1a2126e
  - Added: ae63efe
- HLTrigger/Configuration/python/HLT_75e33/modules/randomEngineStateProducer_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterRECO_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simAPVsaturation_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHGCalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHcalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/services/DependencyGraph_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/tasks/patAlgosToolsTask_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterFEVT_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simCastorDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/paths/endjob_step_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterAOD_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/sequences/HLTDiphoton3024IsoCaloIdUnseededSequence_cfi.py:
  - Added: 1a2126e
  - Deleted: 641fbc9

cmsbuild · 2024-02-20T09:39:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44025/38956

This PR adds an extra 396KB to repository
Found files with invalid states:
- HLTrigger/Configuration/python/HLT_75e33/source_cfi.py:
  - Added: 1a2126e
  - Deleted: 30478c1
- HLTrigger/Configuration/python/HLT_75e33/modules/simSiStripDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/fullDumpPhase2_HLT_OriginalNoTimingGeneric.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simSiPixelDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simEcalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHFNoseUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/options_cfi.py:
  - Deleted: 1a2126e
  - Added: ae63efe
- 24834.0_TTbar_14TeV+2026D98/genericPhase2SinglePath.py:
  - Added: f9f77b8
  - Modified: 2f14caf, 68959fd, 4d0efca, 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/maxEvents_cfi.py:
  - Deleted: 1a2126e
  - Added: ae63efe
- HLTrigger/Configuration/python/HLT_75e33/modules/randomEngineStateProducer_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterRECO_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simAPVsaturation_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHGCalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simHcalUnsuppressedDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/services/DependencyGraph_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/tasks/patAlgosToolsTask_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterFEVT_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/modules/simCastorDigis_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/paths/endjob_step_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/psets/MEtoEDMConverterAOD_cfi.py:
  - Added: 1a2126e
  - Deleted: 442a458
- HLTrigger/Configuration/python/HLT_75e33/sequences/HLTDiphoton3024IsoCaloIdUnseededSequence_cfi.py:
  - Added: 1a2126e
  - Deleted: 641fbc9

cmsbuild · 2024-02-20T09:40:41Z

A new Pull Request was created by @rovere for master.

It involves the following packages:

24834.0_TTbar_14TeV+2026D98/genericPhase2SinglePath.py (****)
HLTrigger/Configuration (hlt)

The following packages do not have a category, yet:

24834.0_TTbar_14TeV+2026D98/genericPhase2SinglePath.py
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@cmsbuild, @mmusich, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@SohamBhattacharya, @wang0jin, @thomreis, @Martin-Grunewald, @silviodonato, @argiro, @rchatter, @ReyerBand, @missirol this is something you requested to watch as well.
@rappoccio, @sextonkennedy, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

rovere · 2024-02-20T09:43:25Z

@cmsbuild please test

rovere · 2024-02-20T09:45:07Z

I'm not sure why the bot is convinced that we need a new package. The file he is complaining about is part of the git history (and was needed while developing the PR), but it is not part of the final set of changes of this PR.
Ideas?

Martin-Grunewald · 2024-02-20T09:52:42Z

Remove and force-push?

mmusich · 2024-02-20T09:59:21Z

Ideas?

Do we need the whole history? Otherwise squash the commits to 1.

rovere · 2024-02-20T10:02:12Z

Ciao @mmusich @Martin-Grunewald I added to the git history for a reason and would rather have it in.
The history has been already rewritten via several rebases. I would be in favour of not squashing it into 1 single commit.
Besides the bot being confused, is there any other drawback in leaving it in?

Martin-Grunewald · 2024-02-20T10:07:34Z

Hmm, the bot should not create a 24834.0_TTbar_14TeV+2026D98/ even if empty.
Maybe ask @smuzaffar on what to do here so that the bot gets rid of the new-package-pending status?

iarspider · 2024-02-20T10:32:59Z

@rovere This PR touches 430 files, thus causing the bot to fail. If this is intentional, I think the best course of action would be to split it this PR two or more - e.g., move 1a2126e into a separate PR and remove it from history. You can use please test with #XYZ syntax to make bot merge additional PRs on top of this one for testing.

rovere · 2024-02-20T10:38:54Z

Ciao @iarspider thanks for your reply.
Can you clarify what do you mean by "causing the bot to fail"?
I'm not willing to split a PR based on the number of files changed.

@rovere This PR touches 430 files, thus causing the bot to fail. If this is intentional, I think the best course of action would be to split it this PR two or more - e.g., move 1a2126e into a separate PR and remove it from history. You can use please test with #XYZ syntax to make bot merge additional PRs on top of this one for testing.

iarspider · 2024-02-20T10:49:02Z

@rovere In order to reduce the number of GitHub API calls used per PR, the bot keeps a cache of some information regarding the PR (list of all commits, including squashed; list of files changed in each commit; reactions). That information is stored inside a comment, and thus is limited to 65k bytes. The bot will fail, i.e. it will not process a PR, if cache size is too big to be stored.

cmsbuild · 2024-02-20T11:02:35Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-008d13/37571/summary.html
COMMIT: 30478c1
CMSSW: CMSSW_14_1_X_2024-02-19-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44025/37571/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

25034.999

----- Begin Fatal Exception 20-Feb-2024 11:36:31 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'MC_Ele5_Open_L1Seeded'
   [2] Calling method for module Phase2TrackerClusterizer/'siPhase2Clusters'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::DetSetVector<Phase2TrackerDigi>
Looking for module label: mix
Looking for productInstanceName: Tracker

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

cmsbuild · 2024-03-08T05:59:38Z

Pull request has been put on hold by @mmusich
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

rovere · 2024-03-08T07:59:15Z

Ciao @mmusich thanks for these further checks. For completeness, these are the changes I made to the collections (to try and have a uniform approach and naming convention, far from being finalized, yet):

ecalRecHit:EcalRecHitsEB $\to$ hltEcalRecHit:EcalRecHitsEB
ecalRecHit:EcalRecHitsEE $\to$ hltEcalRecHit:EcalRecHitsEE (useless, it's empty by definition, should be removed in the long run)
hfprereco $\to$ hltHfprereco
hfreco $\to$ hltHfreco
horeco $\to$ hltHoreco

I wrote a Python script that compares the entries (and their members) of these collections before/after the PR, and they are identical.

On a side note, I just realized that the new naming convention is such that the hltHfprereco is not part of the FEVTHLTDEBUG event content, since the rule is keep *_hfprereco_*_*.

Should we consider this as a shower-stopper for this PR to be merged or can we have another PR to fix this?

mmusich · 2024-03-08T08:10:07Z

Thanks @rovere for the checks.

I wrote a Python script that compares the entries (and their members) of these collections before/after the PR, and they are identical.

Can you link this from here?

Should we consider this as a shower-stopper for this PR to be merged or can we have another PR to fix this?

Let's go ahead, but please open a bug-fix PR as soon as possible.

mmusich · 2024-03-08T08:10:18Z

unhold

cmsbuild · 2024-03-08T08:11:09Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

rovere · 2024-03-08T08:18:00Z

It is bread and butter, but it helped quite a bit:

link

cmsbuild · 2024-03-08T08:26:05Z

REMINDER @rappoccio, @sextonkennedy, @antoniovilela: This PR was tested with #44041, please check if they should be merged together

antoniovilela · 2024-03-11T16:18:17Z

Let's move on with this, but for our knowledge: what is automatically generated and what is manually maintained from the 430 files?

antoniovilela · 2024-03-11T16:18:30Z

+1

antoniovilela · 2024-03-11T16:30:40Z

Let's move on with this, but for our knowledge: what is automatically generated and what is manually maintained from the 430 files?

@rovere

mmusich · 2024-03-11T16:32:25Z

what is automatically generated and what is manually maintained from the 430 files?

we're not using confDB (yet) for phase-2, so this is completely manual (of course I guess @rovere used scripts to adjust and cross-check the configurations) @antoniovilela

rovere · 2024-03-12T08:02:14Z

@antoniovilela I mentioned the script used in the PR description. It is called edmConfigSplit and appropriately splits a fully expanded configuration into files and folders.
All the configurations should be checked and maintained by the corresponding developers.

antoniovilela · 2024-03-12T15:17:09Z

@antoniovilela I mentioned the script used in the PR description. It is called edmConfigSplit and appropriately splits a fully expanded configuration into files and folders. All the configurations should be checked and maintained by the corresponding developers.

I guess what I wanted to make sure (just because of my current hat in ORP) is that there is no risk of affecting modules that can be used outside of the Phase 2 processing. I guess so, but it does not hurt to confirm.

cmsbuild added this to the CMSSW_14_1_X milestone Feb 20, 2024

cmsbuild added hlt-pending pending-signatures tests-pending orp-pending new-package-pending code-checks-pending labels Feb 20, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 20, 2024

cmsbuild added tests-started and removed tests-pending labels Feb 20, 2024

cmsbuild added tests-rejected tests-pending and removed tests-started tests-rejected code-checks-approved labels Feb 20, 2024

cmsbuild added pending-signatures hold and removed fully-signed labels Mar 8, 2024

cmsbuild added fully-signed and removed pending-signatures hold labels Mar 8, 2024

cmsbuild added orp-approved and removed orp-pending labels Mar 11, 2024

cmsbuild merged commit faf732f into cms-sw:master Mar 11, 2024
11 checks passed

This was referenced Mar 12, 2024

Enable TF XLA, add tensorflow-xla-runtime. cms-sw/cmsdist#9005

Merged

Update to CUDA 12.4.0 cms-sw/cmsdist#9046

Merged

cmsbuild mentioned this pull request Mar 12, 2024

Using consistent Eigen alignment settings cms-sw/cmsdist#9060

Merged

cmsbuild mentioned this pull request Mar 13, 2024

Enable vectorization of lwtnn, opencv and pytorch cms-sw/cmsdist#9061

Merged

mmusich mentioned this pull request Mar 18, 2024

Produce r9 and trackIso(hollow cone) in EGamma development path #44423

Merged

swagata87 mentioned this pull request Mar 18, 2024

[14_0_X] Produce r9 and trackIso(hollow cone) in EGamma development path #44452

Merged

swagata87 mentioned this pull request Apr 2, 2024

Fix ordering of egamma HLT modules #44592

Merged

rovere mentioned this pull request Apr 3, 2024

[14.0.X] HLT phase-2: fix HGCAL Layercluster and create common constants configuration for HGCAL #44596

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sequence-based simplified HLT phase2 menu. #44025

Sequence-based simplified HLT phase2 menu. #44025

rovere commented Feb 20, 2024 •

edited

Loading

cmsbuild commented Feb 20, 2024 •

edited

Loading

cmsbuild commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

rovere commented Feb 20, 2024

rovere commented Feb 20, 2024

Martin-Grunewald commented Feb 20, 2024

mmusich commented Feb 20, 2024

rovere commented Feb 20, 2024

Martin-Grunewald commented Feb 20, 2024

iarspider commented Feb 20, 2024

rovere commented Feb 20, 2024

iarspider commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

cmsbuild commented Mar 8, 2024

rovere commented Mar 8, 2024

mmusich commented Mar 8, 2024

mmusich commented Mar 8, 2024

cmsbuild commented Mar 8, 2024

rovere commented Mar 8, 2024

cmsbuild commented Mar 8, 2024

antoniovilela commented Mar 11, 2024

antoniovilela commented Mar 11, 2024

antoniovilela commented Mar 11, 2024

mmusich commented Mar 11, 2024

rovere commented Mar 12, 2024

antoniovilela commented Mar 12, 2024

Sequence-based simplified HLT phase2 menu. #44025

Sequence-based simplified HLT phase2 menu. #44025

Conversation

rovere commented Feb 20, 2024 • edited Loading

PR description:

PR validation:

cmsbuild commented Feb 20, 2024 • edited Loading

cmsbuild commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

rovere commented Feb 20, 2024

rovere commented Feb 20, 2024

Martin-Grunewald commented Feb 20, 2024

mmusich commented Feb 20, 2024

rovere commented Feb 20, 2024

Martin-Grunewald commented Feb 20, 2024

iarspider commented Feb 20, 2024

rovere commented Feb 20, 2024

iarspider commented Feb 20, 2024

cmsbuild commented Feb 20, 2024

RelVals

cmsbuild commented Mar 8, 2024

rovere commented Mar 8, 2024

mmusich commented Mar 8, 2024

mmusich commented Mar 8, 2024

cmsbuild commented Mar 8, 2024

rovere commented Mar 8, 2024

cmsbuild commented Mar 8, 2024

antoniovilela commented Mar 11, 2024

antoniovilela commented Mar 11, 2024

antoniovilela commented Mar 11, 2024

mmusich commented Mar 11, 2024

rovere commented Mar 12, 2024

antoniovilela commented Mar 12, 2024

rovere commented Feb 20, 2024 •

edited

Loading

cmsbuild commented Feb 20, 2024 •

edited

Loading