Add customize to include Alpaka HCal PF Clustering at HLT #43971

waredjeb · 2024-02-15T07:59:00Z

This PR add customizeHLTforAlpakaParticleFlowClustering() to customizeHLTForAlpaka() to integrate the Alpaka HCAL PF Clustering at HLT.

The function defines all the needed modules and replace them in the necessary Sequences. Currently it also run the Alpaka CPU-serial version up to the PFClusterSoA collection, before the Legacy conversion.

To compare the GPU version against the CPU version a follow-up PR will be needed to perform the CPU legacy conversion and the comparison at DQM.

Validation

Validated running the following HLT configuration

hltGetConfiguration /dev/CMSSW_14_0_0/GRun --unprescale --output all --globaltag auto:phase1_2024_realistic --mc --max-events 10 --input /store/mc/Run3Winter24Digi/TT_TuneCP5_13p6TeV_powheg-pythia8/GEN-SIM-RAW/133X_mcRun3_2024_realistic_v8-v2/80000/dc984f7f-2e54-48c4-8950-5daa848b6db9.root --customise HLTrigger/Configuration/customizeHLTforAlpaka.customizeHLTforAlpaka

And also by merging #43701 and running workflow 12434.423

FYI @jsamudio @hatakeyamak

cmsbuild · 2024-02-15T07:59:21Z

cms-bot internal usage

cmsbuild · 2024-02-15T08:03:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43971/38876

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforAlpaka.py modified in PR(s): Alpaka vs CUDA DQM compare modules for pixel tracks objects #43964

cmsbuild · 2024-02-15T08:03:50Z

A new Pull Request was created by @waredjeb (Wahid Redjeb) for master.

It involves the following packages:

HLTrigger/Configuration (hlt)

@cmsbuild, @Martin-Grunewald, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @silviodonato this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2024-02-15T08:06:32Z

enable gpu

mmusich · 2024-02-15T08:06:36Z

test parameters:

workflow_opts_gpu= -w upgrade
workflows_gpu= 12434.423

cmsbuild · 2024-02-15T08:16:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43971/38877

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforAlpaka.py modified in PR(s): Alpaka vs CUDA DQM compare modules for pixel tracks objects #43964

cmsbuild · 2024-02-15T08:16:56Z

Pull request #43971 was updated. @Martin-Grunewald, @cmsbuild, @mmusich can you please check and sign again.

fwyzard · 2024-02-15T08:20:08Z

please test

cmsbuild · 2024-02-15T11:06:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b6a17/37478/summary.html
COMMIT: 2b472df
CMSSW: CMSSW_14_1_X_2024-02-14-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43971/37478/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 62 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3248554
DQMHistoTests: Total failures: 1189
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3247343
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 48 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 40949
DQMHistoTests: Total failures: 1835
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 39114
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 15 edm output root files, 4 DQM output files
TriggerResults: found differences in 1 / 3 workflows

mmusich · 2024-02-15T14:24:27Z

Some preliminary checks from hlt side:

customization function looks OK from visual spot-check
workflow 12434.423, in particular step2 run without errors / warnings
private TSG check with [1] doesn't reveal issues.

@waredjeb where is the 14.0.X backport? it was privately communicated that this needs to enter 14_0_0.

[1]

hltGetConfiguration /dev/CMSSW_14_0_0/GRun --unprescale --output all --globaltag auto:phase1_2024_realistic --mc --max-events 10 --input /store/mc/Run3Winter24Digi/TT_TuneCP5_13p6TeV_powheg-pythia8/GEN-SIM-RAW/133X_mcRun3_2024_realistic_v8-v2/80000/dc984f7f-2e54-48c4-8950-5daa848b6db9.root --customise HLTrigger/Configuration/customizeHLTforAlpaka.customizeHLTforAlpaka > & hlt.py &
cmsRun hlt.py > & hlt.log &

Martin-Grunewald · 2024-02-15T14:30:09Z

HLTrigger/Configuration/python/customizeHLTforAlpaka.py

+            src = cms.InputTag("hltPFRecHitSoAProducerHCALCPUSerial")
+            )
+
+    process.particleFlowRecHitHF = cms.EDProducer("PFRecHitProducer",


Modules run at HLT start their label with lower-case hlt. Also, it actually seems this module is not further used anywhere?

That's right, I am checking that everything works correctly without this module

updated removing this module, thanks for spotting it!

cmsbuild · 2024-02-15T14:52:23Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43971/38893

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforAlpaka.py modified in PR(s): Alpaka vs CUDA DQM compare modules for pixel tracks objects #43964

cmsbuild · 2024-02-15T14:52:44Z

Pull request #43971 was updated. @mmusich, @Martin-Grunewald, @cmsbuild can you please check and sign again.

mmusich · 2024-02-15T15:36:15Z

@cmsbuild, please test

waredjeb · 2024-02-15T16:30:07Z

Possible that the tests failed with this error?

17:23:23 Traceback (most recent call last):
17:23:23   File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43979/37499/CMSSW_14_0_X_2024-02-15-1100/bin/el8_amd64_gcc12/runTheMatrix.py", line 762, in <module>
17:23:23     ret = runSelected(opt)
17:23:23   File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43979/37499/CMSSW_14_0_X_2024-02-15-1100/bin/el8_amd64_gcc12/runTheMatrix.py", line 31, in runSelected
17:23:23     if len(undefSet)>0: raise ValueError('Undefined workflows: '+', '.join(map(str,list(undefSet))))
17:23:23 ValueError: Undefined workflows: 12434.423

mmusich · 2024-02-15T16:34:11Z

Possible that the tests failed with this error?

it shows as failed here because it's the same branch as the backport and the backport is failing because of that reason.
I assume the reason is that the backport of #43701 is not yet in an IB an thus workflow 12434.423 is undefined.
Test should finish successfully here as they did before.

waredjeb · 2024-02-15T16:34:44Z

Possible that the tests failed with this error?

it shows as failed here because it's the same branch as the backport and it's failing because of that reason. I assume the reason is that the backport of #43701 is not yet in an IB an thus workflow 12434.423 is undefined. Test should finish successfully here as they did before.

I see, thanks for the clarification

cmsbuild · 2024-02-15T22:01:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b6a17/37498/summary.html
COMMIT: 28e9220
CMSSW: CMSSW_14_1_X_2024-02-15-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43971/37498/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 4 lines from the logs
Reco comparison results: 57 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3248554
DQMHistoTests: Total failures: 1192
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3247340
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 40949
DQMHistoTests: Total failures: 506
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 40443
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 15 edm output root files, 4 DQM output files
TriggerResults: found differences in 1 / 3 workflows

Martin-Grunewald · 2024-02-16T10:31:18Z

+1

cmsbuild · 2024-02-16T10:31:41Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2024-02-16T12:47:39Z

+1

missirol

A couple of late questions.

missirol · 2024-02-17T15:45:14Z

HLTrigger/Configuration/python/customizeHLTforAlpaka.py

+    process.hltPFRecHitHCALParamsESProducer = cms.ESProducer('PFRecHitHCALParamsESProducer@alpaka',
+      energyThresholdsHB = cms.vdouble(
+        0.1,
+        0.2,
+        0.3,
+        0.3
+      ),
+      energyThresholdsHE = cms.vdouble(
+        0.1,
+        0.2,
+        0.2,
+        0.2,
+        0.2,
+        0.2,
+        0.2
+      ),
+      appendToDataLabel = cms.string(''),
+    )


I thought these numbers should now be taken from the GT, so is this ESProducer still needed ? (I'm not thinking about the customisation per se, but about what we will actually integrate in the HLT menu)

My understanding is that, yes, these numbers are now taken from GT, but we still need to provide some "default" numbers to get the code function (somewhat historic). Perhaps something we can try to see if we can design the code differently.

When adding DB thresholds, my understanding was to maintain function with hardcoded parameters if needed. An example of the current implementation starts here:

cmssw/RecoParticleFlow/PFRecHitProducer/plugins/alpaka/PFRecHitProducerKernel.dev.cc

Line 62 in 82470f5

if (topology.cutsFromDB()) {

If there is need to change the functionality to DB only, I can look into it.

Thank you for the reminder. It's probably wise to keep the functionality to use numbers via config.

missirol · 2024-02-17T15:47:57Z

HLTrigger/Configuration/python/customizeHLTforAlpaka.py

+
+    process.hltLegacyPFClusterProducer = cms.EDProducer("LegacyPFClusterProducer",
+            src = cms.InputTag("hltPFClusterSoAProducer"),
+            pfClusterParams = cms.ESInputTag("pfClusterParamsESProducer:"),


Should this be hltPFClusterParamsESProducer: instead ? (I guess it does not matter, since the tests passed ? From a quick look, it seems that pfClusterParams is not really used inside LegacyPFClusterProducer)

Right, I guess we can directly drop this parameter from LegacyPFClusterProducer

First attempt to customize HLT for Alpaka based PF

1c678f2

cmsbuild added this to the CMSSW_14_1_X milestone Feb 15, 2024

cmsbuild added hlt-pending pending-signatures tests-pending orp-pending code-checks-pending labels Feb 15, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 15, 2024

waredjeb force-pushed the integratePFAlpakaClusteringAtHLT_14_0_0_pre3 branch from df9cb7f to 2b472df Compare February 15, 2024 08:12

cmsbuild added code-checks-pending and removed code-checks-approved labels Feb 15, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 15, 2024

cmsbuild added tests-started and removed tests-pending labels Feb 15, 2024

cmsbuild added tests-approved and removed tests-started labels Feb 15, 2024

cmsbuild mentioned this pull request Feb 15, 2024

Alpaka vs CUDA DQM compare modules for pixel tracks objects #43964

Closed

Martin-Grunewald reviewed Feb 15, 2024

View reviewed changes

cmsbuild added tests-pending code-checks-pending and removed code-checks-approved labels Feb 15, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 15, 2024

cmsbuild added tests-started and removed tests-pending labels Feb 15, 2024

cmsbuild added tests-approved and removed tests-started labels Feb 15, 2024

cmsbuild added hlt-approved fully-signed and removed hlt-pending pending-signatures labels Feb 16, 2024

cmsbuild added orp-approved and removed orp-pending labels Feb 16, 2024

cmsbuild merged commit 82470f5 into cms-sw:master Feb 16, 2024
29 checks passed

waredjeb mentioned this pull request Feb 16, 2024

Fix HLT customization for alpaka PF #43999

Merged

This was referenced Feb 17, 2024

cms.allowed now accepts a dict when assigning value #44002

Merged

[ROOTMaster] Updated root to tip of branch master cms-sw/cmsdist#8999

Merged

missirol reviewed Feb 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add customize to include Alpaka HCal PF Clustering at HLT #43971

Add customize to include Alpaka HCal PF Clustering at HLT #43971

waredjeb commented Feb 15, 2024

cmsbuild commented Feb 15, 2024 •

edited

Loading

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

mmusich commented Feb 15, 2024

mmusich commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

fwyzard commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

mmusich commented Feb 15, 2024

Martin-Grunewald Feb 15, 2024

waredjeb Feb 15, 2024

waredjeb Feb 15, 2024

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

mmusich commented Feb 15, 2024

waredjeb commented Feb 15, 2024 •

edited

Loading

mmusich commented Feb 15, 2024 •

edited

Loading

waredjeb commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

Martin-Grunewald commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

antoniovilela commented Feb 16, 2024

missirol left a comment

missirol Feb 17, 2024

hatakeyamak Feb 18, 2024 •

edited

Loading

jsamudio Feb 18, 2024

hatakeyamak Feb 18, 2024

missirol Feb 17, 2024

waredjeb Feb 19, 2024 •

edited

Loading

Add customize to include Alpaka HCal PF Clustering at HLT #43971

Add customize to include Alpaka HCal PF Clustering at HLT #43971

Conversation

waredjeb commented Feb 15, 2024

Validation

cmsbuild commented Feb 15, 2024 • edited Loading

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

mmusich commented Feb 15, 2024

mmusich commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

fwyzard commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

Comparison Summary

GPU Comparison Summary

mmusich commented Feb 15, 2024

Martin-Grunewald Feb 15, 2024

Choose a reason for hiding this comment

waredjeb Feb 15, 2024

Choose a reason for hiding this comment

waredjeb Feb 15, 2024

Choose a reason for hiding this comment

cmsbuild commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

mmusich commented Feb 15, 2024

waredjeb commented Feb 15, 2024 • edited Loading

mmusich commented Feb 15, 2024 • edited Loading

waredjeb commented Feb 15, 2024

cmsbuild commented Feb 15, 2024

Comparison Summary

GPU Comparison Summary

Martin-Grunewald commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

antoniovilela commented Feb 16, 2024

missirol left a comment

Choose a reason for hiding this comment

missirol Feb 17, 2024

Choose a reason for hiding this comment

hatakeyamak Feb 18, 2024 • edited Loading

Choose a reason for hiding this comment

jsamudio Feb 18, 2024

Choose a reason for hiding this comment

hatakeyamak Feb 18, 2024

Choose a reason for hiding this comment

missirol Feb 17, 2024

Choose a reason for hiding this comment

waredjeb Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

cmsbuild commented Feb 15, 2024 •

edited

Loading

waredjeb commented Feb 15, 2024 •

edited

Loading

mmusich commented Feb 15, 2024 •

edited

Loading

hatakeyamak Feb 18, 2024 •

edited

Loading

waredjeb Feb 19, 2024 •

edited

Loading