[Do not merge] Testing TF 2.15 GPU

[smuzaffar]

No description provided.

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_2_X/tf.

@aandvalenzuela, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

test parameters:

• enable = gpu
• addpkg = RecoTracker/PixelTrackFitting,PhysicsTools/TensorFlow

[smuzaffar]

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

[smuzaffar]

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

[cmsbuild]

-1

Failed Tests: GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7f8188/41310/summary.html
COMMIT: 0fd2549
CMSSW: CMSSW_14_2_TF_X_2024-09-02-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9394/41310/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @matt2275 Fix issue with ZDCSimpleRecAlgo_Run3.cc cmssw#45868
• @stahlleiton Add option to sort candidates by pT in Unified Particle Transformer cmssw#45843
• @jfernan2 [DT] Position fix and new filter to TPs from Phase2 L1T emulator cmssw#44924
• @fwyzard Implement a test for the copyFromDense and copyToDense utilities cmssw#45852
• @bsunanda Phase2-hgx359X Modify the testing code in several packages in view of changes made in the Configuration/Geometry area for Phase2 scenarios cmssw#45822
• @namapane Minor fixes to variables for lepton MVA cmssw#45860
• @AdrianoDee Reduce RelVal Events for B0ToJpsiK0s_JMM_Filter_DGamma0 cmssw#45798
• @nurfikri89 [BTV, NanoAOD] Add UParT discriminants for strange-jet tagging cmssw#45684
• @tahuang1991 Fixed CLCT sorting in CSC TP emulator and synchronize the config to the Point-5 data-taking settings cmssw#45829
• @mmusich add few more default geometry tests in Geometry/TrackerGeometryBuilder cmssw#45866
• @pallabidas [Phase 2 L1T] Updates to GCT objects in HF cmssw#45540
• @iarspider [GCC13][AlCa] Avoid Wdangling-reference in SiPhase2OuterTrackerFakeLorentzAngleESSource cmssw#45730
• @mmusich Add certificates handling to G2GainsValidator.py cmssw#45867
• @mmusich TrackingRecHitsSoACollection: early return hostData in CopyHost::copyAsync() when there aren't hits cmssw#45837
• @bsunanda Phase2-hgx359Y Update the scripts in Alignment/OfflineValidation and Geometry/HcalCommonData|MTDCommonData for the changes in Configuration/Geometry cmssw#45848
• @mmusich improve pagination of L1TUtmTriggerMenuPayloadInspector plots cmssw#45813
• @makortel Add possibility for a CopyToHost::postCopy() operation cmssw#45801
• @cms-sw [LLVM] Update to version 18.1.6 #9386
• @cms-sw cmssw-osenv: added extra paths to mount; set LC_ALL=C #9389

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7f8188/41310/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7f8188/41310/git-merge-result

GPU Unit Tests

I found 9 errors in the following unit tests:

---> test testTFConstSessionCUDA had ERRORS
---> test testTFHelloWorldCUDA had ERRORS
---> test testTFGraphLoadingCUDA had ERRORS
and more ...

Comparison Summary

Summary:

• You potentially added 608 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 2446 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3328276
• DQMHistoTests: Total failures: 34988
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3293268
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 36.092000000000006 KiB( 43 files compared)
• DQMHistoSizes: changed ( 11634.0,... ): 2.879 KiB Physics/NanoAODDQM
• DQMHistoSizes: changed ( 13234.0,... ): 1.771 KiB Physics/NanoAODDQM
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: found differences in 2 / 42 workflows