[13_0_X] Online DQM: replace the hard-coded output directory name of event display clients with an input argument, backport of 41986 and 42128 #42132

syuvivida · 2023-06-29T10:36:11Z

PR description:

This is a backport of PR 41986 and PR 42128.

We are working on the upgrade of online DQM machines [0][1]. There will be a few months that we share the same CMSSW code between the new and the current machines.
In the current (old) DQM machines, the disks of bu-c2f11-09-01 and bu-c2f11-13-01 are mounted on our fu machines as /fff/BU0. Event display clients visualization-live and visualization-live-secondInstance produce output root files at /fff/BU0/output.

However, the mount point (path) has changed in the new online DQM machines [1]. In order to use the same event display client codes for both old and new machines and also to make the path name more flexible, we replace the output path with an input argument (with a default value of /fff/BU0/output). The old machines will use an old hltd version and take the default value of the argument outputBaseDir, while the input values for the new machines will be determined by hltd and startDqmRun.sh.

[0] twiki about the upgrade of DQM machines
[1] JIRA ticket that includes the communication with DAQ
[2] JIRA ticket of the tests during TS1

PR validation:

This PR has been tested at lxplus by running the hlt, hcal, and ecal clients standalone with CMSSW_13_0_X_2023-06-27-1100 , CMSSW_13_1_X_2023-06-27-1100, and CMSSW_13_2_X_2023-06-26-2300 with the streamers at /eos/cms/store/group/comm_dqm/Collisions23_tempStreamers/.
This PR has been tested at the current (old) online DQM playback machines and ran all clients without problem when using the default value of the input argument.
This PR has been deployed/tested when we tested the data transfer during TS1 [2].

cmsbuild · 2023-06-29T10:36:34Z

A new Pull Request was created by @syuvivida for CMSSW_13_0_X.

It involves the following packages:

DQM/Integration (dqm)

@nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@threus, @batinkov, @francescobrivio this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Online DQM: add an input argument outputBaseDir in pbsource_cfi.py (to fit with the new hltd version) #42128

emanueleusai · 2023-07-03T04:52:13Z

please test

cmsbuild · 2023-07-03T05:57:58Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c31fb2/33508/summary.html
COMMIT: 9a024fe
CMSSW: CMSSW_13_0_X_2023-07-02-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/42132/33508/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10224.0

----- Begin Fatal Exception 03-Jul-2023 07:19:44 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 5 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module SiStripDigiToRawModule/'SiStripDigiToRaw'
   [5] Calling method for module MixingModule/'mix'
   [6] Calling RootInputFileSequence::initTheFile()
   [7] Calling StorageFactory::open()
   [8] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root
      [c] Input file root://xrootd-cms.infn.it//store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_12_0_0_pre4/RelValMinBias_13/GEN-SIM/113X_mc2017_realistic_v5-v1/00000/a21693e9-4d25-496a-96e2-c28232a7a712.root?tried=+1213llrxrd-redir.in2p3.fr,&xrdcl.requuid=64739e27-a3ab-4443-b1af-da4ebde1f1ce
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

4.294.29_RunMinBias2011B/step2_RunMinBias2011B.log
134.806134.806_RunMuonEG2015C/step2_RunMuonEG2015C.log
134.808134.808_RunSingleMuPrpt2015C/step2_RunSingleMuPrpt2015C.log

Expand to see more relval errors ...

perrotta · 2023-07-04T05:20:57Z

backport of #42128
(also #41986)

syuvivida · 2023-07-04T21:06:46Z

Hello, just to note that the test failure has nothing to do with this PR.

perrotta · 2023-07-04T21:16:51Z

please test

cmsbuild · 2023-07-05T01:29:30Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c31fb2/33550/summary.html
COMMIT: 9a024fe
CMSSW: CMSSW_13_0_X_2023-07-04-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/42132/33550/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 8 lines to the logs
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3317136
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3317111
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

emanueleusai · 2023-07-05T06:47:42Z

+1

cmsbuild · 2023-07-05T06:48:07Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_13_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-07-08T12:29:29Z

+1

add outputBaseDir input argument for online DQM

9a024fe

cmsbuild added this to the CMSSW_13_0_X milestone Jun 29, 2023

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending labels Jun 29, 2023

cmsbuild added tests-started and removed tests-pending labels Jul 3, 2023

cmsbuild added tests-rejected and removed tests-started labels Jul 3, 2023

cmsbuild added the backport label Jul 4, 2023

cmsbuild added tests-started and removed tests-rejected labels Jul 4, 2023

cmsbuild added tests-approved and removed tests-started labels Jul 5, 2023

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jul 5, 2023

cmsbuild added backport-ok and removed backport labels Jul 7, 2023

cmsbuild added orp-approved and removed orp-pending labels Jul 8, 2023

cmsbuild merged commit baf8989 into cms-sw:CMSSW_13_0_X Jul 8, 2023

This was referenced Aug 7, 2023

SiStripClusters2ApproxClusters: run peakFilter only if cluster is usable #42486

Merged

[13.2.X] SiStripClusters2ApproxClusters: run peakFilter only if cluster is usable #42516

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[13_0_X] Online DQM: replace the hard-coded output directory name of event display clients with an input argument, backport of 41986 and 42128 #42132

[13_0_X] Online DQM: replace the hard-coded output directory name of event display clients with an input argument, backport of 41986 and 42128 #42132

syuvivida commented Jun 29, 2023

cmsbuild commented Jun 29, 2023 •

edited

Loading

emanueleusai commented Jul 3, 2023

cmsbuild commented Jul 3, 2023

perrotta commented Jul 4, 2023

syuvivida commented Jul 4, 2023

perrotta commented Jul 4, 2023

cmsbuild commented Jul 5, 2023

emanueleusai commented Jul 5, 2023

cmsbuild commented Jul 5, 2023

perrotta commented Jul 8, 2023

[13_0_X] Online DQM: replace the hard-coded output directory name of event display clients with an input argument, backport of 41986 and 42128 #42132

[13_0_X] Online DQM: replace the hard-coded output directory name of event display clients with an input argument, backport of 41986 and 42128 #42132

Conversation

syuvivida commented Jun 29, 2023

PR description:

PR validation:

cmsbuild commented Jun 29, 2023 • edited Loading

emanueleusai commented Jul 3, 2023

cmsbuild commented Jul 3, 2023

RelVals

RelVals-INPUT

perrotta commented Jul 4, 2023

syuvivida commented Jul 4, 2023

perrotta commented Jul 4, 2023

cmsbuild commented Jul 5, 2023

Comparison Summary

emanueleusai commented Jul 5, 2023

cmsbuild commented Jul 5, 2023

perrotta commented Jul 8, 2023

cmsbuild commented Jun 29, 2023 •

edited

Loading