Update some HLT tests for python3 #34563

davidlange6 · 2021-07-20T13:07:12Z

HLTrigger/Configuration/test/runAll.csh Fake appears to work after these changes.

cmsbuild · 2021-07-20T13:16:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34563/24099

This PR adds an extra 36KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/ParameterSet/scripts/edmConfigDump modified in PR(s): Used importlib in scripts #34490
- File HLTrigger/Configuration/python/Tools/confdb.py modified in PR(s): Remove six python2->3 compatibility helper usage #34534

cmsbuild · 2021-07-20T13:16:36Z

A new Pull Request was created by @davidlange6 (David Lange) for master.

It involves the following packages:

FWCore/ParameterSet (core)
HLTrigger/Configuration (hlt)

@Martin-Grunewald, @smuzaffar, @cmsbuild, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

davidlange6 · 2021-07-20T13:18:26Z

please test
@Martin-Grunewald - do the relevant HLT tests run in the PR tests?

cmsbuild · 2021-07-20T13:26:27Z

-1

Failed Tests: Build HeaderConsistency ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17f73d/17019/summary.html
COMMIT: 591cfe3
CMSSW: CMSSW_12_0_X_2021-07-19-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34563/17019/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17f73d/17019/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17f73d/17019/git-merge-result

Build

I found compilation error when building:

>> Leaving Package HeterogeneousCore/SonicTriton
>> Package HeterogeneousCore/SonicTriton built
Entering library rule at src/HeterogeneousCore/SonicTriton/plugins
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc:3:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/interface/TritonService.h:13:10: fatal error: grpc_client.h: No such file or directory
   13 | #include "grpc_client.h"
      |          ^~~~~~~~~~~~~~~
compilation terminated.
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc:3:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/interface/TritonService.h:13:10: fatal error: grpc_client.h: No such file or directory

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicCore/test/SonicDummyOneAnalyzer.cc
>> Entering Package HeterogeneousCore/SonicTriton
Entering library rule at src/HeterogeneousCore/SonicTriton/plugins
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc:3:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/interface/TritonService.h:13:10: fatal error: 'grpc_client.h' file not found
#include "grpc_client.h"
         ^~~~~~~~~~~~~~~
1 error generated.
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/plugins/TritonService.cc:3:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-19-1100/src/HeterogeneousCore/SonicTriton/interface/TritonService.h:13:10: fatal error: 'grpc_client.h' file not found

Martin-Grunewald · 2021-07-20T13:30:40Z

Unfortunately the (failing) tests are not run in PR tests, but in IB tests.

Martin-Grunewald · 2021-07-20T13:36:10Z

Including this PR in CMSSW_12_0_X_2021-07-19-2300, there appears the following SyntaxWarning:

src/HLTrigger/Configuration/python/Tools/options.py:135: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if name is 'menu' and not isinstance(value, ConnectionHLTMenu):
src/HLTrigger/Configuration/python/Tools/options.py:138: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'l1' and not isinstance(value, ConnectionL1TMenu):
src/HLTrigger/Configuration/python/Tools/options.py:141: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'l1Xml' and not isinstance(value, ConnectionL1TMenuXml):
src/HLTrigger/Configuration/python/Tools/options.py:144: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'open' and value:
src/HLTrigger/Configuration/python/Tools/options.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'prescale' and value is not None:
src/HLTrigger/Configuration/python/Tools/options.py:152: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'profiling' and value:
src/HLTrigger/Configuration/python/Tools/options.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif name is 'timing' and value:
>> All python modules compiled

davidlange6 · 2021-07-20T13:41:24Z

Right - Those are pre-existing bugs (most likely anyway) that python3 now shows you (instead which python2 ignored them)

…

On Jul 20, 2021, at 3:36 PM, Martin Grunewald ***@***.***> wrote: Including this PR in CMSSW_12_0_X_2021-07-19-2300, there appears the following SyntaxWarning: src/HLTrigger/Configuration/python/Tools/options.py:135: SyntaxWarning: "is" with a literal. Did you mean "=="? if name is 'menu' and not isinstance(value, ConnectionHLTMenu): src/HLTrigger/Configuration/python/Tools/options.py:138: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'l1' and not isinstance(value, ConnectionL1TMenu): src/HLTrigger/Configuration/python/Tools/options.py:141: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'l1Xml' and not isinstance(value, ConnectionL1TMenuXml): src/HLTrigger/Configuration/python/Tools/options.py:144: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'open' and value: src/HLTrigger/Configuration/python/Tools/options.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'prescale' and value is not None: src/HLTrigger/Configuration/python/Tools/options.py:152: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'profiling' and value: src/HLTrigger/Configuration/python/Tools/options.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="? elif name is 'timing' and value: >> All python modules compiled — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

makortel · 2021-07-20T14:08:04Z

FWCore/ParameterSet/scripts/edmConfigDump

+loader=SourceFileLoader("pycfg", options.filename)
+loaded = types.ModuleType("pycfg",loader.name)
+loader.exec_module(loaded)
+#cfo = imp.load_source("pycfg", options.filename, handle)


Is the commented out line still useful? It seems to me that handle becomes now unnecessary.

davidlange6 · 2021-07-20T14:25:55Z

On Jul 20, 2021, at 4:08 PM, Matti Kortelainen ***@***.***> wrote: @makortel commented on this pull request. In FWCore/ParameterSet/scripts/edmConfigDump: > @@ -19,7 +20,10 @@ parser.add_argument("--pruneVerbose", action="store_true", options = parser.parse_args() handle = open(options.filename, 'r') -cfo = imp.load_source("pycfg", options.filename, handle) +loader=SourceFileLoader("pycfg", options.filename) +loaded = types.ModuleType("pycfg",loader.name) +loader.exec_module(loaded) +#cfo = imp.load_source("pycfg", options.filename, handle) Is the commented out line still useful? It seems to me that handle becomes now unnecessary.

Yes that can be cleaned up. Thanks for noticing that

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Martin-Grunewald · 2021-07-20T14:48:57Z

FWCore/ParameterSet/scripts/edmConfigDump

+loader=SourceFileLoader("pycfg", options.filename)
+loaded = types.ModuleType("pycfg",loader.name)
+loader.exec_module(loaded)
+#cfo = imp.load_source("pycfg", options.filename, handle)
 cmsProcess = cfo.process


This line crashes as cfo is no longer defined.

davidlange6 · 2021-07-20T14:54:24Z

Ha - that is obviously correct. What test gave you that error?

…

On Jul 20, 2021, at 4:50 PM, Martin Grunewald ***@***.***> wrote: @Martin-Grunewald commented on this pull request. In FWCore/ParameterSet/scripts/edmConfigDump: > @@ -19,7 +20,10 @@ parser.add_argument("--pruneVerbose", action="store_true", options = parser.parse_args() handle = open(options.filename, 'r') -cfo = imp.load_source("pycfg", options.filename, handle) +loader=SourceFileLoader("pycfg", options.filename) +loaded = types.ModuleType("pycfg",loader.name) +loader.exec_module(loaded) +#cfo = imp.load_source("pycfg", options.filename, handle) cmsProcess = cfo.process This line crashes as cfo is no longer defined. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Martin-Grunewald · 2021-07-20T14:55:37Z

runAll.csh Fake
Even though the script says exit code 0, the log file of the HLT_Integration test run as a part of it displays the error.

cmsbuild · 2021-07-20T17:04:17Z

Pull request #34563 was updated. @Martin-Grunewald, @smuzaffar, @cmsbuild, @Dr15Jones, @makortel can you please check and sign again.

makortel · 2021-07-20T17:10:40Z

@cmsbuild, please test

cmsbuild · 2021-07-21T04:10:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17f73d/17026/summary.html
COMMIT: 3153094
CMSSW: CMSSW_12_0_X_2021-07-20-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34563/17026/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2996268
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2996245
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

makortel · 2021-07-21T12:57:02Z

+1

Martin-Grunewald · 2021-07-21T13:50:26Z

urgent

Martin-Grunewald · 2021-07-21T13:50:32Z

+1

cmsbuild · 2021-07-21T13:51:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-07-21T14:55:53Z

+1

Update of the functions "getHltConfiguration" and "loadHltConfiguration" defined in HLTrigger/Configuration/python/Utilities.py - remove unnecessary dependencies on external packages like "types" - fix "getHltConfiguration" (was broken since the changes in cms-sw#34563) - return cms.Process object in "loadHltConfiguration" (backward-compatible change) - add checks in both functions to catch errors in ConfDB queries

davidlange6 added 4 commits July 20, 2021 11:33

python3 cleaning

79afc5c

python3 compatibility

3470b00

python3 compatibility

fc92a71

python3 compatibility

591cfe3

cmsbuild added this to the CMSSW_12_0_X milestone Jul 20, 2021

cmsbuild added code-checks-pending core-pending hlt-pending orp-pending pending-signatures tests-pending labels Jul 20, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 20, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 20, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 20, 2021

cmsbuild mentioned this pull request Jul 20, 2021

Remove six python2->3 compatibility helper usage #34534

Merged

makortel reviewed Jul 20, 2021

View reviewed changes

Martin-Grunewald reviewed Jul 20, 2021

View reviewed changes

cmsbuild added the code-checks-approved label Jul 20, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 20, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 21, 2021

cmsbuild added core-approved and removed core-pending labels Jul 21, 2021

cmsbuild added fully-signed hlt-approved urgent and removed hlt-pending pending-signatures labels Jul 21, 2021

cmsbuild added orp-approved and removed orp-pending labels Jul 21, 2021

cmsbuild merged commit 968e15f into cms-sw:master Jul 21, 2021

This was referenced Jul 21, 2021

scram-tools repo to provide scram toolfile definitions cms-sw/cmsdist#7147

Merged

support global::OutputModule<ExternalWork> #34579

Merged

Remove EventSetup use from PdtEntry #34581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update some HLT tests for python3 #34563

Update some HLT tests for python3 #34563

davidlange6 commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

davidlange6 commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

Martin-Grunewald commented Jul 20, 2021

Martin-Grunewald commented Jul 20, 2021

davidlange6 commented Jul 20, 2021 via email

makortel Jul 20, 2021

davidlange6 commented Jul 20, 2021 via email

Martin-Grunewald Jul 20, 2021

davidlange6 commented Jul 20, 2021 via email

Martin-Grunewald commented Jul 20, 2021 •

edited

Loading

cmsbuild commented Jul 20, 2021

makortel commented Jul 20, 2021

cmsbuild commented Jul 21, 2021

makortel commented Jul 21, 2021

Martin-Grunewald commented Jul 21, 2021

Martin-Grunewald commented Jul 21, 2021

cmsbuild commented Jul 21, 2021

qliphy commented Jul 21, 2021

Update some HLT tests for python3 #34563

Update some HLT tests for python3 #34563

Conversation

davidlange6 commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

davidlange6 commented Jul 20, 2021

cmsbuild commented Jul 20, 2021

Build

Clang Build

Martin-Grunewald commented Jul 20, 2021

Martin-Grunewald commented Jul 20, 2021

davidlange6 commented Jul 20, 2021 via email

makortel Jul 20, 2021

Choose a reason for hiding this comment

davidlange6 commented Jul 20, 2021 via email

Martin-Grunewald Jul 20, 2021

Choose a reason for hiding this comment

davidlange6 commented Jul 20, 2021 via email

Martin-Grunewald commented Jul 20, 2021 • edited Loading

cmsbuild commented Jul 20, 2021

makortel commented Jul 20, 2021

cmsbuild commented Jul 21, 2021

Comparison Summary

makortel commented Jul 21, 2021

Martin-Grunewald commented Jul 21, 2021

Martin-Grunewald commented Jul 21, 2021

cmsbuild commented Jul 21, 2021

qliphy commented Jul 21, 2021

Martin-Grunewald commented Jul 20, 2021 •

edited

Loading