Fix of LumiList.py so to allow the merge of JSON files

[syuvivida]

PR description:

While merging the JSON files with mergeJSON.py, we ran into this error:
File "/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_0/bin/el9_amd64_gcc12/mergeJSON.py", line 44, in <module> finalList = finalList | localList File "/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_0/src/FWCore/PythonUtilities/python/LumiList.py", line 186, in __or__ runs = set(aruns + bruns) TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_keys'

as in Python 3.x, dict.keys doesn't return a list, but instead a view object, dict_keys. Therefore, this PR converted all the runs to a list to solve this error.

PR validation:

The PR has been validated by copying mergeJSON.py to a local area at lxplus and point to the local LumiList.py, with this PR, two JSONs could be merged without errors. Additionally, we also use this version of LumiList.py to launch cron jobs of DCS-only JSON files at lxplus in the central DQM-DC team. The cron jobs patch the new runs with good siStrip/pixel DCS status every day on top of the previous runs (using mergeJSON.py). These cron jobs have been launched 7 days ago and produced results without error.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44201/39183

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @syuvivida for master.

It involves the following packages:

• FWCore/PythonUtilities (core)

@smuzaffar, @Dr15Jones, @makortel, @cmsbuild can you please review it and eventually sign? Thanks.
@makortel, @wddgit, @missirol this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[makortel]

Could you change back the cases where the generator is sufficient (because it is more efficient to loop over than creating an intermediate list)?

[makortel]

Here the earlier use of the items() generator would work, and would be more efficient

Suggested change

	for run, lumis in list(runLumiList.items()):
	for run, lumis in runLumiList.items():

[makortel]

This would be better with the generator as well

Suggested change

	for run in list(runsAndLumis.keys()):
	for run in runsAndLumis.keys():

[makortel]

This too

Suggested change

	for run in list(compactList.keys()):
	for run in compactList.keys():

[makortel]

And here

Suggested change

	for run in list(self.compactList.keys()):
	for run in self.compactList.keys():

[makortel]

And here

Suggested change

	for run in list(self.compactList.keys()):
	for run in self.compactList.keys():

[syuvivida]

Hi @makortel
The latest commit has reverted all the previous changes when looping over runs back to the original LumiList.py. This commit has also been checked locally with mergeJSON.py and cron-job of DQM DCS-only JSON production.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44201/39268

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #44201 was updated. @smuzaffar, @cmsbuild, @makortel, @Dr15Jones can you please check and sign again.

[makortel]

Thanks! Could you squash the three commits into one in order to have simpler history?

[syuvivida]

Thanks! Could you squash the three commits into one in order to have simpler history?

Sorry does this mean I shall close this PR and create a new one?

[makortel]

Thanks! Could you squash the three commits into one in order to have simpler history?

Sorry does this mean I shall close this PR and create a new one?

Opening a new PR is not necessary. You can use

• git cms-squash-topic --current as instructed in https://cms-sw.github.io/tutorial-resolve-conflicts.html
• Follow https://cms-sw.github.io/faq.html#how-do-i-collapse-multiple-commits-into-one
• Use interactive rebase git rebase -i $CMSSW_VERSION
  • or if done outside of a CMSSE developer area, replace $CMSSW_VERSION with the tag/commit id of the base of your master branch

to collapse the commits to one, and then force-push (git push -f ...) to the same branch, and this PR will be updated.

I'd suggest to make a "backup" of your branch before doing anything else, e.g. git branch master_backup master just in case.

[syuvivida]

Thanks! Could you squash the three commits into one in order to have simpler history?

Sorry does this mean I shall close this PR and create a new one?

Opening a new PR is not necessary. You can use

* `git cms-squash-topic --current` as instructed in https://cms-sw.github.io/tutorial-resolve-conflicts.html

* Follow https://cms-sw.github.io/faq.html#how-do-i-collapse-multiple-commits-into-one

* Use interactive rebase `git rebase -i $CMSSW_VERSION`
  
  * or if done outside of a CMSSE developer area, replace `$CMSSW_VERSION` with the tag/commit id of the base of your `master` branch

to collapse the commits to one, and then force-push (git push -f ...) to the same branch, and this PR will be updated.

I'd suggest to make a "backup" of your branch before doing anything else, e.g. git branch master_backup master just in case.

Thanks for the tips! Now the 3 commits are merged into one.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44201/39288

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #44201 was updated. @makortel, @Dr15Jones, @cmsbuild, @smuzaffar can you please check and sign again.

[makortel]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9f33c7/37823/summary.html
COMMIT: 8c6369e
CMSSW: CMSSW_14_1_X_2024-02-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44201/37823/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 72 lines from the logs
• Reco comparison results: 56 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3338784
• DQMHistoTests: Total failures: 9
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3338753
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[makortel]

Comparisons show the usual

• Non reproducibility in wf 136.793 #43293
• Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999 #43790

[makortel]

+core

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1