[LTO] Improve link time by using multiple process for linking #8840

smuzaffar · 2023-11-28T16:08:02Z

This should avoid the warnings like [a] and also it should use the GNU Make job server[b] to run multiple processes for at link step

[a]

lto-wrapper: warning: using serial compilation of 6 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information

[b]

Use -flto=auto to use GNU make’s job server, if available, or otherwise fall back to autodetection of the number of CPU threads present in your system.

cmsbuild · 2023-11-28T16:08:26Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_14_0_X/master.

@cmsbuild, @iarspider, @smuzaffar, @aandvalenzuela can you please review it and eventually sign? Thanks.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.
cms-bot commands are listed here

smuzaffar · 2023-11-28T16:08:57Z

test parameters:

full_cmssw= true

smuzaffar · 2023-11-28T16:09:29Z

please test

smuzaffar · 2023-11-28T16:11:34Z

please test for CMSSW_14_0_CLANG_X

cmsbuild · 2023-11-28T22:36:56Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36131/summary.html
COMMIT: 0fb4e3a
CMSSW: CMSSW_14_0_X_2023-11-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8840/36131/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36131/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36131/git-merge-result

Comparison Summary

Summary:

You potentially added 94 lines to the logs
Reco comparison results: 2478 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3367918
DQMHistoTests: Total failures: 3836
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3364060
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: found differences in 1 / 48 workflows

fwyzard · 2023-11-28T22:42:31Z

This does silence the warning, but is there a way to check if the LTO step is actually running in parallel, using slots from the Make job server ?

cmsbuild · 2023-11-29T01:47:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36132/summary.html
COMMIT: 0fb4e3a
CMSSW: CMSSW_14_0_CLANG_X_2023-11-27-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8840/36132/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36132/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36132/git-merge-result

Comparison Summary

Summary:

You potentially added 9816 lines to the logs
Reco comparison results: 31137 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3367918
DQMHistoTests: Total failures: 59309
DQMHistoTests: Total nulls: 224
DQMHistoTests: Total successes: 3308363
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.019 KiB( 49 files compared)
DQMHistoSizes: changed ( 141.044 ): -0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 4.53 ): 0.004 KiB JetMET/SUSYDQM
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: found differences in 6 / 48 workflows

smuzaffar · 2023-11-29T07:05:24Z

please test

smuzaffar · 2023-11-29T07:52:00Z

This does silence the warning, but is there a way to check if the LTO step is actually running in parallel, using slots from the Make job server ?

Yes @fwyzard , it does run in parallel e.g. I tested it to build big Simulation plugin and I see the processes like [a] at link time. so basically it generates a temp makefile and then call make -jN . The link time for big simulation plugin dropped to 1m45s from 9m45. For building full release one might not see any build time improvements as during release build our cpu utilization is already nearly 100% (make already running nproc jobs in parallel).

[a]

 5308 08:23  0.5  \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../libexec/gcc/x86_64-redhat-linux-gnu/12.3.1/lto-wrappe
 5476 08:24  0.2    \_ make -f /tmp/muzaffar/ccRFEigZ.mk -j16 all
 6091 08:24  0.0    \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -xlto -c -fno-openmp -fno-openacc -fcf-prote
 6092 08:24 96.7    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/../../libexec/gcc/x86_64-redhat-l
 6093 08:24  2.2    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.3.1/..
 6099 08:24  0.0    \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -xlto -c -fno-openmp -fno-openacc -fcf-prote
 6101 08:24 94.5    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/../../libexec/gcc/x86_64-redhat-l
 6106 08:24  2.2    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.3.1/..
 6102 08:24  0.0    \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -xlto -c -fno-openmp -fno-openacc -fcf-prote
 6107 08:24 90.0    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/../../libexec/gcc/x86_64-redhat-l
 6116 08:24  2.5    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.3.1/..
 6105 08:24  0.0    \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -xlto -c -fno-openmp -fno-openacc -fcf-prote
 6108 08:24 93.7    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/../../libexec/gcc/x86_64-redhat-l
 6117 08:24  2.5    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.3.1/..
 6118 08:24  0.0    \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -xlto -c -fno-openmp -fno-openacc -fcf-prote
 6120 08:24 88.5    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/../../libexec/gcc/x86_64-redhat-l
 6121 08:24  2.5    |   \_ /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02813/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.3.1/..

fwyzard · 2023-11-29T08:22:48Z

Nice.
Does it reuse the parallel slots from the parent make process, or does it always run in parallel on its own ?

smuzaffar · 2023-11-29T08:32:30Z

doc says that -flto=auto to use GNU make’s job server, so I am assuming the it should use slots from parent gmake but let me test it

fwyzard · 2023-11-29T08:35:50Z

Mhm, in the description of the PR you've written (emphasis mine)

Use -flto=auto to use GNU make’s job server, if available, or otherwise fall back to autodetection of the number of CPU threads present in your system.

smuzaffar · 2023-11-29T09:38:32Z

as scram uses gmake so for cmssw the gmake job server will be available but let me double check

smuzaffar · 2023-11-29T10:33:49Z

@fwyzard , right, so with existing build rules -flto=auto is not using gmake job server e.g. running scram b -j4 still shows that lto-wrapper is running make -j16 (on 16 core machine) jobs. I need to update build rules to explicitly prefix link command with + (https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html#MAKE-Variable) and then it uses our make job server

cmsbuild · 2023-11-29T10:48:06Z

Pull request #8840 was updated.

smuzaffar · 2023-11-29T10:48:14Z

please test

smuzaffar · 2023-11-29T14:29:02Z

Some build timing on 16 core build nodes

-flto i.e current default for CMSSW: 2h 38mins
-flto=auto without using scram's gmake job server: 2h 35m
- Each link process can start nproc jobs and can overload system
-flto=auto with properly using scram's gmake job server: 2h 40m

cmsbuild · 2023-11-29T16:52:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36154/summary.html
COMMIT: 7344e89
CMSSW: CMSSW_14_0_X_2023-11-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8840/36154/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@peteryouBuffalo UPDATED: Adding of jet purity and efficiency plots to PFValidation package cmssw#43304

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36154/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-223d81/36154/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 23 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3367918
DQMHistoTests: Total failures: 1568
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3366328
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 802.5509999999997 KiB( 49 files compared)
DQMHistoSizes: changed ( 10024.0,... ): 24.809 KiB ParticleFlow/JetResponse
DQMHistoSizes: changed ( 10024.0,... ): 0.812 KiB ParticleFlow/Offset
DQMHistoSizes: changed ( 25.0,... ): 10.494 KiB ParticleFlow/JetResponse
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

smuzaffar · 2023-11-29T17:01:15Z

+externals

comparison differences are due to additional cms-sw/cmssw#43304

cmsbuild · 2023-11-29T17:01:41Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_0_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar added 2 commits November 28, 2023 17:05

[LTO] Improve link time by using 2 process for linking

bf1a205

Update compilation_flags_lto.file

0fb4e3a

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Nov 28, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 28, 2023

cmsbuild added tests-pending and removed tests-started labels Nov 28, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 28, 2023

cmsbuild added tests-approved and removed tests-started labels Nov 28, 2023

smuzaffar added a commit that referenced this pull request Nov 29, 2023

[do not merge] Ref for lto build time improvements in #8840

4ffb922

cmsbuild added tests-started and removed tests-approved labels Nov 29, 2023

enable make job server for link commands if lto is enabled

7344e89

cmsbuild added tests-pending and removed tests-started labels Nov 29, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 29, 2023

cmsbuild added tests-approved and removed tests-started labels Nov 29, 2023

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Nov 29, 2023

smuzaffar merged commit 539f3b8 into IB/CMSSW_14_0_X/master Nov 30, 2023
10 checks passed

smuzaffar deleted the smuzaffar-patch-3 branch December 18, 2023 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LTO] Improve link time by using multiple process for linking #8840

[LTO] Improve link time by using multiple process for linking #8840

smuzaffar commented Nov 28, 2023 •

edited

Loading

cmsbuild commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

cmsbuild commented Nov 28, 2023

fwyzard commented Nov 28, 2023

cmsbuild commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

smuzaffar commented Nov 29, 2023 •

edited

Loading

fwyzard commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

fwyzard commented Nov 29, 2023

smuzaffar commented Nov 29, 2023 •

edited

Loading

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

[LTO] Improve link time by using multiple process for linking #8840

[LTO] Improve link time by using multiple process for linking #8840

Conversation

smuzaffar commented Nov 28, 2023 • edited Loading

cmsbuild commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

smuzaffar commented Nov 28, 2023

cmsbuild commented Nov 28, 2023

Comparison Summary

fwyzard commented Nov 28, 2023

cmsbuild commented Nov 29, 2023

Comparison Summary

smuzaffar commented Nov 29, 2023

smuzaffar commented Nov 29, 2023 • edited Loading

fwyzard commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

fwyzard commented Nov 29, 2023

smuzaffar commented Nov 29, 2023 • edited Loading

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

Comparison Summary

smuzaffar commented Nov 29, 2023

cmsbuild commented Nov 29, 2023

smuzaffar commented Nov 28, 2023 •

edited

Loading

smuzaffar commented Nov 29, 2023 •

edited

Loading

smuzaffar commented Nov 29, 2023 •

edited

Loading