Extremely long startup time when loading dictionaries with pyroot

[MarkusFrankATcernch]

Check duplicate issues.

• Checked for duplicates

Description

In the presence of LD_LIBRARY_PATH containing directories with many entries (typically when using the LCG view),
it takes a very, very long time to load a dictionary using pyroot. After some initial debugging with Axel he admitted
that there is something going wrong when loading dictionaries. The expired time is way too long.

The observed behaviour is not limited to the reproducer below, but also occurs on standalone installations.

Reproducer

1. Use a LCG view on cvmfs

. /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc12-dbg/setup.sh

2. Load a dictionary from pyroot:

$ python
Python 3.9.12 (main, Jul 31 2023, 15:05:19) 
[GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.ctime(); import dd4hep; time.ctime()
'Wed Dec 13 09:15:34 2023'
'Wed Dec 13 09:16:27 2023'

ROOT version

Standard root vervion of LCG 103 on cvmfs:

$ which root
/cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc12-dbg/bin/root
$ /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc12-dbg/bin/root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.28/04                        https://root.cern |
  | (c) 1995-2022, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on May 08 2023, 02:44:07                 |
  | From tags/v6-28-04@v6-28-04                                      |
  | With g++ (GCC) 12.1.0                                            |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

Installation method

Pre-build, pre-installed on cvmfs

Operating system

Linux lxplus940.cern.ch 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Additional context

No response

[dpiparo]

Hi @MarkusFrankATcernch ,

I am sorry you experienced this issue.
We know that importing ROOT using LCG releases, perhaps on lxplus nodes can be slow. For this reason, we implemented a first pack of optimisations in PyROOT for the ROOT release 6.30. Unfortunately, this is not available yet in any LCG stack.
Is it easy for you to perform the same measurement with a dev3 lcg stack? That would allow you to use ROOT master and benefit from the latest optimisations.
I know this might not be a solution to the problem you report, but it might be a start.

[MarkusFrankATcernch]

Hi @dpiparo,

the problem is not the top-short-term fix, but the longer term. I started to see this behavior in summer and reported it to Axel, when I could not use the LCG views anymore to build dd4hep. Unfortunately Axel was highly busy since and I am glad you now take care!

However, now I also start seeing the behavior also in the LHCb-online nightly builds, because tests sometimes tend to fail, because images activated from python using pyROOT simetimes take several 100 seconds (I have set the timeout now to 300 seconds) to start up. This is not really good anymore given that the dictionaries for these apps are far from gigantic and starting (during the same test run) the same app using some kind of ini files does not at all suffer from this problem.

Tell me if I should test using a private build with root master.

Markus

[pcanal]

@MarkusFrankATcernch There was significant progress in this area recently in the master. Can you check if you are still seeing this?

[MarkusFrankATcernch]

@pcanal From what I see in the nightly builds, things have significantly improved in the dev slot. This kind of timeout seems to have disappeared. Hence, for me this looks fine.
I think this issue may be closed.

[dpiparo]

Thanks for your help, @MarkusFrankATcernch