Add tables for map version 160812 for 3 and 3.8 T

[namapane]

For future field maps. No change in existing functionality or current default behavior.
cf: https://indico.cern.ch/event/642366/contributions/2606389/

[cmsbuild]

A new Pull Request was created by @namapane (Nicola Amapane) for branch master.

@cmsbuild, @smuzaffar, @mrodozov, @iahmad-khan, @davidlange6 can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

external issue cms-sw/cmsdist#3094

[davidlange6]

I belatedly notice - 22k files (11k per map?).. can we not condense these somewhat?

[davidlange6]

ok, i guess this is continuing an old design...

find /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/data-MagneticField-Interpolation/V01-00-00/MagneticField/Interpolation/data/grid_130503_3_8t_v9_small | wc
9662 9662 1555410

[smuzaffar]

right, currently in tag V01-00-00 we have 48K files with total size of 380MB

[namapane]

These are for different map, ie for different nominal current, and separately for RunI/RunII. There are also different version with more and more refined models, and eventually these new ones we are preparing now will replace some of the old ones. If we badly need to reduce the overall number of files, we could remove two sets that are not strictly needed (grid_1103l_071212_4t and grid_120812_3_8t_v7_small). Regarding condensing: a possibility that has been considered in the past is to move the whole amount of data to the conditions db; this would require some work and some performance studies for which manpower is unfortunatley quite short. Cheers, Nicola

…

On 13-Jun-17 10:11, Malik Shahzad Muzaffar wrote: right, current in tag V01-00-00 we have 48K files with total size of 380MB — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE7eUJMSBFKSBT7wu87djNHM1i6gDs2Vks5sDkQcgaJpZM4Ny4ks>.

[namapane]

I picked a recent nightly build and I find this under $CMSSW_DATA_PATH/data-MagneticField-Interpolation/V00-00-01/... and it works.
However, I also see that $CMSSW_DATA_PATH/data-MagneticField-Interpolation/V00-00-00/... exists, which is also populated. What is the reason for keeping both? In this way a good fraction of this large number of files is distributed twice, which does not seem to make a lot of sense.

[slava77]

On 6/23/17 9:32 AM, Nicola Amapane wrote: I picked a recent nightly build and I find this under $CMSSW_DATA_PATH/data-MagneticField-Interpolation/V00-00-01/... and it works. However, I also see that $CMSSW_DATA_PATH/data-MagneticField-Interpolation/V00-00-00/... exists, which is also populated. What is the reason for keeping both? In this way a good fraction of this large number of files is distributed twice, which does not seem to make a lot of sense.

the same area is used for distributions of all supported past releases.

…

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdcbpX-3dIjZVn-ZdVm_IyO-5NJW4dZks5sG-i3gaJpZM4Ny4ks>.

[namapane]

Yes but in this case V00-00-01 contains exactly the same files of V00-00-00, plus two sets in addition which do not interfere with supported past releases. There is really no point in duplicating these 48k files, couldn't we have added the new files and created a new set only when we really break backward compatibility (ie remove old sets that are still used in supported past releases).

Moreover, I'm probably missing something but how does a recent CMSSW version know that it should look at V00-00-01 and not at V00-00-00? Both are found under CMSSW_DATA_PATH.

In any case, Slava Kl has prepared the last two map sets much earlier than expected, and I would like to add these as well, will this result in a V00-00-02 and yet another copy of all the files, or can we update (or drop) V00-00-01 given it did not go to an actual release yet?

[slava77]

On 6/23/17 2:29 PM, Nicola Amapane wrote: Yes but in this case V00-00-01 contains exactly the same files of V00-00-00, plus two sets in addition which do not interfere with supported past releases. There is really no point in duplicating these 48k files, couldn't we have added the new files and created a new set only when we really break backward compatibility (ie remove old sets that are still used in supported past releases). Moreover, I'm probably missing something but how does a recent CMSSW version know that it should look at V00-00-01 and not at V00-00-00? Both are found under CMSSW_DATA_PATH.

Applications use FileInPath and look for relevant package data in $CMSSW_SEARCH_PATH which contains, in particular (expanded) $CMSSW_RELEASE_BASE/external/$SCRAM_ARCH/data so they will find MagneticField/Interpolation/ in $CMSSW_RELEASE_BASE/external/$SCRAM_ARCH/data/MagneticField/Interpolation/data applications do not look in $CMSSW_DATA_PATH

In any case, Slava Kl has prepared the last two map sets much earlier than expected, and I would like to add these as well, will this result in a V00-00-02 and yet another copy of all the files, or can we update (or drop) V00-00-01 given it did not go to an actual release yet?

A new version will correspond to another copy. There is no practical way to avoid copies at this level: basic distribution pattern that breaks the idea of cleanup in old area is when a new version of the data external has the same file names but different content of the files. For the pattern with the new package always adding new files and not modifying the old ones it is indeed more practical to overwrite old locations, but technically this would require re-release and re-installation of old releases.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdcbg3oTsNlIoPjMarDueH-J2irMiaOks5sHC4vgaJpZM4Ny4ks>.

[slava77]

the incremental solution would be to instead distribute every field map version subdirectory as a separate external

[namapane]

What do you prefer? I am now ready to make a PR for the the last two table folders of this map set. If you are not concerned about this duplication I would go on in the usual way.

…

On 24-Jun-17 1:13, Slava Krutelyov wrote: the incremental solution would be to instead distribute every field map version subdirectory as a separate external — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE7eUEO9rmimuo9OhRY9HrYv9K4dse_pks5sHEakgaJpZM4Ny4ks>.

[slava77]

It would be good to get some feedback here from David and Shahzad.
@davidlange6 @smuzaffar

[davidlange6]

Is there another iteration after this one expected? (Indeed I thought the last iteration was the last ...) Cheers, David On 27 Jun 2017, at 20:10, Slava Krutelyov <notifications@github.com<mailto:notifications@github.com>> wrote: It would be good to get some feedback here from David and Shahzad. @davidlange6<https://github.com/davidlange6> @smuzaffar<https://github.com/smuzaffar> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEzywwC1x6MgZCnWqdUHctS3KiXaXjhMks5sIUVngaJpZM4Ny4ks>.

[smuzaffar]

@namapane, as @slava77 mentioned, if we want to avoid duplication then we have to bundle these maps in separate externals which means you have to update cmssw code too. How much more data you would like to include ? Note that data-MagneticField-Interpolation github repo is nearly 500MB now and there is 1GB size limitation on github repos.

[namapane]

@davidlange6, These would be the last two sets of tables for this map set. (I actually did not expect them to be ready in time for 93X and that's why I made a PR for the first two) @smuzaffar: the additional data is about 170 MB, so we are not hitting the quota anytime soon. Also, once these maps will become the default (hopefully in 93X) it will be possible to remove the following sets which will be by then obsolete: grid_1103l_071212_3t grid_1103l_071212_3_5t grid_120812_3_8t_v7_large grid_120812_3_8t_v7_small grid_130503_3_5t_v9_large grid_130503_3_8t_v9_large grid_130503_3_8t_v9_small (though I don't know if removal really frees quota given that files remain in the github history?) BTW, in fact one of these maps sets (grid_120812_3_8t_v7_small) can be removed right away as it has been superseded before being adopted in any release. Any objections in removing it? Anyhow, I have a mild preference not to separate this in different externals for the time being, unless you have strong opinions. Cheers, Nicola

…

On 27-Jun-17 21:27, David Lange wrote: Is there another iteration after this one expected? (Indeed I thought the last iteration was the last ...) Cheers, David On 27 Jun 2017, at 20:10, Slava Krutelyov ***@***.******@***.***>> wrote: It would be good to get some feedback here from David and Shahzad. @davidlange6<https://github.com/davidlange6> @smuzaffar<https://github.com/smuzaffar> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEzywwC1x6MgZCnWqdUHctS3KiXaXjhMks5sIUVngaJpZM4Ny4ks>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE7eUJ_M17F76pjDAd29DJxM2fjfNZWeks5sIVeNgaJpZM4Ny4ks>.

[smuzaffar]

I am fine with keeping these in single external.