__malloc_hook issue

[igor-ivanov]

This issue came from UCX:
btl openib sets __malloc_hook to handle alignment (added in this commit cd17fee)

when it’s closed (component_close), for example because another component has higher priority, it restores the original handler.

The problem is that the __malloc_hook may have been changed since then by other component (e.g UCX), and restoring the original handler overrides this other component setting (UCX).

As a result, when UCX is running, it installs __malloc_hook, but btl openib changes it “under its feet”

I think this should be fixed by checking that the currentl __malloc_hook still points to btl_openib’s replacement, before restoring the original.

[ggouaillardet]

if you do that, when ucx is closed, it might restore the openib malloc hook.
and if the openib btl was unmmap'ed, very bad things will happen.

I am afraid we need something more complex
for example a "stack" of hooks, and the possibility to remove a hook in the middle of the stack

[jsquyres]

I agree with @ggouaillardet -- sounds like some OPAL-level infrastructure to track the malloc hook is required.

[yosefe]

@igor-ivanov maybe at least we can install malloc hook for btl_openib, only if it's really used, and not if it is not eventually selected?