Memory hooks

UCX includes the libucm library, which provides methods to intercept events which map and unmap memory to/from the current process. These may be used by transports to optimize their handling of user-allocated memory, for example:

• Maintain memory registration cache (and get notification when cached memory is unmapped)
• Modify the way memory is allocated (e.g hugepages, MAP_PRIVATE vs MAP_SHARED)

Events API

libucm API allows registering for the following events:

UCM_EVENT_MMAP	mmap() is called
UCM_EVENT_MUNMAP	munmap() is called
UCM_EVENT_MREMAP	mremap() is called
UCM_EVENT_SHMAT	shmat() is called
UCM_EVENT_SHMDT	shmdt() is called
UCM_EVENT_SBRK	sbrk() is called
UCM_EVENT_VM_MAPPED	memory is mapped to the process
UCM_EVENT_VM_UNMAPPED	memory is unmapped from the process

An event handler may modify the parameters, set the result, or do nothing and continue to the next handler.

Installing the hooks

We use the following algorithm to install the memory hooks:

1. Install hooks for mmap/munmap/mremap/shmat/shmdt/sbrk

2. libucm contains symbols with these names. If libucm is loaded before any other implementation of those functions (for example, by using LD_PRELOAD), nothing else should be done here. This is detected by calling hte functions and checking if the events work.

3. If this didn't work, modify the relocation tables or all currently loaded objects (and objects that will be loaded in the future*) to point to our implementation of these functions.

4. TBD modify the loaded code of glibc to call our hooks (IBM's method)

5. Test events again. If this failed, notify the user we can't install memory events.

6. Install hooks for malloc/free/realloc/memalign

7. Sometimes it's enough to have hooks for mmap/... to get those events when they are called from malloc/... as well. So first we do some memory allocations and check if we are able to get all events this way.

8. If we can't, install malloc hooks using the old "deprecated" way (__malloc_hook).
   We have our own implementation of heap manager in libucm - ptmalloc3. After we replace the original heap manager, we keep track of which pointers were allocated by our library, so we would know ignore all others (since they were allocated by the previous heap manager). Also, we can't restore the previous state, so libucm.so is marked as 'nodelete'.

9. If the former didn't work, modify the relocation tables to point to our implementation of malloc (and friends).

10. If even that didn't work, notify the user we can't install memory events.

11. If one of the methods was successful, modify the relocation tables to point to our versions of malloc_trim, malloc_stats, mallinfo, and so on.