Does CUDA-aware MPICH reserve additional memory?

[NamjaeChoi]

Hello.
I'm using CUDA-aware MPICH and observing a strange memory usage.
I'm running a single process but I'm seeing that the same non-negligible amount of memory is being reserved in all the available GPUs from the beginning of execution.
Is this the memory used by MPICH?
If I use 16 processes, those will appear 16 times, making it almost unusable.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  On   | 00000000:01:00.0 Off |                    0 |
| N/A   31C    P0    62W / 400W |    574MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM...  On   | 00000000:41:00.0 Off |                    0 |
| N/A   33C    P0    61W / 400W |    574MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM...  On   | 00000000:81:00.0 Off |                    0 |
| N/A   30C    P0    60W / 400W |    574MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM...  On   | 00000000:C1:00.0 Off |                    0 |
| N/A   28C    P0    59W / 400W |    574MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   2184650      C   ./griffin-opt                     571MiB |
|    1   N/A  N/A   2184650      C   ./griffin-opt                     571MiB |
|    2   N/A  N/A   2184650      C   ./griffin-opt                     571MiB |
|    3   N/A  N/A   2184650      C   ./griffin-opt                     571MiB |
+-----------------------------------------------------------------------------+

[hzhou]

Which MPICH version are you using?

[NamjaeChoi]

With v4.1, processes no longer reserve memory in every GPUs but GPU 0. Is 571MB per process essential memory that cannot be removed?

[hzhou]

The gpu optimization allocates some gpu buffers. In principle, the GPU buffer pool can be shared, but unfortunately, it is currently per-process. I think we can -

1. expose control variable of buffer pool slab size, so users can try a smaller size to save memory
2. Implement inter-process buffer pool

We'll follow up.

[NamjaeChoi]

Okay, good to know. Thank you.

[raffenet]

#6422 and #6417 made it so MPICH will not allocate GPU memory in MPI_Init. MPICH will lazily register some communication buffers with the GPU once it detects the application is doing communication to/from GPU buffers. This will take up some memory, but hopefully in a more targeted manner. Please give the latest main branch a try, if you can, and confirm our changes.

[leofang]

With v4.1, processes no longer reserve memory in every GPUs but GPU 0.

Thank you @NamjaeChoi for noting this. I was about to file a bug report for this behavior with v4.0.3, but decided to do some search first 😄 I'll give v4.1 a shot.