Improve the Arena allocator to reduce memory fragmentation

[rongou]

Currently the arena allocator divides GPU memory into a global arena and per-thread arenas. For smaller allocations, a per-thread arena allocates large chunks of memory (superblocks) from the global arena and divides them up for individual allocations. However, when deallocating from another arena (producer/consumer pattern), or when we run out of memory and return everything to the global arena, the superblock boundaries are broken. Overtime, this could cause the memory to get more and more fragmented.

This PR makes superblocks concrete objects, not just virtual boundaries, and the only units of exchange between the global arena and per-thread arenas. This should make the allocator more resistant to memory fragmentation, especially for long running processes under constant memory pressure.

Other notable changes:

• The allocator now allocates a fixed but configurable amount of memory from CUDA. This introduces less fragmentation comparing to growing the pool size gradually.
• Switched to C++17 std::shared_mutex.
• Added a bunch of unit tests.

fixes #919
fixes #906

[harrism]

Please use draft PRs instead of "WIP" titles.

[rongou]

rerun tests

[harrism]

rerun tests

[rongou]

@harrism @jrhemstad I think this is ready to be merged. Please take another look. Thanks!

[harrism]

Looks good. A few really minor cleanup requests.

[harrism]

@gpucibot merge