ec2-multi: Dask EC2 cluster example timing out

[jameslamb]

Description

For #434, I tried running the steps at https://docs.rapids.ai/deployment/nightly/cloud/aws/ec2-multi/ using the 24.10 version of RAPIDS (which pulls in dask==2024.9.0 and distributed==2024.9.0).

That example uses dask_cloudprovider.aws.EC2Cluster to create a cluster of EC2 instances, then encourages users to try creating a cudf DataFrame, distributing it over the cluster, and operating on it in parallel (see "Reproducible Example" below).

I saw EC2Cluster() successfully create a cluster of EC2 instances (1 scheduler and 3 workers), but the first time I tried to call .compute() on a dask_cudf.DataFrame to get a result back from the cluster, that operation hung indefinitely (did not time out after 15 minutes).

Reproducible Example

I created an EC2 instances using the "NVIDIA GPU-Optimized AMI", following https://docs.rapids.ai/deployment/nightly/cloud/aws/ec2/.

Then ran the following:

docker run \
    --rm \
    --gpus all \
    --network host \
    --env AWS_PROFILE=oct2024 \
    --entrypoint="" \
    -it rapidsai/notebooks:24.10a-cuda12.5-py3.11 \
    bash

Created a file ~/.aws/credentials in the container, with an [oct2024] profile.

Installed dask-cloudprovider, with the same version as distributed and dask.

conda install --yes -c rapidsai-nightly -c conda-forge \
    'dask-cloudprovider==2024.9.0'

Tried running the example code.

from dask_cloudprovider.aws import EC2Cluster
from dask.distributed import Client
import cudf
import dask_cudf

cluster = EC2Cluster(
    instance_type="g4dn.4xlarge",
    docker_image="rapidsai/base:24.10a-cuda12.5-py3.11",
    worker_class="dask_cuda.CUDAWorker",
    worker_options={"rmm-managed-memory": True},
    docker_args="--shm-size=256m",
    n_workers=3,
    security=False,
    availability_zone="us-east-2a",
    region="us-east-2",
)

client = Client(cluster)

df = dask_cudf.from_cudf(cudf.datasets.timeseries(), npartitions=2)

# hangs on this line
df.x.mean().compute()

This warning is emitted immediately

/opt/conda/lib/python3.11/site-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 39.55 MiB.
This may cause some slowdown.
Consider loading the data with Dask directly
or using futures or delayed objects to embed the data into the graph without repetition.
See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.
warnings.warn(

And then that .mean().compute() does not complete (after 15+ minutes).

output of 'conda info' (click me)

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/rapids/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 24.9.1
    conda-build version : not installed
         python version : 3.11.10.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=cascadelake
                          __conda=24.9.1=0
                          __cuda=12.5=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/rapidsai-nightly/linux-64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/dask/label/dev/linux-64
                          https://conda.anaconda.org/dask/label/dev/noarch
                          https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
          package cache : /opt/conda/pkgs
                          /home/rapids/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/rapids/.conda/envs
               platform : linux-64
             user-agent : conda/24.9.1 requests/2.32.3 CPython/3.11.10 Linux/6.5.0-1018-aws ubuntu/22.04.5 glibc/2.35 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.10
                UID:GID : 1001:1000
             netrc file : None
           offline mode : False

output of 'conda env export' (click me)

name: base
channels:
  - rapidsai-nightly
  - dask/label/dev
  - pytorch
  - conda-forge
  - nvidia
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - affine=2.4.0=pyhd8ed1ab_0
  - aiobotocore=2.15.1=pyhd8ed1ab_0
  - aiohappyeyeballs=2.4.3=pyhd8ed1ab_0
  - aiohttp=3.10.9=py311h9ecbd09_0
  - aioitertools=0.12.0=pyhd8ed1ab_0
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - alsa-lib=1.2.12=h4ab18f5_0
  - anyio=4.6.0=pyhd8ed1ab_1
  - aom=3.9.1=hac33072_0
  - archspec=0.2.3=pyhd8ed1ab_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py311h9ecbd09_5
  - arrow=1.3.0=pyhd8ed1ab_0
  - asttokens=2.4.1=pyhd8ed1ab_0
  - async-lru=2.0.4=pyhd8ed1ab_0
  - attrs=24.2.0=pyh71513ae_0
  - aws-c-auth=0.7.22=hf36ad8f_6
  - aws-c-cal=0.6.15=h816f305_1
  - aws-c-common=0.9.23=h4ab18f5_0
  - aws-c-compression=0.2.18=he027950_7
  - aws-c-event-stream=0.4.2=hb72ac1a_14
  - aws-c-http=0.8.2=h75ac8c9_3
  - aws-c-io=0.14.9=hd3d3696_3
  - aws-c-mqtt=0.10.4=hb0abfc5_7
  - aws-c-s3=0.5.10=h44b787d_4
  - aws-c-sdkutils=0.1.16=he027950_3
  - aws-checksums=0.1.18=he027950_7
  - aws-crt-cpp=0.26.12=he940a02_1
  - aws-sdk-cpp=1.11.329=h0f5bab0_6
  - awscli=2.18.1=py311h38be061_0
  - awscrt=0.20.12=py311h1684233_1
  - azure-core-cpp=1.13.0=h935415a_0
  - azure-identity-cpp=1.9.0=hd126650_0
  - azure-storage-blobs-cpp=12.13.0=h1d30c4a_0
  - azure-storage-common-cpp=12.8.0=ha3822c6_0
  - azure-storage-files-datalake-cpp=12.12.0=h0f25b8a_0
  - babel=2.14.0=pyhd8ed1ab_0
  - beautifulsoup4=4.12.3=pyha770c72_0
  - bleach=6.1.0=pyhd8ed1ab_0
  - blosc=1.21.6=hef167b5_0
  - bokeh=3.5.2=pyhd8ed1ab_0
  - boltons=24.0.0=pyhd8ed1ab_0
  - botocore=1.35.23=pyge310_1234567_0
  - branca=0.7.2=pyhd8ed1ab_0
  - brotli=1.1.0=hb9d3cd8_2
  - brotli-bin=1.1.0=hb9d3cd8_2
  - brotli-python=1.1.0=py311hfdbb021_2
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=h4bc722e_7
  - c-ares=1.33.1=heb4867d_0
  - c-blosc2=2.15.1=hc57e6cf_0
  - ca-certificates=2024.8.30=hbcca054_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cachetools=5.5.0=pyhd8ed1ab_0
  - cairo=1.18.0=hebfffa5_3
  - certifi=2024.8.30=pyhd8ed1ab_0
  - cffi=1.17.1=py311hf29c0ef_0
  - charls=2.4.2=h59595ed_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - click-plugins=1.1.1=py_0
  - cligj=0.7.2=pyhd8ed1ab_1
  - cloudpickle=3.0.0=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - colorcet=3.1.0=pyhd8ed1ab_0
  - comm=0.2.2=pyhd8ed1ab_0
  - conda=24.9.1=py311h38be061_0
  - conda-libmamba-solver=24.9.0=pyhd8ed1ab_0
  - conda-package-handling=2.3.0=pyh7900ff3_0
  - conda-package-streaming=0.10.0=pyhd8ed1ab_0
  - contourpy=1.3.0=py311hd18a35c_2
  - coverage=7.6.1=py311h9ecbd09_1
  - cryptography=40.0.2=py311h9b4c7bb_0
  - cucim=24.10.00a21=cuda12_py311_241008_g289e340_21
  - cuda-cccl_linux-64=12.5.39=ha770c72_0
  - cuda-crt-dev_linux-64=12.5.82=ha770c72_0
  - cuda-crt-tools=12.5.82=ha770c72_0
  - cuda-cudart=12.5.82=he02047a_0
  - cuda-cudart-dev=12.5.82=he02047a_0
  - cuda-cudart-dev_linux-64=12.5.82=h85509e4_0
  - cuda-cudart-static=12.5.82=he02047a_0
  - cuda-cudart-static_linux-64=12.5.82=h85509e4_0
  - cuda-cudart_linux-64=12.5.82=h85509e4_0
  - cuda-nvcc-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvcc-impl=12.5.82=hd3aeb46_0
  - cuda-nvcc-tools=12.5.82=hd3aeb46_0
  - cuda-nvrtc=12.5.82=he02047a_0
  - cuda-nvvm-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvvm-impl=12.5.82=h59595ed_0
  - cuda-nvvm-tools=12.5.82=h59595ed_0
  - cuda-profiler-api=12.5.39=ha770c72_0
  - cuda-python=12.6.0=py311h817de4b_0
  - cuda-version=12.5=hd4f0392_3
  - cudf=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cudf_kafka=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cugraph=24.10.00a96=cuda12_py311_241008_gacf502987_96
  - cuml=24.10.00a67=cuda12_py311_241008_g70fe52684_67
  - cuproj=24.10.00a48=cuda12_py311_241008_gcc8fd60b_48
  - cupy=13.3.0=py311h1c6efab_0
  - cupy-core=13.3.0=py311h95322c3_0
  - curl=8.10.1=hbbe4b11_0
  - cuspatial=24.10.00a48=cuda12_py311_241008_gcc8fd60b_48
  - custreamz=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cuvs=24.10.00a92=cuda12_py311_241008_gc14b879_92
  - cuxfilter=24.10.00a25=cuda12_py311_241008_g0cd1568_25
  - cycler=0.12.1=pyhd8ed1ab_0
  - cyrus-sasl=2.1.27=h54b06d7_7
  - cython=3.0.11=py311h55d416d_3
  - cytoolz=1.0.0=py311h9ecbd09_1
  - dask=2024.9.0=pyhd8ed1ab_0
  - dask-cloudprovider=2024.9.1=pyhd8ed1ab_0
  - dask-core=2024.9.0=pyhd8ed1ab_0
  - dask-cuda=24.10.00a23=py311_241008_g1c84a6a_23
  - dask-cudf=24.10.00a401=cuda12_py311_241007_g8a9df040e1_401
  - dask-expr=1.1.14=pyhd8ed1ab_0
  - dask-glm=0.3.2=pyhd8ed1ab_0
  - dask-labextension=7.0.0=pyhd8ed1ab_0
  - dask-ml=2023.3.24=pyhd8ed1ab_1
  - datashader=0.16.3=pyhd8ed1ab_0
  - dav1d=1.2.1=hd590300_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.8.6=py311hfdbb021_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - distributed=2024.9.0=pyhd8ed1ab_0
  - distributed-ucxx=0.40.00a=py3.11_241008_g1d169a4_33
  - distro=1.8.0=pyhd8ed1ab_0
  - dlpack=0.8=h59595ed_3
  - docutils=0.19=py311h38be061_1
  - double-conversion=3.3.0=h59595ed_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - exceptiongroup=1.2.2=pyhd8ed1ab_0
  - execnet=2.1.1=pyhd8ed1ab_0
  - executing=2.1.0=pyhd8ed1ab_0
  - expat=2.6.3=h5888daf_0
  - fastrlock=0.8.2=py311hb755f60_2
  - fmt=11.0.2=h434a139_0
  - folium=0.17.0=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_3
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.54.1=py311h9ecbd09_0
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - freexl=2.0.0=h743c826_0
  - frozendict=2.4.4=py311h9ecbd09_1
  - frozenlist=1.4.1=py311h9ecbd09_1
  - fsspec=2024.9.0=pyhff2d567_0
  - gdal=3.9.2=py311h5159542_7
  - geopandas=1.0.1=pyhd8ed1ab_1
  - geopandas-base=1.0.1=pyha770c72_1
  - geos=3.13.0=h5888daf_0
  - geotiff=1.7.3=h77b800c_3
  - gettext=0.22.5=he02047a_3
  - gettext-tools=0.22.5=he02047a_3
  - gflags=2.2.2=h5888daf_1005
  - giflib=5.2.2=hd590300_0
  - glog=0.7.1=hbabe93e_0
  - graphite2=1.3.13=h59595ed_1003
  - h11=0.14.0=pyhd8ed1ab_0
  - h2=4.1.0=pyhd8ed1ab_0
  - harfbuzz=9.0.0=hda332d3_1
  - holoviews=1.19.1=pyhd8ed1ab_0
  - hpack=4.0.0=pyh9f0ad1d_0
  - httpcore=1.0.6=pyhd8ed1ab_0
  - httpx=0.27.2=pyhd8ed1ab_0
  - hyperframe=6.0.1=pyhd8ed1ab_0
  - icu=75.1=he02047a_0
  - idna=3.10=pyhd8ed1ab_0
  - imagecodecs=2024.6.1=py311h1c1dfb1_5
  - imageio=2.35.1=pyh12aca89_0
  - importlib-metadata=8.5.0=pyha770c72_0
  - importlib_metadata=8.5.0=hd8ed1ab_0
  - importlib_resources=6.4.5=pyhd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - ipykernel=6.29.5=pyh3099207_0
  - ipython=8.17.2=pyh41d4057_0
  - ipywidgets=8.1.5=pyhd8ed1ab_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - jedi=0.19.1=pyhd8ed1ab_0
  - jinja2=3.1.4=pyhd8ed1ab_0
  - jmespath=1.0.1=pyhd8ed1ab_0
  - joblib=1.4.2=pyhd8ed1ab_0
  - json-c=0.18=h6688a6e_0
  - json5=0.9.25=pyhd8ed1ab_0
  - jsonpatch=1.33=pyhd8ed1ab_0
  - jsonpointer=3.0.0=py311h38be061_1
  - jsonschema=4.23.0=pyhd8ed1ab_0
  - jsonschema-specifications=2023.12.1=pyhd8ed1ab_0
  - jsonschema-with-format-nongpl=4.23.0=hd8ed1ab_0
  - jupyter=1.1.1=pyhd8ed1ab_0
  - jupyter-lsp=2.2.5=pyhd8ed1ab_0
  - jupyter-server-proxy=4.4.0=pyhd8ed1ab_0
  - jupyter_client=8.6.3=pyhd8ed1ab_0
  - jupyter_console=6.6.3=pyhd8ed1ab_0
  - jupyter_core=5.7.2=pyh31011fe_1
  - jupyter_events=0.10.0=pyhd8ed1ab_0
  - jupyter_server=2.14.2=pyhd8ed1ab_0
  - jupyter_server_terminals=0.5.3=pyhd8ed1ab_0
  - jupyterlab=4.2.5=pyhd8ed1ab_0
  - jupyterlab-nvdashboard=0.11.00=py_240603_gf481e0c_0
  - jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
  - jupyterlab_server=2.27.3=pyhd8ed1ab_0
  - jupyterlab_widgets=3.0.13=pyhd8ed1ab_0
  - jxrlib=1.1=hd590300_3
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.7=py311hd18a35c_0
  - krb5=1.21.3=h659f571_0
  - lazy-loader=0.4=pyhd8ed1ab_1
  - lazy_loader=0.4=pyhd8ed1ab_1
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.43=h712a8e2_1
  - lerc=4.0.0=h27087fc_0
  - libabseil=20240116.2=cxx17_he02047a_1
  - libaec=1.1.3=h59595ed_0
  - libarchive=3.7.4=hfca40fe_0
  - libarrow=16.1.0=h9102155_9_cpu
  - libarrow-acero=16.1.0=hac33072_9_cpu
  - libarrow-dataset=16.1.0=hac33072_9_cpu
  - libarrow-substrait=16.1.0=h7e0c224_9_cpu
  - libasprintf=0.22.5=he8f35ee_3
  - libasprintf-devel=0.22.5=he8f35ee_3
  - libavif16=1.1.1=h104a339_1
  - libblas=3.9.0=24_linux64_openblas
  - libbrotlicommon=1.1.0=hb9d3cd8_2
  - libbrotlidec=1.1.0=hb9d3cd8_2
  - libbrotlienc=1.1.0=hb9d3cd8_2
  - libcblas=3.9.0=24_linux64_openblas
  - libclang-cpp19.1=19.1.0=default_hb5137d0_0
  - libclang13=19.1.0=default_h9c6a7e4_0
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libcublas=12.5.3.2=he02047a_0
  - libcublas-dev=12.5.3.2=he02047a_0
  - libcucim=24.10.00a21=cuda12_241008_g289e340_21
  - libcudf=24.10.00a401=cuda12_241008_g8a9df040e1_401
  - libcudf_kafka=24.10.00a401=cuda12_241008_g8a9df040e1_401
  - libcufft=11.2.3.61=he02047a_0
  - libcufile=1.10.1.7=he02047a_0
  - libcufile-dev=1.10.1.7=he02047a_0
  - libcugraph=24.10.00a96=cuda12_241008_gacf502987_96
  - libcugraph_etl=24.10.00a96=cuda12_241008_gacf502987_96
  - libcugraphops=24.10.00a15=cuda12_241008_g0bfc7c51_15
  - libcuml=24.10.00a67=cuda12_241008_g70fe52684_67
  - libcumlprims=24.10.00a=cuda12_241008_g51defe5_9
  - libcups=2.3.3=h4637d8d_4
  - libcurand=10.3.6.82=he02047a_0
  - libcurand-dev=10.3.6.82=he02047a_0
  - libcurl=8.10.1=hbbe4b11_0
  - libcusolver=11.6.3.83=he02047a_0
  - libcusolver-dev=11.6.3.83=he02047a_0
  - libcusparse=12.5.1.3=he02047a_0
  - libcusparse-dev=12.5.1.3=he02047a_0
  - libcuspatial=24.10.00a48=cuda12_241008_gcc8fd60b_48
  - libcuvs=24.10.00a92=cuda12_241008_gc14b879_92
  - libdeflate=1.22=hb9d3cd8_0
  - libdrm=2.4.123=hb9d3cd8_0
  - libedit=3.1.20191231=he28a2e2_2
  - libegl=1.7.0=ha4b6fd6_1
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libexpat=2.6.3=h5888daf_0
  - libffi=3.4.2=h7f98852_5
  - libgcc=14.1.0=h77fa898_1
  - libgcc-ng=14.1.0=h69a702a_1
  - libgdal-core=3.9.2=hd5b9bfb_7
  - libgettextpo=0.22.5=he02047a_3
  - libgettextpo-devel=0.22.5=he02047a_3
  - libgfortran=14.1.0=h69a702a_1
  - libgfortran-ng=14.1.0=h69a702a_1
  - libgfortran5=14.1.0=hc5f4f2c_1
  - libgl=1.7.0=ha4b6fd6_1
  - libglib=2.82.1=h2ff4ddf_0
  - libglvnd=1.7.0=ha4b6fd6_1
  - libglx=1.7.0=ha4b6fd6_1
  - libgomp=14.1.0=h77fa898_1
  - libgoogle-cloud=2.25.0=h2736e30_0
  - libgoogle-cloud-storage=2.25.0=h3d9a0c8_0
  - libgrpc=1.62.2=h15f2491_0
  - libhwy=1.1.0=h00ab1b0_0
  - libiconv=1.17=hd590300_2
  - libidn2=2.3.7=hd590300_0
  - libjpeg-turbo=3.0.0=hd590300_1
  - libjxl=0.11.0=hdb8da77_1
  - libkml=1.3.0=hf539b9f_1021
  - libkvikio=24.10.00a=cuda12_241008_g5a733bd_53
  - liblapack=3.9.0=24_linux64_openblas
  - libllvm14=14.0.6=hcd5def8_4
  - libllvm19=19.1.1=ha7bfdaf_0
  - libmamba=1.5.10=hf72d635_1
  - libmambapy=1.5.10=py311h18a8eac_1
  - libnghttp2=1.58.0=h47da74e_1
  - libnl=3.10.0=h4bc722e_0
  - libnsl=2.0.1=hd590300_0
  - libntlm=1.4=h7f98852_1002
  - libnvjitlink=12.5.82=he02047a_0
  - libnvjpeg=12.3.2.81=he02047a_0
  - libopenblas=0.3.27=pthreads_hac2b453_1
  - libopengl=1.7.0=ha4b6fd6_1
  - libparquet=16.1.0=h6a7eafb_9_cpu
  - libpciaccess=0.18=hd590300_0
  - libpng=1.6.44=hadc24fc_0
  - libpq=17.0=h04577a9_2
  - libprotobuf=4.25.3=hd5b35b9_1
  - libraft=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - libraft-headers=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - libraft-headers-only=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - librdkafka=2.5.3=h95ba008_0
  - libre2-11=2023.09.01=h5a48ba9_2
  - librmm=24.10.00a42=cuda12_241008_gab6e2961_42
  - librttopo=1.1.0=h97f6797_17
  - libsodium=1.0.20=h4ab18f5_0
  - libsolv=0.7.30=h3509ff9_0
  - libspatialite=5.1.0=h1b4f908_11
  - libsqlite=3.46.1=hadc24fc_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx=14.1.0=hc0a3c3a_1
  - libstdcxx-ng=14.1.0=h4852527_1
  - libthrift=0.19.0=hb90f79a_1
  - libtiff=4.7.0=he137b08_1
  - libucxx=0.40.00a=cuda12_241008_g1d169a4_33
  - libunistring=0.9.10=h7f98852_0
  - libutf8proc=2.8.0=h166bdaf_0
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.49.0=hb9d3cd8_0
  - libwebp-base=1.4.0=hd590300_0
  - libwholegraph=24.10.00a20=cuda12_241008_gf8ad9a1_20
  - libxcb=1.17.0=h8a09558_0
  - libxcrypt=4.4.36=hd590300_1
  - libxgboost=2.1.1=rapidsai_h01f03eb_5
  - libxkbcommon=1.7.0=h2c5496b_1
  - libxml2=2.12.7=he7c6b58_4
  - libxslt=1.1.39=h76b75d6_0
  - libzlib=1.3.1=hb9d3cd8_2
  - libzopfli=1.0.3=h9c3ff4c_0
  - linkify-it-py=2.0.3=pyhd8ed1ab_0
  - llvmlite=0.43.0=py311h9c9ff8c_1
  - locket=1.0.0=pyhd8ed1ab_0
  - lz4=4.3.3=py311h2cbdf9a_1
  - lz4-c=1.9.4=hcb278e6_0
  - lzo=2.10=hd590300_1001
  - mamba=1.5.10=py311h3072747_1
  - mapclassify=2.8.1=pyhd8ed1ab_0
  - markdown=3.6=pyhd8ed1ab_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=3.0.0=py311h9ecbd09_0
  - matplotlib=3.9.2=py311h38be061_1
  - matplotlib-base=3.9.2=py311h2b939e6_1
  - matplotlib-inline=0.1.7=pyhd8ed1ab_0
  - mdit-py-plugins=0.4.2=pyhd8ed1ab_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - menuinst=2.1.2=py311h38be061_1
  - minizip=4.0.7=h401b404_0
  - mistune=3.0.2=pyhd8ed1ab_0
  - msgpack-python=1.1.0=py311hd18a35c_0
  - multidict=6.1.0=py311h9ecbd09_0
  - multipledispatch=0.6.0=pyhd8ed1ab_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=9.0.1=h266115a_1
  - mysql-libs=9.0.1=he0572af_1
  - nbclient=0.10.0=pyhd8ed1ab_0
  - nbconvert-core=7.16.4=pyhd8ed1ab_1
  - nbformat=5.10.4=pyhd8ed1ab_0
  - nccl=2.23.4.1=h52f6c39_0
  - ncurses=6.5=he02047a_1
  - nest-asyncio=1.6.0=pyhd8ed1ab_0
  - networkx=3.3=pyhd8ed1ab_1
  - nodejs=22.9.0=hf235a45_0
  - notebook=7.2.2=pyhd8ed1ab_0
  - notebook-shim=0.2.4=pyhd8ed1ab_0
  - numba=0.60.0=py311h4bc866e_0
  - numpy=1.26.4=py311h64a7726_0
  - nvcomp=4.0.1=hbc370b7_0
  - nvtx=0.2.10=py311h9ecbd09_2
  - nx-cugraph=24.10.00a96=py311_241008_gacf502987_96
  - openjpeg=2.5.2=h488ebb8_0
  - openldap=2.6.8=hedd0468_0
  - openssl=3.3.2=hb9d3cd8_0
  - orc=2.0.1=h17fec99_1
  - osmnx=1.9.3=pyhd8ed1ab_0
  - overrides=7.7.0=pyhd8ed1ab_0
  - packaging=24.1=pyhd8ed1ab_0
  - pandas=2.2.2=py311h14de704_1
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - panel=1.5.2=pyhd8ed1ab_0
  - param=2.1.1=pyhff2d567_0
  - parso=0.8.4=pyhd8ed1ab_0
  - partd=1.4.2=pyhd8ed1ab_0
  - patsy=0.5.6=pyhd8ed1ab_0
  - pcre2=10.44=hba22ea6_2
  - perl=5.32.1=7_hd590300_perl5
  - pexpect=4.9.0=pyhd8ed1ab_0
  - pickleshare=0.7.5=py_1003
  - pillow=10.4.0=py311h4aec55e_1
  - pip=24.2=pyh8b19718_1
  - pixman=0.43.2=h59595ed_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - platformdirs=4.3.6=pyhd8ed1ab_0
  - pluggy=1.5.0=pyhd8ed1ab_0
  - ply=3.11=pyhd8ed1ab_2
  - proj=9.5.0=h12925eb_0
  - prometheus_client=0.21.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.38=pyha770c72_0
  - prompt_toolkit=3.0.38=hd8ed1ab_0
  - psutil=6.0.0=py311h9ecbd09_1
  - pthread-stubs=0.4=hb9d3cd8_1002
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.3=pyhd8ed1ab_0
  - py-cpuinfo=9.0.0=pyhd8ed1ab_0
  - py-xgboost=2.1.1=rapidsai_pyh199b97d_5
  - pyarrow=16.1.0=py311hbd00459_4
  - pyarrow-core=16.1.0=py311h8c3dac4_4_cpu
  - pyarrow-hotfix=0.6=pyhd8ed1ab_0
  - pybind11-abi=4=hd8ed1ab_3
  - pycosat=0.6.6=py311h459d7ec_0
  - pycparser=2.22=pyhd8ed1ab_0
  - pyct=0.5.0=pyhd8ed1ab_0
  - pydeck=0.8.0=pyhd8ed1ab_0
  - pygments=2.18.0=pyhd8ed1ab_0
  - pylibcudf=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - pylibcugraph=24.10.00a96=cuda12_py311_241008_gacf502987_96
  - pylibraft=24.10.00a48=cuda12_py311_241008_g6c4fdfb3_48
  - pylibwholegraph=24.10.00a20=cuda12_py311_241008_gf8ad9a1_20
  - pynvjitlink=0.3.0=py311hd269673_0
  - pynvml=11.4.1=pyhd8ed1ab_0
  - pyogrio=0.10.0=py311h5fbebbf_0
  - pyparsing=3.1.4=pyhd8ed1ab_0
  - pyproj=3.7.0=py311h0f98d5a_0
  - pyside6=6.7.3=py311h9053184_1
  - pysocks=1.7.1=pyha2e5f31_6
  - pytest=8.3.3=pyhd8ed1ab_0
  - pytest-benchmark=4.0.0=pyhd8ed1ab_0
  - pytest-cov=5.0.0=pyhd8ed1ab_0
  - pytest-xdist=3.6.1=pyhd8ed1ab_0
  - python=3.11.10=hc5c86c4_2_cpython
  - python-confluent-kafka=2.5.3=py311h9ecbd09_0
  - python-dateutil=2.9.0=pyhd8ed1ab_0
  - python-fastjsonschema=2.20.0=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python-louvain=0.16=pyhd8ed1ab_0
  - python-tzdata=2024.2=pyhd8ed1ab_0
  - python_abi=3.11=5_cp311
  - pytz=2024.2=pyhd8ed1ab_0
  - pyviz_comms=3.0.3=pyhd8ed1ab_0
  - pywavelets=1.7.0=py311h9f3472d_1
  - pyyaml=6.0.2=py311h9ecbd09_1
  - pyzmq=26.2.0=py311h7deb3e3_2
  - qhull=2020.2=h434a139_5
  - qt6-main=6.7.3=h6e8976b_1
  - raft-dask=24.10.00a48=cuda12_py311_241008_g6c4fdfb3_48
  - rapids=24.10.00a=cuda12_py311_240930_gfb695dd_11
  - rapids-dask-dependency=24.10.00a8=py_0
  - rapids-xgboost=24.10.00a=cuda12_py311_240930_gfb695dd_11
  - rasterio=1.4.1=py311hfbe26e2_0
  - rav1e=0.6.6=he8a937b_2
  - rdma-core=54.0=h5888daf_0
  - re2=2023.09.01=h7f4b329_2
  - readline=8.2=h8228510_1
  - referencing=0.35.1=pyhd8ed1ab_0
  - reproc=14.2.4.post0=hd590300_1
  - reproc-cpp=14.2.4.post0=h59595ed_1
  - requests=2.32.3=pyhd8ed1ab_0
  - rfc3339-validator=0.1.4=pyhd8ed1ab_0
  - rfc3986-validator=0.1.1=pyh9f0ad1d_0
  - rich=13.9.2=pyhd8ed1ab_0
  - rmm=24.10.00a42=cuda12_py311_241008_gab6e2961_42
  - rpds-py=0.20.0=py311h9e33e62_1
  - ruamel.yaml=0.17.21=py311h2582759_3
  - ruamel.yaml.clib=0.2.8=py311h459d7ec_0
  - s2n=1.4.16=he19d79f_0
  - scikit-image=0.24.0=py311h044e617_2
  - scikit-learn=1.5.0=py311he08f58d_1
  - scipy=1.14.1=py311he1f765f_0
  - seaborn=0.13.2=hd8ed1ab_2
  - seaborn-base=0.13.2=pyhd8ed1ab_2
  - send2trash=1.8.3=pyh0d859eb_0
  - setuptools=75.1.0=pyhd8ed1ab_0
  - shapely=2.0.6=py311h2fdb869_2
  - simpervisor=1.0.0=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.2.1=ha2e4443_0
  - sniffio=1.3.1=pyhd8ed1ab_0
  - snuggs=1.4.7=pyhd8ed1ab_1
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - soupsieve=2.5=pyhd8ed1ab_1
  - sparse=0.15.4=pyh267e887_1
  - spdlog=1.14.1=hed91bc2_1
  - sqlite=3.46.1=h9eae976_0
  - stack_data=0.6.2=pyhd8ed1ab_0
  - statsmodels=0.14.4=py311h9f3472d_0
  - streamz=0.6.4=pyh6c4a22f_0
  - svt-av1=2.2.1=h5888daf_0
  - tblib=3.0.0=pyhd8ed1ab_0
  - terminado=0.18.1=pyh0d859eb_0
  - threadpoolctl=3.5.0=pyhc1e730c_0
  - thriftpy2=0.5.2=py311h9ecbd09_1
  - tifffile=2024.9.20=pyhd8ed1ab_0
  - tinycss2=1.3.0=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.2=pyhd8ed1ab_0
  - toolz=1.0.0=pyhd8ed1ab_0
  - tornado=6.4.1=py311h9ecbd09_1
  - tqdm=4.66.5=pyhd8ed1ab_0
  - traitlets=5.14.3=pyhd8ed1ab_0
  - treelite=4.3.0=py311he8f9275_0
  - truststore=0.9.2=pyhd8ed1ab_0
  - types-python-dateutil=2.9.0.20241003=pyhff2d567_0
  - typing-extensions=4.12.2=hd8ed1ab_0
  - typing_extensions=4.12.2=pyha770c72_0
  - typing_utils=0.1.0=pyhd8ed1ab_0
  - tzdata=2024b=hc8b5060_0
  - uc-micro-py=1.0.3=pyhd8ed1ab_0
  - ucx=1.17.0=h05e919c_3
  - ucx-proc=1.0.0=gpu
  - ucx-py=0.40.00a15=py311_241008_gaf0bf5c_15
  - ucxx=0.40.00a=cuda12_py3.11_241008_g1d169a4_33
  - uri-template=1.3.0=pyhd8ed1ab_0
  - uriparser=0.9.8=hac33072_0
  - urllib3=1.26.19=pyhd8ed1ab_0
  - vim=9.1.0611=py311pl5321ha5a8562_1
  - wayland=1.23.1=h3e06ad9_0
  - wcwidth=0.2.13=pyhd8ed1ab_0
  - webcolors=24.8.0=pyhd8ed1ab_0
  - webencodings=0.5.1=pyhd8ed1ab_2
  - websocket-client=1.8.0=pyhd8ed1ab_0
  - wget=1.21.4=hda4d442_0
  - wheel=0.44.0=pyhd8ed1ab_0
  - widgetsnbextension=4.0.13=pyhd8ed1ab_0
  - wrapt=1.16.0=py311h9ecbd09_1
  - xarray=2024.9.0=pyhd8ed1ab_0
  - xcb-util=0.4.1=hb711507_2
  - xcb-util-cursor=0.1.5=hb9d3cd8_0
  - xcb-util-image=0.4.0=hb711507_2
  - xcb-util-keysyms=0.4.1=hb711507_0
  - xcb-util-renderutil=0.3.10=hb711507_0
  - xcb-util-wm=0.4.2=hb711507_0
  - xerces-c=3.2.5=h988505b_2
  - xgboost=2.1.1=rapidsai_pyh9f47a55_5
  - xkeyboard-config=2.43=hb9d3cd8_0
  - xorg-libice=1.1.1=hb9d3cd8_1
  - xorg-libsm=1.2.4=he73a12e_1
  - xorg-libx11=1.8.10=h4f16b4b_0
  - xorg-libxau=1.0.11=hb9d3cd8_1
  - xorg-libxcomposite=0.4.6=hb9d3cd8_2
  - xorg-libxcursor=1.2.2=hb9d3cd8_0
  - xorg-libxdamage=1.1.6=hb9d3cd8_0
  - xorg-libxdmcp=1.1.5=hb9d3cd8_0
  - xorg-libxext=1.3.6=hb9d3cd8_0
  - xorg-libxfixes=6.0.1=hb9d3cd8_0
  - xorg-libxi=1.8.2=hb9d3cd8_0
  - xorg-libxrandr=1.5.4=hb9d3cd8_0
  - xorg-libxrender=0.9.11=hb9d3cd8_1
  - xorg-libxt=1.3.0=hb9d3cd8_2
  - xorg-libxtst=1.2.5=hb9d3cd8_3
  - xorg-libxxf86vm=1.1.5=hb9d3cd8_3
  - xorg-xorgproto=2024.1=hb9d3cd8_1
  - xyzservices=2024.9.0=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yaml-cpp=0.8.0=h59595ed_0
  - yarl=1.13.1=py311h9ecbd09_0
  - zeromq=4.3.5=h3b0a872_6
  - zfp=1.0.1=h5888daf_2
  - zict=3.0.0=pyhd8ed1ab_0
  - zipp=3.20.2=pyhd8ed1ab_0
  - zlib=1.3.1=hb9d3cd8_2
  - zlib-ng=2.2.2=h5888daf_0
  - zstandard=0.23.0=py311hbc35293_1
  - zstd=1.5.6=ha6fb4c9_0
prefix: /opt/conda

Notes

Maybe related: #343