Allow testing of earliest/latest dependencies.

[bdice]

Description

This PR allows the testing of earliest/latest dependencies. This is meant to be interpreted by each repository as it wishes, with CI scripts and rapids-dependency-file-generator to mediate which dependency versions are selected. For RMM, we are pinning to earliest supported numpy/numba versions.

See rapidsai/shared-workflows#228.

This is an experimental PR to prompt discussion / further work with @seberg.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[bdice]

For wheel tests, we will have to engineer a way to use the oldest dependencies. We determine the test dependencies at build time, via [test] extras that are baked into the package. However, for testing earliest dependency pinnings, we really just want to add some extra pinnings to the environment at test time based on the value of RAPIDS_DEPENDENCIES. We may have to add a call in ci/test_wheel.sh that invokes rapids-dependency-file-generator to create a requirements.txt file that we install after installing the previously-built wheel.

[seberg]

Nice, that makes sense to me (and it is really cool to see how straight forward this is looking like).
Don't want to mess with the builds, just testing, so branching between using package[test] and a custom requirements install on RAPIDS_DEPENDENCIES="earliest" seems good.

[bdice]

This worked as desired:

RAPIDS logger » [07/11/24 17:12:53]
┌─────────────────────────────────────┐
|    Create test conda environment    |
└─────────────────────────────────────┘

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
- cuda-version=11.8
- numba==0.57.*
- numpy==1.23.*
- pytest
- pytest-cov
- python=3.9
name: test_python_cuda-118_arch-x86_64_py-39_dependencies-earliest

...

numba                     0.57.1           py39hb75a051_0    conda-forge
numpy                     1.23.5           py39h3d75532_0    conda-forge

[mroeschke]

1. An optional suggestion for terminology: maybe "oldest" or "minimum" instead of "earliest"?
2. Not sure if any RAPIDS libraries support optional dependencies, but one would be able to both required and optional dependencies here?
3. Speculating a use case: I would imagine for a particular common block, a project would ideally want to test both the latest and earliest dependencies so would a specification like:

    common:
      - output_types: [conda, requirements, pyproject]
      - version_types: [latest, earliest]

make sense?

[seberg]

I don't have an opinion on naming, the name "earliest" already exists in the common workflow (but not sure it is actually used for much there).

earliest/latest is only really relevant for testing, so I think we don't need any of the other things mentioned and this is just fine.
I.e. all we need is a way to generate two variations for the testing step, which is already fully separated from the build step.

[seberg]

Needed a bit to realize that a currently valid way to specify say that something works for say Python 3.12 and 3.13 is 3.1[23] fromn the fnmatch synatx.

Likely not often needed in rapids, but it occurred to me that minimal versions can often be Python version dependent.
(It's unfortunate that conda/pip have no tool to ask to resolve the oldest versions.)

[bdice]

In RAPIDS we have historically chosen lower bounds that work for all RAPIDS-supported Python versions.

[bdice]

I chose “earliest/latest” because we named our CUDA drivers that way in shared-workflows. There is also precedent for “oldest” in packages like “oldest-supported-numpy.” Please feel free to change it — I have bequeathed these pull requests to @seberg.

[bdice]

I have a very weak preference to avoid calling it “minimum” versions because we usually say a package with few dependencies or no optional dependencies has “minimal dependencies.”