Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce CI jobs. #236

Merged
merged 3 commits into from
Aug 23, 2024
Merged

Reduce CI jobs. #236

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6419956
Reduce CI jobs.
bdice Aug 21, 2024
7ecd39c
Use more Python 3.11.
bdice Aug 22, 2024
5b01aa9
Use more CUDA 12.2 on ARM.
bdice Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 11 additions & 13 deletions .github/workflows/conda-cpp-tests.yaml
View file
Open in desktop
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes: Look at the conda-python-tests.yaml file for a cleaner diff. This one has the same matrix changes but also added DEPENDENCIES.

Original file line number Diff line number Diff line change
Expand Up @@ -63,27 +63,25 @@ jobs:

# please keep the matrices sorted in ascending order by the following:
#
# [ARCH, PY_VER, CUDA_VER, LINUX_VER, GPU, DRIVER]
# [ARCH, PY_VER, CUDA_VER, LINUX_VER, GPU, DRIVER, DEPENDENCIES]
#
export MATRICES="
pull-request:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
bdice marked this conversation as resolved.
Show resolved Hide resolved
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/conda-python-tests.yaml
View file
Open in desktop
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes:

  • I dropped the ARM nightly job for CUDA 11.4. This should be adequately covered by amd64 tests with CUDA 11.4.
  • Added DEPENDENCIES to match Python matrices
  • I dropped one amd64 nightly job for CUDA 11.4 since we had two. That felt unnecessary.
  • I added a test for ARM with the latest CUDA version. I think we need some test of CUDA-latest on both ARM and amd64. However this means there are no CUDA 12.2 jobs... would we rather change the CUDA 12.0 to 12.2?

Copy link
Member

@jakirkham jakirkham Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this all seems reasonable. Would expect ARM users to more often be on newer hardware and thus have newer dependencies (especially CTK). So this seems like the right direction to shift resources

Edit: Personally I would lean towards on ARM on CUDA 12.2 and x86_64 on CUDA 12.0. Some things (like libcufile) weren't available on ARM until CUDA 12.2 and would like for us to test those

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I would lean towards on ARM on CUDA 12.2 and x86_64 on CUDA 12.0. Some things (like libcufile) weren't available on ARM until CUDA 12.2 and would like for us to test those

Seconded

Copy link
Contributor Author

@bdice bdice Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this to use more CUDA 12.2 in ARM nightlies. For some PR tests I left ARM on CUDA 12.0 where we only have three PR jobs (two x86 and one ARM). I want to cover (11.8, 12.0, 12.5), which is our usual testing strategy of prioritizing CUDA (last major latest minor, current major earliest minor, current major latest minor).

Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,13 @@ jobs:
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
bdice marked this conversation as resolved.
Show resolved Hide resolved
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down
14 changes: 5 additions & 9 deletions .github/workflows/wheels-test.yaml
View file
Open in desktop
Copy link
Contributor Author

@bdice bdice Aug 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes:

  • Previously we had no tests of wheels on Rocky Linux 8. That is important to cover because our wheels are specifically built for glibc 2.28 as a lower bound, which corresponds to Rocky Linux 8.
  • I cut the nightly amd64 jobs quite a bit. It seems like 6 jobs is overkill.
  • I deleted a nightly ARM job for CUDA 11.8 because there were two.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding wheels that was likely an oversight when moving from CentOS 7 to Rocky Linux 8. Agree we should cover them on images where the GLIBC allows

Generally agree with the cuts. Though would shift things a bit amongst those jobs. Will comment below

Original file line number Diff line number Diff line change
Expand Up @@ -80,22 +80,18 @@ jobs:
export MATRICES="
pull-request:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
bdice marked this conversation as resolved.
Show resolved Hide resolved
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
bdice marked this conversation as resolved.
Show resolved Hide resolved
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
bdice marked this conversation as resolved.
Show resolved Hide resolved
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down