Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build wheels without build isolation #1473

Open
wants to merge 8 commits into
base: branch-24.12
Choose a base branch
from

Conversation

jameslamb
Copy link
Member

@jameslamb jameslamb commented Oct 11, 2024

Description

Contributes to rapidsai/build-planning#108

Proposes building with with --no-build-isolation, to improve the rate of sccache cache hits and therefore reduce CI times.

Notes for Reviewers

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

How I tested this

Ran a first build with these changes to populate the cache, then ran again... saw a 100% cache hit rate and the actual wheel-building part of a wheel-build-libcuspatial job take only 37 seconds.

Successfully built libcuspatial-cu11
Compile requests                     28
Compile requests executed            28
Cache hits                           28
Cache hits (c [gcc])                  2
Cache hits (c++ [gcc])                2
Cache hits (cuda [nvcc])             24

(build link)

@jameslamb jameslamb added 2 - In Progress Currenty a work in progress 5 - DO NOT MERGE Hold off on merging; see PR for details labels Oct 11, 2024
@github-actions github-actions bot added the ci label Oct 11, 2024
--disable-pip-version-check \
.

sccache --show-adv-stats
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anticipating a review comment like "is this left over from debugging?"... I've intentionally left this in. I think it's generally useful to print the sccache stats in these CI logs, as they could provide a hint when you experience issues like CI timeouts or out-of-memory.

This only adds 27 lines to the logs, like this:

Compile requests                     28
Compile requests executed            28
Cache hits                           28
Cache hits (c [gcc])                  2
Cache hits (c++ [gcc])                2
Cache hits (cuda [nvcc])             24
Cache misses                          0
Cache timeouts                        0
Cache read errors                     0
Forced recaches                       0
Cache write errors                    0
Compilation failures                  0
Cache errors                          0
Non-cacheable compilations            0
Non-cacheable calls                   0
Non-compilation calls                 0
Unsupported compiler calls            0
Average cache write               0.000 s
Average compiler                  0.000 s
Average cache read hit            0.077 s
Failed distributed compilations       0
Cache location                  s3, name: rapids-sccache-east, prefix: /
Version (client)                0.7.7

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is good IMO. I'm also going to add this kind of thing into the telemetry.

@jameslamb jameslamb added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currenty a work in progress 5 - DO NOT MERGE Hold off on merging; see PR for details labels Oct 14, 2024
@jameslamb jameslamb changed the title WIP: [DO NOT MERGE] test wheel-build sccache fixes build wheels without build isolation Oct 14, 2024
@jameslamb jameslamb added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Oct 14, 2024
@jameslamb jameslamb marked this pull request as ready for review October 14, 2024 14:56
@jameslamb jameslamb requested a review from a team as a code owner October 14, 2024 14:56
declare -r matrix_selectors="cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true"

rapids-dependency-file-generator \
--output requirements \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach means we're committing to always specifying requirements in dependencies.yaml alongside pyproject. I've been guilty of neglecting this in the past, and I suspect others are too. (This approach also means people may start neglecting pyproject and we wouldn't notice.) Plus, it means we're duplicating all of pip's build dependency installation logic. Is there any way we can keep the duplication to a minimum?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately, I think the correct approach is adding a --build-isolation-dir argument to pip wheel so that we can control the prefix for sccache purposes, but that obviously won't work in the short term.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team ci improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

3 participants