Improve build-manifest to work with the improved promote-release #77407
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit improves the way build-manifest calculates the checksums included in the manifest, speeding it up: * Instead of calculating all the hashes beforehand and then using the ones we need, the manifest is first generated with placeholder hashes, and then a function walks through the manifest and calculates only the needed checksums. * Calculating the checksums is now done in parallel with rayon, to better utilize all the available disk bandwidth. * Calculating the checksums now uses the sha2 crate instead of the sha256sum CLI tool: this avoids the overhead of calling another process, but more importantly uses hardware acceleration whenever available (the CLI tool doesn't support it at all).
This avoids overloading the old server, and disrupting the other programs running on it.
rust-highfive
added
the
S-waiting-on-review
|
@bors r=Mark-Simulacrum |
|
📌 Commit 9352062 has been approved by |
bors
added
S-waiting-on-bors
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Oct 3, 2020
…=Mark-Simulacrum Improve build-manifest to work with the improved promote-release This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to [promote-release](https://github.com/rust-lang/promote-release). A new way to invoke the tool was added: `./x.py run src/tools/build-manifest`. The new invocation disables the generation of `.sha256` files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them in `promote-release` will improve the maintenability of our release process. Invocations through the old command (`./x.py dist hash-and-sign`) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server. Improvements were also made on how the checksums included in the manifest are generated: * The manifest is first generated with placeholder checksums, and then a function walks through the manifes and calculates only the needed hashes. Before this PR, all the hashes were calculated beforehand, including the hashes of unused files. * Calculating the hashes is now done in parallel with rayon, to better utilize all the available disk bandwidth. * The `sha2` crate is now used instead of the `sha256sum` CLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (the `sha256sum` CLI tool doesn't support it at all). r? @Mark-Simulacrum This PR is best reviewed commit-by-commit.
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Oct 5, 2020
Rollup of 11 pull requests Successful merges: - rust-lang#75853 (Use more intra-doc-links in `core::fmt`) - rust-lang#75928 (Remove trait_selection error message in specific case) - rust-lang#76329 (Add check for doc alias attribute at crate level) - rust-lang#77219 (core::global_allocator docs link to std::alloc::GlobalAlloc) - rust-lang#77395 (BTreeMap: admit the existence of leaf edges in comments) - rust-lang#77407 (Improve build-manifest to work with the improved promote-release) - rust-lang#77426 (Include scope id in SocketAddrV6::Display) - rust-lang#77439 (Fix missing diagnostic span for `impl Trait` with const generics, and add various tests for `min_const_generics` and `const_generics`) - rust-lang#77471 (BTreeMap: refactoring around edges, missed spots) - rust-lang#77512 (Allow `Abort` terminators in all const-contexts) - rust-lang#77514 (Replace some once(x).chain(once(y)) with [x, y] IntoIter) Failed merges: r? `@ghost`
|
@rustbot modify labels: beta-nominated T-infra Getting this on beta would reduce the transition period for the new release process. |
rustbot
added
beta-nominated
|
Just leaving a note here: this is beta-backport for 1.48, not 1.47 which is releasing later this week. |
|
The beta backport was approved by the infrastructure team! @rustbot modify labels: beta-accepted |
rustbot
added
the
beta-accepted
Mark-Simulacrum
removed
the
beta-nominated
Merged
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Oct 8, 2020
…ulacrum [beta] backports This backports the following: * Improve build-manifest to work with the improved promote-release rust-lang#77407 * Force posix-style quoting on lld, independent of host platform rust-lang#77543 * Fix miscompile in SimplifyBranchSame rust-lang#77549 * Update RLS and Rustfmt rust-lang#77590 * Move `EarlyOtherwiseBranch` to mir-opt-level 2 rust-lang#77582
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to promote-release.
A new way to invoke the tool was added:
./x.py run src/tools/build-manifest. The new invocation disables the generation of.sha256files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them inpromote-releasewill improve the maintenability of our release process. Invocations through the old command (./x.py dist hash-and-sign) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server.Improvements were also made on how the checksums included in the manifest are generated:
sha2crate is now used instead of thesha256sumCLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (thesha256sumCLI tool doesn't support it at all).r? @Mark-Simulacrum
This PR is best reviewed commit-by-commit.