Produce source package in rust-installer format #34366

Diggsey · 2016-06-19T17:50:08Z

See rust-lang-deprecated/rust-buildbot#102

There may be a better way to do this, wasn't sure how to clean-up the rust-src-image directory when it's used by multiple make rules.

rust-highfive · 2016-06-19T17:50:20Z

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2016-06-19T21:59:41Z

Thanks! Could you also update src/bootstrap/build/dist.rs with a rule for this as well?

Diggsey · 2016-06-19T23:05:57Z

The rustbuild system doesn't seem to produce a source tarball at all right now - should I make it produce the vanilla tarball in addition to the rust-installer package?

alexcrichton · 2016-06-20T15:42:20Z

Oh whoops! Eh it's fine to not do that as part of this PR if you'd prefer, but if you wanna throw it in I'm also fine with that!

brson · 2016-06-20T23:36:27Z

mk/dist.mk

+	$(Q)$(S)src/rust-installer/gen-installer.sh \
+		--product-name=Rust \
+		--rel-manifest-dir=rustlib \
+		--success-message=Awesome-Source. \


brson · 2016-06-20T23:43:48Z

cc #19535

brson · 2016-06-20T23:48:44Z

So after installation of the rust-src package the full Rust source will be at lib/rustlib/rust-src meaning the source to std will be at lib/rustlib/rust-src/src/libstd.

We've also discussed about the potential to need other source installations, but without concrete use cases. You might imagine that rust-src gets split into rust-src, rust-core-src, rustc-src etc. or we add a cargo-src. Should we consider making the scheme for storing the source expandable by putting this at e.g. lib/rust/lib/src/rust/?

cc @phildawes @alexreg @MarkSwanson this PR is deciding the scheme for storing the installed Rust source on disk. Once we deploy it it will be very hard to change.

cc brson/multirust#77

brson · 2016-06-20T23:52:14Z

It might also be worth considering the impact on side-by-side installation. Right now it's in theory vaguely possible to install multiple Rust's right on top of each other because of the disambiguated crate names, though in practice there are artifacts in rustlib that are not disambiguated. Putting a single copy of the source at a known location in rustlib means that only one can exist at a time. This is fine for rustup, but not fine for e.g. Linux distros.

I think I'm kinda fine just saying that every rustlib belongs to a single Rust installation, but it does put a nail in the coffin of any hope of treating rustlib as a more general 'global assembly cache'.

brson · 2016-06-21T00:26:04Z

cc @jauhien @vadimcn @sylvestre packaging people. This PR is trying to finalize the storage location for the Rust source code (#19535), for use by various tools, and I want to check what impact it might have on packagers. In rustup it'll be an optional package called rust-src, and tentatively installed to lib/rustlib/rust-src.

The concern I have regarding distro packaging is the impact on side-by-side installation. In particular, if SxS is achieved by just installing multiple revisions of Rust directly on top of each other then this src location doesn't work because multiple revisions will want to put the source there.

From previous conversations with @jauhien my impression was that at least Gentoo had a more sophisticated SxS installation scheme of their own. What I would like to do for simplicity is abandon any pretense that rustlib is shared by multiple versions of Rust. Let distros deal with SxS installation their own way and reexamine how it impacts us when the time comes.

Of course distros may not want to use our scheme for installing the source at all, but then they are going to have to deal with the integration with tools separately.

Diggsey · 2016-06-21T00:59:15Z

I've updated rustbuild to produce the two source distributions, and fixed the typo in rustlib.

durka · 2016-06-21T01:04:19Z

src/bootstrap/build/dist.rs

@@ -287,6 +289,97 @@ pub fn std(build: &Build, compiler: &Compiler, target: &str) {
    t!(fs::remove_dir_all(&image));
 }

+/// Creates the `rust-std` installer component as compiled by `compiler` for the
+/// target `target`.


copy-pasted comment?

alexcrichton · 2016-06-21T03:04:04Z

I'd personally lean a bit towards rustlib/src/rust instead of rustlib/rust-src just to allow for Cargo one day (and perhaps other tools like rustfmt), but I don't think we'll go too far in the directon of rustlib/rust-core-src as it'd basically just be a space optimization.

MarkSwanson · 2016-06-21T04:58:03Z

I'm glad this is being worked on!
I like rustlib/src/rust.
I like making it easy to perhaps add the sources for cargo / rustfmt etc. Maybe llvm headers. clippy? etc.

brson · 2016-06-23T17:43:51Z

Let's go with rustlib/src/rust and keep pressing forward.

sylvestre · 2016-06-24T11:24:27Z

Sorry for the dumb question but what is SxS ?
Besides that, we don't ship the sources as a package in Debian. We are using the tag from github to get the tarball.

sorear · 2016-06-25T19:27:55Z

@sylvestre side-by-side installation, in other words having more than one version of rust (e.g. nightly and stable) installed and usable on the same machine at the same time

bors · 2016-07-02T13:32:10Z

☔ The latest upstream changes (presumably #34611) made this pull request unmergeable. Please resolve the merge conflicts.

brson · 2016-07-08T22:08:10Z

Something to consider re the std-aware cargo RFC: in order to teach cargo how to build std, we're most likely going to end up having cargo look in the sysroot for the source - that is, the source we're packaging in this PR.

So far we've not made any particular commitments about what the 'shape' of that source will like, we're just dumping the source tree into rustlib/src/rust. ISTM though that Cargo might want a deterministic scheme for finding crates with a specific name, so that if it learns that e.g. collections is part of the crate tag, it knows to look in a specific directory for a Cargo.toml corresponding to the collections package.

ISTM there's also some overlap here with the idea of 'local crates.io mirrors' as well. @alexcrichton does it make sense for Cargo to treat part of the sysroot as a standard local crates.io mirror?

I guess I see a few options here:

we can go forward with this 'dump' of the raw source into the sysroot and expect tools to just kinda do there best (I imagine they would have a two stage lookup: first, use hardcoded knowledge of the official Rust source tree to find a crate; second walk the tree looking for Cargo.toml files)
we can instead plan a more structured layout for source in the sysroot where tools know exactly how to find the source to a specific crate
we can go ahead with the current patch but plan on adding the additional structure later.
a combination of the above, but structuring the source in the sysroot to fit Cargo's hypothetical 'local registry' layout.

For the sake of expediency I think I'm inclined to just keep going forward with this patch and force tools like Cargo to do their bust to locate the crates in the source tree.

cc @Ericson2314

brson · 2016-07-08T22:08:26Z

@Diggsey what are the prospects for whipping this patch back into shape?

Diggsey · 2016-07-08T22:30:19Z

Just need to fix the merge conflicts then in that case.

eternaleye · 2016-07-08T22:53:47Z

@brson: Personally, I'd prefer option 3 - roll this out as is, but plan on adding structure.

In the end, that results in the same two-phase lookup being written into tools, but rather than privileging rustc (and making every other compiler implementation take a perf hit unless they mimic a historical accident, and one that may change in the future if rustc's repo changes anything), it makes the fallback less and less frequently-hit over time.

It's just as expedient for Rust (no extra work before rollout), more expedient for Cargo (implement only the "look for Cargo.toml" path now), and doesn't privilege rustc over future implementations (at least one is already being written as mrustc.)

alexcrichton · 2016-07-09T01:26:43Z

@brson this is basically already in the shape of a "path source" that Cargo expects today (or a git repository), so it should interoperate regardless. If we want to work with a local registry or vendored directory then we'd probably want more structure, but as you say I'm fine adding that later.

Diggsey · 2016-07-09T12:49:56Z

Fixed conflicts.

Diggsey · 2016-08-09T23:54:58Z

Failure looks to be spurious, and was caused by remove_dir_all being broken on windows (#29497)

brson · 2016-08-10T00:08:56Z

@bors retry

bors · 2016-08-10T02:12:49Z

⌛ Testing commit 777bd6f with merge 0eefd73...

bors · 2016-08-10T04:00:19Z

💔 Test failed - auto-mac-64-opt-rustbuild

Diggsey · 2016-08-11T00:16:51Z

Ugh... That should fix it for mac. 🍎 🔫

brson · 2016-08-11T00:32:46Z

@bors r+

bors · 2016-08-11T00:32:47Z

📌 Commit 511c0d1 has been approved by brson

bors · 2016-08-11T02:24:17Z

⌛ Testing commit 511c0d1 with merge 7f6705d...

bors · 2016-08-11T04:42:48Z

💔 Test failed - auto-mac-64-opt-rustbuild

alexcrichton · 2016-08-11T16:06:05Z

@bors: retry

bors · 2016-08-12T02:37:01Z

⌛ Testing commit 511c0d1 with merge 1447527...

bors · 2016-08-12T05:10:14Z

💔 Test failed - auto-mac-64-opt-rustbuild

…a tarball Copy source files from rust code Add missing wildcard Remove unused function Remove use of tar --transform

Diggsey · 2016-08-12T22:27:08Z

I think this is caused by the addition of the metadata section and package checksums to the Cargo.lock format in new versions of cargo, but I have no idea why other PRs aren't affected, or why a different version of cargo would be being used.

Mark-Simulacrum · 2016-08-12T22:35:27Z

Other PRs wouldn't be affected if they didn't change Cargo.toml as far as I know; I think this is true of most PRs.

alexcrichton · 2016-08-12T23:44:57Z

Lots of Cargo.lock exists that shouldn't in those logs, for example libpanic_unwind/Cargo.lock shouldn't ever exist. I wonder how they're being created?

The Cargo in use to generate these is pinned, it's downloaded as part of the bootstrap process

Diggsey · 2016-08-13T00:38:14Z

@alexcrichton Those files exist in the repository...

Diggsey · 2016-08-13T22:06:21Z

This should fix the tidy check.

There were two problems:

This specific check did not work on windows. msys's git diff-index ignores filenames with backslashes in, so the files would always show as unchanged. I fixed this by replacing backslashes with forward slashes before passing it to git.
git diff-index produces false positives after running make dist. I still haven't figured out why this happens, but using git diff instead fixes the problem.

alexcrichton · 2016-08-14T07:07:20Z

@bors: r=brson b3908d0

bors · 2016-08-14T16:34:18Z

⌛ Testing commit b3908d0 with merge 92ae4ce...

Produce source package in rust-installer format See rust-lang-deprecated/rust-buildbot#102 There may be a better way to do this, wasn't sure how to clean-up the `rust-src-image` directory when it's used by multiple make rules.

bors · 2016-08-14T19:28:46Z

Diggsey · 2016-08-14T19:29:33Z

🎆

rust-highfive assigned brson Jun 19, 2016

brson reviewed Jun 20, 2016
View reviewed changes

durka reviewed Jun 21, 2016
View reviewed changes

Diggsey force-pushed the rust-src-pkg branch from fc5b57c to bb938d1 Compare June 23, 2016 18:18

Diggsey force-pushed the rust-src-pkg branch from bb938d1 to 81673eb Compare June 25, 2016 11:55

Diggsey force-pushed the rust-src-pkg branch from 81673eb to 21ccad9 Compare July 9, 2016 12:36

Diggsey force-pushed the rust-src-pkg branch from 777bd6f to 511c0d1 Compare August 11, 2016 00:13

Produce source package in rust-installer format in addition to vanill…

7341d68

…a tarball Copy source files from rust code Add missing wildcard Remove unused function Remove use of tar --transform

Fix make-tidy lock file checks

b3908d0

Diggsey force-pushed the rust-src-pkg branch from 511c0d1 to b3908d0 Compare August 13, 2016 21:41

bors merged commit b3908d0 into rust-lang:master Aug 14, 2016

Produce source package in rust-installer format #34366

Produce source package in rust-installer format #34366

Conversation

Diggsey commented Jun 19, 2016

rust-highfive commented Jun 19, 2016

alexcrichton commented Jun 19, 2016

Diggsey commented Jun 19, 2016

alexcrichton commented Jun 20, 2016

brson Jun 20, 2016

Choose a reason for hiding this comment

brson commented Jun 20, 2016

brson commented Jun 20, 2016

brson commented Jun 20, 2016

brson commented Jun 21, 2016

Diggsey commented Jun 21, 2016

durka Jun 21, 2016

Choose a reason for hiding this comment

Diggsey Jun 21, 2016

Choose a reason for hiding this comment

alexcrichton commented Jun 21, 2016

MarkSwanson commented Jun 21, 2016

brson commented Jun 23, 2016

sylvestre commented Jun 24, 2016

sorear commented Jun 25, 2016

bors commented Jul 2, 2016

brson commented Jul 8, 2016

brson commented Jul 8, 2016

Diggsey commented Jul 8, 2016

eternaleye commented Jul 8, 2016

alexcrichton commented Jul 9, 2016

Diggsey commented Jul 9, 2016

Diggsey commented Aug 9, 2016

brson commented Aug 10, 2016

bors commented Aug 10, 2016

bors commented Aug 10, 2016

Diggsey commented Aug 11, 2016

brson commented Aug 11, 2016

bors commented Aug 11, 2016

bors commented Aug 11, 2016

bors commented Aug 11, 2016

alexcrichton commented Aug 11, 2016

bors commented Aug 12, 2016

bors commented Aug 12, 2016

Diggsey commented Aug 12, 2016

Mark-Simulacrum commented Aug 12, 2016

alexcrichton commented Aug 12, 2016

Diggsey commented Aug 13, 2016

Diggsey commented Aug 13, 2016

alexcrichton commented Aug 14, 2016

bors commented Aug 14, 2016

bors commented Aug 14, 2016

Diggsey commented Aug 14, 2016