-
-
Notifications
You must be signed in to change notification settings - Fork 13.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: GCC with separated runtime libraries #132343
base: master
Are you sure you want to change the base?
Conversation
Taking a bunch of stuff from 11 packages
* `builtins.attrNames gccNgPackages_11` now evaluates * Added missing inputs * Removed a bunch of clang-related things - clang-only packages: compiler-rt, libcxx, libcxxabi, openmp - useLLVM logic - stdenvs / cc-wrappers which are not applicable anymore * Removed LLVM-specific relics like the default callPackage bindings, clean up default callPackage scopes for libraries and tools * Add runtime libraries (may forgot some) to libraries set
Oh no, I forgot to push my stuff! I'll merge. Actually comments won't be too bad. |
Create patch so that thread model can be specified explicitly rather than being inferred from GCC.
There is a still a build failure, but I am going to just use Musl and ignore this for now.
Unfortunately, libstdcxx doesn't link, evidentally because inconsistent `-fPIC`.
mkdir -p "$out/lib" "$dev/include" | ||
ln -s "$out/lib/gcc/${stdenv.hostPlatform.config}/${version}"/* "$out/lib" | ||
ln -s "$dev/lib/gcc/${stdenv.hostPlatform.config}/${version}/include"/* "$dev/include/" | ||
''; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Meta is missing.
I marked this as stale due to inactivity. → More info |
Unmark stale |
Hi, It is a bummer that this sort of fell by the wayside. I just wanted to mention that enabling the goals of this PR is one of the major benefits sought by #209870. Specifically, if that PR merges, my next project will be to start moving the libraries that are bundled with gcc into separate packages -- like this PR, but one at a time, and incrementally rather than in a big rewrite. I would like to reuse as much of the work here as possible. |
https://github.com/Ericson2314/nixpkgs/tree/gcc-ng-12 This was me trying to update this to a new GCC. |
@amjoseph-nixpkgs #132343 (comment) it is a bummer, and I would love to pair with you on this to get it over the hill at some point. there was just some things I was banging my head against the wall debugging that I need another pair of eyes on. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-10-31-nixpkgs-architecture-team-meeting-45/34846/1 |
#### Summary By default, when you type `make`, GCC will compile itself three times. This PR inhibits that behavior by configuring GCC with `--disable-bootstrap`, and reimplements the triple-rebuild using Nix rather than `make`/`sh`. #### Immediate Benefits - Allow `gcc11` and `gcc12` on `aarch64` (without needing new `bootstrapFiles`) - Faster stdenv rebuilds: the third compilation of gcc (i.e. stageCompare) is no longer a `drvInput` of the final stdenv. This allows Nix to build stageCompare in parallel with the rest of nixpkgs instead of in series. - No more copying `libgcc_s` out of the bootstrap-files or other derivations - No more Frankenstein compiler: the final gcc and the libraries it links against (mpfr, mpc, isl, glibc) are all built by the same compiler (xgcc) instead of a mixture of the bootstrapFiles' compiler and xgcc. - No more [static lib{mpfr,mpc,gmp,isl}.a hack] - Many other small `stdenv` hacks eliminated - `gcc` and `clang` share the same codepath for more of `cc-wrapper`. #### Future Benefits - This should allow using a [foreign] `bootstrap-files` so long as `hostPlatform.canExecute bootstrapFiles`. - This should allow each of the libraries that ship with `gcc` (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty, offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be built in separate (one-liner) derivations which `inherit src;` from `gcc`, much like NixOS#132343 #### Incorporates - NixOS#210004 - NixOS#36948 (unreverted) - NixOS#210325 - NixOS#210118 - NixOS#210132 - NixOS#210109 - NixOS#213909 - NixOS#216136 - NixOS#216237 - NixOS#210019 - NixOS#216232 - NixOS#216016 - NixOS#217977 - NixOS#217995 #### Closes - Closes NixOS#108305 - Closes NixOS#108111 - Closes NixOS#201254 - Closes NixOS#208412 #### Credits This project was made possible by three important insights, none of which were mine: 1. @Ericson2314 was the first to advocate for this change, and probably the first to appreciate its advantages. Nix-driven (external) bootstrap is "cross by default". 2. @trofi has figured out a lot about how to get gcc to not mix up the copy of `libstdc++` that it depends on with the copy that it builds, by moving the `bootstrapFiles`' `libstdc++` into a [versioned directory]. This allows a Nix-driven bootstrap of gcc without the final gcc would still having references to the `bootstrapFiles`. 3. Using the undocumented variable [`user-defined-trusted-dirs`] when building glibc. When glibc `dlopen()`s `libgcc_s.so`, it uses a completely different and totally special set of rules for finding `libgcc_s.so`. This trick is the only way we can put `libgcc_s.so` in its own separate outpath without creating circular dependencies or dependencies on the bootstrapFiles. I would never have guessed to use this (or that it existed!) if it were not for a [comment in guix] which @Mic92 [mentioned]. My own role in this PR was basically: being available to go on a coding binge at an opportune moment, so we wouldn't waste a [crisis]. [aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938 [amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857 [nonexistent sysroot]: NixOS#210004 [versioned directory]: NixOS#209054 [`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html [comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253 [mentioned]: NixOS#210112 (comment) [crisis]: NixOS#108305 [foreign]: NixOS#170857 (comment) [static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
Motivation for this change
Opening bare bones PR to collaborate on #132340
cc @Ericson2314 @sternenseemann
Things done
All cross compilation from
x86_64-unknown-linux-{gnu,musl}
. Plug in the config in--crossSystem "{ config = \"$config\"; useGccNg = true; }"
buildPackages.gccNgPackages.gcc-unwrapped
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
gccNgPackages.libgcc
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
libcCross
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
gccNgPackages.libatomic
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
gccNgPackages.libstdcxx
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
-fPIC
and-fno-PIC
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)Upstream patches
Prefixed binaries in
-B
dirsThread model specified per lib