libgcc_s.so is missing from glibc when cross compiling

[lopsided98]

Issue description

I'm not sure what the intended behavior actually is, but there is an issue with libgcc_s.so when cross-compiling, or perhaps the native case is actually incorrect.

I came across this problem when debugging why tmon crashes with the error: libgcc_s.so.1 must be installed for pthread_cancel to work when exiting, but only when it is cross-compiled. I noticed that, for native builds only, glibc copies libgcc_s.so from bootstrap gcc. This libgcc_s.so from the bootstrap tools is present in the glibc used to build all other derivations, which does not seem right to me (it doesn't seem like normal derivations should have access to any libraries/binaries from the the bootstrap tools).

pthreads needs libgcc_s.so, and it is able to find it when built natively because libgcc_s.so is part of glibc, which is on the RPATH. When cross-compiling, glibc does not have libgcc_s.so, and the gcc libs are not on the RPATH, so pthreads cannot find libgcc_s.so.

I can see that there are a number of cases where people have worked around this problem in nixpkgs, by manually linking to libgcc_s. This solves the problem for tmon, but I still think there is a bug because of the different behavior between native and cross builds.

cc @vcunat

[vcunat]

See #36948.

[dtzWill]

We really should try to get this fixed for next release! Please :).

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[trofi]

I stumbled on it today as well. I think I vaguely understand why loss of libgcc_s.so.1 happens. I'd say it's a bug. Native and cross builds should both have (or both not have) libgcc_s.so.1. As native does have it let's try to alight cross to it (unless it's too hard, then we can try the other way around).

The real project to provide libgcc_s.so.1 is gcc. The reason we seem to copy the file into glibc is to fool bootstrapTools (and probably gcc itself) to workaround missing RPATHs for binaries and libraries. Let's ignore the complications and focus on glibc's way to pull libgcc_s.so in.

Native case pulls libgcc_s.so.1 in glibc derivation in preInstall from ${stdenv.cc.cc}/lib/libgcc_s.so.1:

nixpkgs/pkgs/development/libraries/glibc/default.nix

Lines 69 to 76 in b3da007

	preInstall = ''
	if [ -f ${stdenv.cc.cc}/lib/libgcc_s.so.1 ]; then
	mkdir -p $out/lib
	cp ${stdenv.cc.cc}/lib/libgcc_s.so.1 $out/lib/libgcc_s.so.1
	# the .so It used to be a symlink, but now it is a script
	cp -a ${stdenv.cc.cc}/lib/libgcc_s.so $out/lib/libgcc_s.so
	fi
	'';

A few notes:

• it always happens (regardless of bootstrap stages or not)
• it uses gcc's ${stdenv.cc.cc}/lib/libgcc_s.so.1 path

Cross case goes a great length to break this mechanism. A few things I noticed:

• The path is invalid if gcc is a cross-compiler (build != host). The correct path would be something like ${stdenv.cc.cc}/${stdenv.hostPlatform.config}/lib/libgcc_s.so.1
• preInstall is completely removed for cross:

  nixpkgs/pkgs/development/libraries/glibc/common.nix

  Line 295 in b3da007

  preInstall = null; # clobber the native hook

  (probably, out of fear to copy library for a wrong target?)
• [I'm not very certain here] Cross-built glibc (say, via pkgsCross.aarch64-multiplatform.glibc or via pkgsCross.aarch64-multiplatform.stdenv.cc.libc) both use gcc without shared library support. Thus gcc does not provide any libgcc_s.so.1 at that stage.
• make-bootstrap-tools also assumes non-cross paths even for cross builds:

  nixpkgs/pkgs/stdenv/linux/make-bootstrap-tools.nix

  Line 130 in b3da007

  cp -d ${bootGCC.lib}/lib/libgcc_s.so* $out/lib

  (uses cross-built gcc, that has libgcc_s.so.1 in expected location)

I 'll try to tackle a few of these problems.

[trofi]

Oh, it's even trickier: ${stdenv.cc.cc}/lib/libgcc_s.so.1 is a valid path only for bootstrapTools's gcc (as nixpkgs's gcc install those files to ${stdenv.cc.cc.lib}/lib/libgcc_s.so.1 output). gcc never gets built by nixpkgs gcc:

$ nix build -L --impure --expr 'with import <nixpkgs> {}; runCommandCC "hm?" {} "gcc --version | head -n 1; strings ${stdenv.cc.libc}/lib/libc.so.6 | fgrep GCC"'
hm?> gcc (GCC) 11.3.0
hm?> GCC: (GNU) 8.3.0

Here we see that on a system with gcc-11 as a default glibc is built by gcc-8.3.0 from bootstrapTools (and reuses it's libgcc_s.so.1). It's not even a gcc-static (used by cross builds to get an equivalent).

[trofi]

Removing libgcc_s.so.1 from glibc also needs a bit of work. Added TODOs in #187225

[vcunat]

This is most likely resolved now by PR #209870 or related changes. Or at least it needs more concrete reformulation of what's wrong, based on the changed situation.

[vcunat]

Ah, it still doesn't find the library, at least by the reproducer described in #213453 (so let's keep that one open)

[ghost]

The complete fix is in #238154