Set Up Your RPi as a distcc Client

Let your RPi4 (or RPi3) invoke remote cross-compilation on your PC, via distcc, and massively speed up the build process!

Prerequisites

This guide assumes that you have already set up your PC as a distcc server as detailed here. So, if you haven't yet, please do this first, before proceeding.

Configuring the RPi4/3 as a distcc Client

In what follows, I'm going to assume your PC is at the fixed address 192.168.1.116; obviously, adapt the instructions as appropriate to match your setup.

As we are using a stable-branch compiler on the PC, but the RPi image uses the 'testing' (aka ~arm64) branch by default, begin by downgrading your gcc compiler (and binutils, for consistency) on the RPi to the 'stable' version (this version is also available on the binhost, so the following process shouldn't take long). Issue:

pi64 ~ # echo "sys-devel/binutils -~arm64" >> /etc/portage/package.accept_keywords/binutils
pi64 ~ # echo "sys-devel/gcc -~arm64" >> /etc/portage/package.accept_keywords/gcc
pi64 ~ # emerge -u sys-devel/binutils sys-devel/gcc
pi64 ~ # gcc-config --list-profiles

Take a note of the index number of the 'stable branch' version of the compiler returned by the last command above (this will probably be 1), and then ensure the profile is set (substitute it for 1 in the below, if different):

pi64 ~ # gcc-config 1
pi64 ~ # env-update && source /etc/profile
pi64 ~ # emerge --oneshot sys-devel/libtool

Next, if you haven't already got distcc on your RPi (it is pre-installed on this project's image), begin by emerging it now. Issue:

pi64 ~ # emerge --ask --verbose sys-devel/distcc

Then, tell distcc where the server is; edit the file /etc/distcc/hosts (on the RPi) and append the line:

192.168.1.116/10,cpp,lzo

Note - you can of course set up multiple servers here, but that's beyond the scope of this short guide; see the distcc homepage for further details. Note also that with a relatively slow system like the RPi3, I would not recommend adding localhost to the hosts list. It will fall back to local compilation where necessary in any event. By contrast, on an RPi4, particularly if you have the 2GiB or 4GiB variant (and sufficient cooling), localhost may safely be used.

Note - the /10 is nothing to do with the IP address (confusingly); it is the maximum number of simultaneous jobs to send to that server (here, 10). The default is 4 if unspecified. Adjust to suit your particular distcc server's capabilities.

Next, edit the file /etc/portage/make.conf (on the RPi4/3), and append (or uncomment, if already present) the following line:

FEATURES="${FEATURES} distcc"

That's it! You're now ready to roll.

Using distcc

Now you have everything set up, using distributed cross-compilation from your RPi4 (or RPi3) is really easy. In fact, with the FEATURES variable set as above, emerge will automatically make use of it, so the next time you build a package or update your system, you should notice a significant speed-up, without having to do anything further.

To use distcc when performing a normal make, simply prefix the command with pump. So for example:

pi64 mycproject # pump make -j10

Remember to specify a reasonably large job number with using make with distcc as shown above, to maximize the remote processing capacity. That's it!

Monitoring distcc

You can check on distcc's operation at the client end with:

pi64 ~ # watch DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text

and at the server, most simply, by

gentoo_pc ~ # journalctl -f --unit=distccd

if using systemd, or

gentoo_pc ~ # tail -f /var/log/messages | grep distccd --line-buffered

if using OpenRC.

Troubleshooting

Distributed cross-compilation with distcc/crossdev is neat, but it is not a magic bullet. For example:

• some parts of the build process (such as autotools ./configure) must still run locally on the RPi (and are considerably time-consuming);
• some packages (like sys-devel/gcc) rely on running tools that were created during the build, as part of the build; such operations cannot be distributed with distcc; and
• languages other than C and C++ are not supported by distcc.

In addition, you will occasionally encounter packages that do not build correctly under distcc (although this is quite rare). You can simply build locally in such cases (and force this to happen in future by adding an entry to /etc/portage/package.env/<pkgname> - see e.g. /etc/portage/package.env/numpy on the image, for an example of this).

As to other issues, this is only a short guide, and you should read the notes on the distcc site and Gentoo wiki if you experience problems. One important point that is worth mentioning here though - you must ensure that gcc is the same version on your RPi and in your crossdev environment on the PC; bad things are almost guaranteed to happen if not ><