Problems with cargo-bazel and cargo when using rules_rust crates_universe (bzlmod)

[sirbeancounter]

This is written bearing in mind that the rules_rust documentation states the following regarding Bzlmod:

Note that rules_rust bzlmod support is still a work in progress. Most features should work, but bugs are more likely. This is not a desired end-state - please report (or better yet, help fix!) bugs you run into.

Background

We have developed a tool in Rust that we would like to use in a large project. This tool generates C++ source code files from an input and the tool is working as want it to. The large project uses bazel for its build system so we decided to start using bazel and rules_rust as the build system for this tool. That now works and the build WORKSPACE is defined using a MODULE.bazel file (bzlmod) instead of a WORKSPACE since bazel will deprecate the latter some time in the future, and the larger project has migrated away from WORKSPACE file to a MODULE.bazel file. This tool is now effectively a bazel module with module() being declared in it's MODULE.bazel file. The large project is able to declare this tool as a module using bazel_dep() in the consuming repository and, if it declares the rust and rust_host_tools toolchains, the tool is available for use in BUILD files for the consuming repository. So far so good. Rules in BUILD files that make use of the tool builds the output as expected for regular users in their environments such that they don't depend upon their own rust toolchain environments.

Problem Description

However we encounter problems when pushing this integration to our CI pipeline where we encounter problems cargo-bazel, a binary built from Rust sources in rules_rust, and we have not been able to isolate the problem, for which an issue #2874 has been raised.

That problem was detected in a production build environment where $USER is an anonymous functional user without a regular home directory at /home/$USER but instead has a home directory set to some other value such as /repo/$USER/project/build. For some reason this provokes the failing case seen in issue #2874.

Questions

1. What is the root cause of the problem detected by cargo-bazel/cargo as seen in issue #2874?
2. How should CARGO_HOME be set when we don't want the cargo home directory to be located at /home/$USER/.cargo?
3. Is the value of CARGO_BAZEL_ISOLATED, as raised in issue #2867 taken into account?

The problem can be easily recreated and we can use that environment for further debugging and testing. Any help to better understand the problems we are encountering would be really appreciated.