Preloaded libFuzzer doesn't allow to use custom mutator.

[kobrineli]

Hi! We've been fuzzing with atheris and faced the problem that when we use LD_PRELOAD=/path/to/preload/asan_with_fuzzer.so for fuzzing with C extensions, the custom mutator that is written in python code is not linked and as a result is not used.

I will demonstrate the problem on this atheris example.
When I run the target like this: /custom_mutator_example.py, I get the following:

INFO: Using built-in libfuzzer 
WARNING: Failed to find function "__sanitizer_acquire_crash_state".
WARNING: Failed to find function "__sanitizer_print_stack_trace".
WARNING: Failed to find function "__sanitizer_set_death_callback".    
INFO: found LLVMFuzzerCustomMutator (0x7ffff767d9b0). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 352984491
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 35Mb

And the LLVMFuzzerCustomMutator is found.

When I run LD_PRELOAD="/usr/local/lib/python3.8/dist-packages/asan_with_fuzzer.so" /custom_mutator_example.py, I get this:

INFO: Using preloaded libfuzzer
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 129126802
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 44Mb

And the LLVMFuzzerCustomMutator is not found and not used.

How can I use a custom mutator while using asan_with_fuzzer.so for external C extensions?

[Nambers]

I found a workaround for this issue, you need to use patchelf to manually link the dependenices (asan library) instead of LD_PRELOAD like:

#!/bin/bash
set -e

PYTHON_PKGS_PATH=$(nix-shell --pure --command "echo \$PYTHON_PKGS_PATH" ./scripts/cpython.nix)
PYTHON_PATH=$(nix-shell --pure --command "echo \$PYTHON_PATH" ./scripts/cpython.nix)
echo PYTHON_PKGS_PATH=$PYTHON_PKGS_PATH
echo PYTHON_PATH=$PYTHON_PATH

ACTION=--add-needed

if [[ ! -z "$1" && $1 == "remove" ]]; then
    ACTION=--remove-needed
    echo "Remove mode"
fi

# build ouputs
patchelf $ACTION $PYTHON_PKGS_PATH/asan_with_fuzzer.so ./build/pyFuzzerHelper.so
patchelf $ACTION $PYTHON_PKGS_PATH/asan_with_fuzzer.so ./build/pyFuzzerTarget.so

# bc PYTHON_PKGS_PATH and PYTHON_PATH under /nix/store is read-only, otherwise sudo is not needed
# ensure ASAN symbols in custom_mutator
sudo patchelf $ACTION $PYTHON_PKGS_PATH/asan_with_fuzzer.so $PYTHON_PKGS_PATH/atheris/custom_mutator.cpython-*.so
# ensure LLVM symbols exported
sudo patchelf $ACTION $PYTHON_PKGS_PATH/atheris/custom_mutator.cpython-*.so $PYTHON_PATH/bin/python

and pass internal_libfuzzer=True in atheris.Setup:

atheris.Setup(sys.argv, TestOneInput, custom_mutator=CustomMutator, internal_libfuzzer=True)

then use preload to load ASAN beforehand, like

nix-shell --pure --command "LD_PRELOAD="\$PYTHON_PKGS_PATH/asan_with_fuzzer.so" \$PYTHON_PATH/bin/python $SRC_PATH/main.py -atheris_runs=5" $SCRIPT_DIR/cpython.nix --argstr py_ver_str $CPYTHON_VERSION