Stack overflow under Valgrind 'rc_mlx5/uct_p2p_err_test.local_access_error/0'

[evgeny-leksikov]

http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/1700/label=master,worker=2/console

Also reproducible with pure master (6b29f42)

configure --enable-debug --enable-gtest --with-valgrind --enable-profiling --enable-frame-pointer --enable-stats --enable-memtrack --enable-fault-injection --enable-debug-data --enable-mt 
make GTEST_FILTER="rc_mlx5/uct_p2p_err_test.local_access_error/0" -C ./test/gtest/ test_valgrind  
...
#0  0x0000000005f53ca9 in waitpid () from /usr/lib64/libpthread.so.0
#1  0x0000000004e80062 in ucs_debugger_attach () at ../../../src/ucs/debug/debug.c:728
#2  0x0000000004e804d4 in ucs_error_freeze (
    error_type=0x4f293ec "address not mapped to object",
    message=0x40453e0 "Caught signal 11 (Segmentation fault: address not mapped to object at address 0xffe801ff8)") at ../../../src/ucs/debug/debug.c:847
#3  0x0000000004e80c96 in ucs_handle_error (
    error_type=0x4f293ec "address not mapped to object",
    message=0x4f29540 "Caught signal %d (%s: %s%s)") at ../../../src/ucs/debug/debug.c:1018
#4  0x0000000004e808d3 in ucs_debug_handle_error_signal (signo=11,
    cause=0x4f293ec "address not mapped to object", fmt=0x4f2955d " at address %p")
    at ../../../src/ucs/debug/debug.c:958
#5  0x0000000004e80a19 in ucs_error_signal_handler (signo=11, info=0x4045c70,
    context=0x4045b40) at ../../../src/ucs/debug/debug.c:981
#6  <signal handler called>
#7  0x00000000054a9870 in ucm_malloc_is_address_in_heap (ptr=0x9aba8f0)
    at ../../../src/ucm/malloc/malloc_hook.c:169
#8  0x00000000054a98cb in ucm_malloc_address_remove_if_managed (ptr=0x9aba8f0,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:180
#9  0x00000000054a9d5d in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:248
#10 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307
#11 0x00000000054a9bb8 in ucm_release_foreign_block (ptr=0x9aba8f0,
    orig_free=0x40b2b0 <free@plt>, debug_name=0x54b6d09 "free")
    at ../../../src/ucm/malloc/malloc_hook.c:216
#12 0x00000000054a9d98 in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:251
#13 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307
#14 0x00000000054a9bb8 in ucm_release_foreign_block (ptr=0x9aba8f0,
    orig_free=0x40b2b0 <free@plt>, debug_name=0x54b6d09 "free")
    at ../../../src/ucm/malloc/malloc_hook.c:216
#15 0x00000000054a9d98 in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:251
#16 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307

[evgeny-leksikov]

git bisect says 778d47b is the first bad commit

[yosefe]

it probably just exposed the issue, the problem is that 'free' points to free@plt instead of the real 'free' function

[FaDee1]

http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/1700/label=master,worker=2/console

ทำซ้ำได้ด้วยต้นแบบบริสุทธิ์ ( 6b29f42 )

configure --enable-debug --enable-gtest --with-valgrind --enable-profiling --enable-frame-pointer --enable-stats --enable-memtrack --enable-fault-injection --enable-debug-data --enable-mt 
make GTEST_FILTER="rc_mlx5/uct_p2p_err_test.local_access_error/0" -C ./test/gtest/ test_valgrind  
...
#0  0x0000000005f53ca9 in waitpid () from /usr/lib64/libpthread.so.0
#1  0x0000000004e80062 in ucs_debugger_attach () at ../../../src/ucs/debug/debug.c:728
#2  0x0000000004e804d4 in ucs_error_freeze (
    error_type=0x4f293ec "address not mapped to object",
    message=0x40453e0 "Caught signal 11 (Segmentation fault: address not mapped to object at address 0xffe801ff8)") at ../../../src/ucs/debug/debug.c:847
#3  0x0000000004e80c96 in ucs_handle_error (
    error_type=0x4f293ec "address not mapped to object",
    message=0x4f29540 "Caught signal %d (%s: %s%s)") at ../../../src/ucs/debug/debug.c:1018
#4  0x0000000004e808d3 in ucs_debug_handle_error_signal (signo=11,
    cause=0x4f293ec "address not mapped to object", fmt=0x4f2955d " at address %p")
    at ../../../src/ucs/debug/debug.c:958
#5  0x0000000004e80a19 in ucs_error_signal_handler (signo=11, info=0x4045c70,
    context=0x4045b40) at ../../../src/ucs/debug/debug.c:981
#6  <signal handler called>
#7  0x00000000054a9870 in ucm_malloc_is_address_in_heap (ptr=0x9aba8f0)
    at ../../../src/ucm/malloc/malloc_hook.c:169
#8  0x00000000054a98cb in ucm_malloc_address_remove_if_managed (ptr=0x9aba8f0,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:180
#9  0x00000000054a9d5d in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:248
#10 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307
#11 0x00000000054a9bb8 in ucm_release_foreign_block (ptr=0x9aba8f0,
    orig_free=0x40b2b0 <free@plt>, debug_name=0x54b6d09 "free")
    at ../../../src/ucm/malloc/malloc_hook.c:216
#12 0x00000000054a9d98 in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:251
#13 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307
#14 0x00000000054a9bb8 in ucm_release_foreign_block (ptr=0x9aba8f0,
    orig_free=0x40b2b0 <free@plt>, debug_name=0x54b6d09 "free")
    at ../../../src/ucm/malloc/malloc_hook.c:216
#15 0x00000000054a9d98 in ucm_free_impl (ptr=0x9aba8f0, orig_free=0x40b2b0 <free@plt>,
    debug_name=0x54b6d09 "free") at ../../../src/ucm/malloc/malloc_hook.c:251
#16 0x00000000054aa001 in ucm_free (ptr=0x9aba8f0, caller=0x40b2b0 <free@plt>)
    at ../../../src/ucm/malloc/malloc_hook.c:307