Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PHP-FPM Signal 13 (SIGPIPE) #15872

Open
frederikbosch opened this issue Sep 13, 2024 · 23 comments
Open

PHP-FPM Signal 13 (SIGPIPE) #15872

frederikbosch opened this issue Sep 13, 2024 · 23 comments

Comments

@frederikbosch
Copy link
Contributor

frederikbosch commented Sep 13, 2024

Description

Today I started seeing crashes on one of my PHP-FPM 8.2.23 servers with a SIGPIPE message.

WARNING: [pool www] child 1206 exited on signal 13 (SIGPIPE) after 27.818575 seconds from start

This server was using a fresh (Docker) build that was created 2 days ago. Another PHP 8.2.23 server that was created directly after its release was not seeing the issues.

My guess is that a version of another linked library or extension was upgraded in the meanwhile and that this causes the newer build to crash whereas the older one does not. When I downgraded the crashing server to a 8.2.22 build of one month ago, the errors disappeared.

So I wondered what library or extension might be causing the trouble. The only result I found online on the SIGPIPE error was related to an update in libxml. Maybe I am running into this problem or a similar one too. I will continue investigating and report what I find.

PHP Version

8.2.23 / 8.3.11

Operating System

Alpine 3.18 (8.2.23) / 3.19 (8.3.11)

@frederikbosch
Copy link
Contributor Author

When looking at my development server's logs, which is on PHP 8.3.11, is also running into the error.

@devnexen
Copy link
Member

Hi @frederikbosch

extension was upgraded in the meanwhile

speaking of this, what would be those extensions ? ie php -m

@frederikbosch
Copy link
Contributor Author

frederikbosch commented Sep 13, 2024

[PHP Modules]
bcmath
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gmp
hash
iconv
imagick (using fixed versions, didn't change)
intl (74.1)
json
libxml (2.11.8)
mbstring
mysqlnd
openssl
pcre (10.42)
PDO
pdo_mysql
pdo_sqlite
Phar
posix
random
readline
redis (6.0.2)
Reflection
session
SimpleXML
sockets
sodium
SPL
sqlite3
standard
tokenizer
xdebug
xml (2.11.8)
xmlreader
xmlwriter
xsl (1.1.39)
Zend OPcache
zip
zlib

[Zend Modules]
Xdebug (is disabled in production where the error also occurs)
Zend OPcache

*striking through what has remained the same since a build that does not contain these crashes

@devnexen
Copy link
Member

Thanks. It might be because of xdebug. Would it be possible to disable it, for a while at least, and sees what happens ?

@frederikbosch
Copy link
Contributor Author

In production there is no Xdebug, and that's where I found the error originally. The above is from my dev server that is now running 8.3.11.

@frederikbosch
Copy link
Contributor Author

While I am programming in PHP for decades (since version 3), I still don't know how to debug these kind of crashes.

@devnexen
Copy link
Member

Thanks for the clarifications. Would debugging/launching fpm with GDB be an option ? I won t have time to look into it only later today, but having a backtrace of the calls would help I think.

@frederikbosch
Copy link
Contributor Author

How do I do that? Can I do that with my current setup, or do I need to build with certain flags enabled?

@devnexen
Copy link
Member

devnexen commented Sep 13, 2024

Once FPM is launched, you might be able to attach a gdb instance per child worker. ie gdb -p <FPM child process id>, launching your requests, once SIGPIPE is triggered to a child process, in the related gdb instance typing bt full. Ideally, fpm with debug symbols is better but at least if we get the call traces it might help determine which extension is problematic.

@devnexen
Copy link
Member

I may come up with other ideas in the meantime unless someone else beat me to it :-)

@frederikbosch
Copy link
Contributor Author

Thanks so far, I am building a new container with gdb installed, and with --enabled-debug and without the stripping in my usual builds.

@frederikbosch
Copy link
Contributor Author

frederikbosch commented Sep 14, 2024

  • docker container is started with cap-add SYS_PTRACE
  • FPM is started with gdb php-fpm
  • gdb -p <id> is attached to every fpm process
  • cat /proc/sys/kernel/yama/ptrace_scope gives 0

And I still get:

ERROR: failed to ptrace(ATTACH) child 41: Operation not permitted (1)

@devnexen
Copy link
Member

I see. Did you try adding --privileged when running your container ?

@frederikbosch
Copy link
Contributor Author

frederikbosch commented Sep 14, 2024

Attaching to process 32
Reading symbols from /usr/local/sbin/php-fpm...
Reading symbols from /usr/lib/libreadline.so.8...
(No debugging symbols found in /usr/lib/libreadline.so.8)
Reading symbols from /usr/lib/libiconv.so.2...
(No debugging symbols found in /usr/lib/libiconv.so.2)
Reading symbols from /usr/lib/libxml2.so.2...
(No debugging symbols found in /usr/lib/libxml2.so.2)
Reading symbols from /lib/libssl.so.3...
(No debugging symbols found in /lib/libssl.so.3)
Reading symbols from /lib/libcrypto.so.3...
(No debugging symbols found in /lib/libcrypto.so.3)
Reading symbols from /usr/lib/libsqlite3.so.0...
(No debugging symbols found in /usr/lib/libsqlite3.so.0)
Reading symbols from /lib/libz.so.1...
(No debugging symbols found in /lib/libz.so.1)
Reading symbols from /usr/lib/libcurl.so.4...
(No debugging symbols found in /usr/lib/libcurl.so.4)
Reading symbols from /usr/lib/libonig.so.5...
(No debugging symbols found in /usr/lib/libonig.so.5)
Reading symbols from /usr/lib/libargon2.so.1...
(No debugging symbols found in /usr/lib/libargon2.so.1)
Reading symbols from /lib/ld-musl-x86_64.so.1...
Reading symbols from /usr/lib/debug//lib/ld-musl-x86_64.so.1.debug...
Reading symbols from /usr/lib/libncursesw.so.6...
(No debugging symbols found in /usr/lib/libncursesw.so.6)
Reading symbols from /usr/lib/liblzma.so.5...
(No debugging symbols found in /usr/lib/liblzma.so.5)
Reading symbols from /usr/lib/libcares.so.2...
(No debugging symbols found in /usr/lib/libcares.so.2)
Reading symbols from /usr/lib/libnghttp2.so.14...
(No debugging symbols found in /usr/lib/libnghttp2.so.14)
Reading symbols from /usr/lib/libidn2.so.0...
(No debugging symbols found in /usr/lib/libidn2.so.0)
Reading symbols from /usr/lib/libbrotlidec.so.1...
(No debugging symbols found in /usr/lib/libbrotlidec.so.1)
Reading symbols from /usr/lib/libunistring.so.5...
(No debugging symbols found in /usr/lib/libunistring.so.5)
Reading symbols from /usr/lib/libbrotlicommon.so.1...
(No debugging symbols found in /usr/lib/libbrotlicommon.so.1)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/opcache.so...
Reading symbols from /usr/lib/libgcc_s.so.1...
(No debugging symbols found in /usr/lib/libgcc_s.so.1)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/xdebug.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/bcmath.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/exif.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/gd.so...
Reading symbols from /usr/lib/libpng16.so.16...
(No debugging symbols found in /usr/lib/libpng16.so.16)
Reading symbols from /usr/lib/libjpeg.so.8...
(No debugging symbols found in /usr/lib/libjpeg.so.8)
Reading symbols from /usr/lib/libfreetype.so.6...
(No debugging symbols found in /usr/lib/libfreetype.so.6)
Reading symbols from /usr/lib/libbz2.so.1...
(No debugging symbols found in /usr/lib/libbz2.so.1)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/gmp.so...
Reading symbols from /usr/lib/libgmp.so.10...
(No debugging symbols found in /usr/lib/libgmp.so.10)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/imagick.so...
Reading symbols from /usr/lib/libgomp.so.1...
(No debugging symbols found in /usr/lib/libgomp.so.1)
Reading symbols from /usr/local/lib/libMagickWand-7.Q16HDRI.so.10...
Reading symbols from /usr/local/lib/libMagickCore-7.Q16HDRI.so.10...
Reading symbols from /usr/lib/libX11.so.6...
(No debugging symbols found in /usr/lib/libX11.so.6)
Reading symbols from /usr/lib/libtiff.so.6...
(No debugging symbols found in /usr/lib/libtiff.so.6)
Reading symbols from /usr/lib/libfontconfig.so.1...
(No debugging symbols found in /usr/lib/libfontconfig.so.1)
Reading symbols from /usr/lib/libwebpmux.so.3...
(No debugging symbols found in /usr/lib/libwebpmux.so.3)
Reading symbols from /usr/lib/libwebpdemux.so.2...
(No debugging symbols found in /usr/lib/libwebpdemux.so.2)
Reading symbols from /usr/lib/libwebp.so.7...
(No debugging symbols found in /usr/lib/libwebp.so.7)
Reading symbols from /usr/lib/libXext.so.6...
(No debugging symbols found in /usr/lib/libXext.so.6)
Reading symbols from /usr/lib/librsvg-2.so.2...
(No debugging symbols found in /usr/lib/librsvg-2.so.2)
Reading symbols from /usr/lib/libgobject-2.0.so.0...
(No debugging symbols found in /usr/lib/libgobject-2.0.so.0)
Reading symbols from /usr/lib/libglib-2.0.so.0...
(No debugging symbols found in /usr/lib/libglib-2.0.so.0)
Reading symbols from /usr/lib/libcairo.so.2...
(No debugging symbols found in /usr/lib/libcairo.so.2)
Reading symbols from /usr/lib/libzip.so.5...
(No debugging symbols found in /usr/lib/libzip.so.5)
Reading symbols from /usr/lib/libxcb.so.1...
(No debugging symbols found in /usr/lib/libxcb.so.1)
Reading symbols from /usr/lib/libzstd.so.1...
(No debugging symbols found in /usr/lib/libzstd.so.1)
Reading symbols from /usr/lib/libexpat.so.1...
(No debugging symbols found in /usr/lib/libexpat.so.1)
Reading symbols from /usr/lib/libsharpyuv.so.0...
(No debugging symbols found in /usr/lib/libsharpyuv.so.0)
Reading symbols from /usr/lib/libcairo-gobject.so.2...
(No debugging symbols found in /usr/lib/libcairo-gobject.so.2)
Reading symbols from /usr/lib/libgdk_pixbuf-2.0.so.0...
(No debugging symbols found in /usr/lib/libgdk_pixbuf-2.0.so.0)
Reading symbols from /usr/lib/libgio-2.0.so.0...
(No debugging symbols found in /usr/lib/libgio-2.0.so.0)
Reading symbols from /usr/lib/libpangocairo-1.0.so.0...
(No debugging symbols found in /usr/lib/libpangocairo-1.0.so.0)
Reading symbols from /usr/lib/libpango-1.0.so.0...
(No debugging symbols found in /usr/lib/libpango-1.0.so.0)
Reading symbols from /usr/lib/libffi.so.8...
(No debugging symbols found in /usr/lib/libffi.so.8)
Reading symbols from /usr/lib/libintl.so.8...
(No debugging symbols found in /usr/lib/libintl.so.8)
Reading symbols from /usr/lib/libpcre2-8.so.0...
(No debugging symbols found in /usr/lib/libpcre2-8.so.0)
Reading symbols from /usr/lib/libXrender.so.1...
(No debugging symbols found in /usr/lib/libXrender.so.1)
Reading symbols from /usr/lib/libxcb-render.so.0...
(No debugging symbols found in /usr/lib/libxcb-render.so.0)
Reading symbols from /usr/lib/libxcb-shm.so.0...
(No debugging symbols found in /usr/lib/libxcb-shm.so.0)
Reading symbols from /usr/lib/libpixman-1.so.0...
(No debugging symbols found in /usr/lib/libpixman-1.so.0)
Reading symbols from /usr/lib/libXau.so.6...
(No debugging symbols found in /usr/lib/libXau.so.6)
Reading symbols from /usr/lib/libXdmcp.so.6...
(No debugging symbols found in /usr/lib/libXdmcp.so.6)
Reading symbols from /usr/lib/libgmodule-2.0.so.0...
(No debugging symbols found in /usr/lib/libgmodule-2.0.so.0)
Reading symbols from /lib/libmount.so.1...
(No debugging symbols found in /lib/libmount.so.1)
Reading symbols from /usr/lib/libpangoft2-1.0.so.0...
(No debugging symbols found in /usr/lib/libpangoft2-1.0.so.0)
Reading symbols from /usr/lib/libharfbuzz.so.0...
(No debugging symbols found in /usr/lib/libharfbuzz.so.0)
Reading symbols from /usr/lib/libfribidi.so.0...
(No debugging symbols found in /usr/lib/libfribidi.so.0)
Reading symbols from /usr/lib/libbsd.so.0...
(No debugging symbols found in /usr/lib/libbsd.so.0)
Reading symbols from /lib/libblkid.so.1...
(No debugging symbols found in /lib/libblkid.so.1)
Reading symbols from /usr/lib/libgraphite2.so.3...
(No debugging symbols found in /usr/lib/libgraphite2.so.3)
Reading symbols from /usr/lib/libmd.so.0...
(No debugging symbols found in /usr/lib/libmd.so.0)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/intl.so...
Reading symbols from /usr/lib/libicuio.so.74...
(No debugging symbols found in /usr/lib/libicuio.so.74)
Reading symbols from /usr/lib/libicui18n.so.74...
(No debugging symbols found in /usr/lib/libicui18n.so.74)
Reading symbols from /usr/lib/libicuuc.so.74...
(No debugging symbols found in /usr/lib/libicuuc.so.74)
Reading symbols from /usr/lib/libstdc++.so.6...
(No debugging symbols found in /usr/lib/libstdc++.so.6)
Reading symbols from /usr/lib/libicudata.so.74...
(No debugging symbols found in /usr/lib/libicudata.so.74)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/pdo_mysql.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/redis.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/sockets.so...
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/sodium.so...
Reading symbols from /usr/lib/libsodium.so.26...
(No debugging symbols found in /usr/lib/libsodium.so.26)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/xsl.so...
Reading symbols from /usr/lib/libexslt.so.0...
(No debugging symbols found in /usr/lib/libexslt.so.0)
Reading symbols from /usr/lib/libxslt.so.1...
(No debugging symbols found in /usr/lib/libxslt.so.1)
Reading symbols from /usr/lib/libgcrypt.so.20...
(No debugging symbols found in /usr/lib/libgcrypt.so.20)
Reading symbols from /usr/lib/libgpg-error.so.0...
(No debugging symbols found in /usr/lib/libgpg-error.so.0)
Reading symbols from /usr/local/lib/php/extensions/debug-non-zts-20230831/zip.so...
__cp_end () at src/thread/x86_64/syscall_cp.s:29

warning: 29	src/thread/x86_64/syscall_cp.s: No such file or directory
(gdb) continue
Continuing.

Program received signal SIGPIPE, Broken pipe.
__cp_end () at src/thread/x86_64/syscall_cp.s:29
29	in src/thread/x86_64/syscall_cp.s
(gdb) bt
#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
#1  0x00007c5a4376e4fc in __syscall_cp_c (nr=1, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=<optimized out>, y=<optimized out>, z=0) at src/thread/pthread_cancel.c:33
#2  0x00007c5a43775944 in write (fd=<optimized out>, buf=<optimized out>, count=<optimized out>) at src/unistd/write.c:6
#3  0x00005d4c2db3f51a in safe_write (req=0x7c5a3ffa4b90, buf=0x7c5a3ffa4bc8, count=3024) at /usr/src/php/main/fastcgi.c:943
#4  0x00005d4c2db40e29 in fcgi_flush (req=0x7c5a3ffa4b90, end=1) at /usr/src/php/main/fastcgi.c:1539
#5  0x00005d4c2db4128b in fcgi_end (req=0x7c5a3ffa4b90) at /usr/src/php/main/fastcgi.c:1664
#6  0x00005d4c2db412cb in fcgi_finish_request (req=0x7c5a3ffa4b90, force_close=0) at /usr/src/php/main/fastcgi.c:1675
#7  0x00005d4c2db4e2d2 in sapi_cgi_deactivate () at /usr/src/php/sapi/fpm/fpm/fpm_main.c:779
#8  0x00005d4c2d911efa in sapi_deactivate_module () at /usr/src/php/main/SAPI.c:526
#9  0x00005d4c2d904d2f in php_request_shutdown (dummy=0x0) at /usr/src/php/main/main.c:1922
#10 0x00005d4c2db514a9 in main (argc=1, argv=0x7fff0a0942b8) at /usr/src/php/sapi/fpm/fpm/fpm_main.c:1970

To be honest, I don't get it. There is no reference to a specific library or extension. Why did this suddenly become a problem?

@nielsdos
Copy link
Member

SIGPIPE, strange... that means the file descriptor is closed on the other end...
If you can do the build yourself you might get more luck bisecting between two versions of PHP.
Note that the base image may have had changes too, so perhaps the problem isn't in PHP but in something that changed in the base.

@frederikbosch
Copy link
Contributor Author

My guess it is a change in Alpine Linux between 3.18.8 -> 3.18.9 and 3.19.3 -> 3.19.4. I lack the knowledge to know where to look for.

@frederikbosch
Copy link
Contributor Author

Because of my suspicion I have created an issue in Alpine too.

@mvanduijker
Copy link

Running into the same issue here. Would upgrading to alpine 3.20 solve it?

@nielsdos
Copy link
Member

Ideally we get something like a Docker image with some code so we can reproduce it on our systems, so far we don't have enough information to know what's going on.

@frederikbosch
Copy link
Contributor Author

Running into the same issue here. Would upgrading to alpine 3.20 solve it?

I don't know. I imagined it could also be related to a (Docker) host update, in my case Ubuntu 22.04.

Ideally we get something like a Docker image with some code so we can reproduce it on our systems, so far we don't have enough information to know what's going on.

That's totally understandable. Until we find out what is causing this, I do think this is the location the issue needs to be reported.

@mvanduijker
Copy link

mvanduijker commented Sep 18, 2024

Ideally we get something like a Docker image with some code so we can reproduce it on our systems, so far we don't have enough information to know what's going on.

I understand, for us the error starts popping up after around 15 minutes after a deployment on production.

@kgogolek
Copy link

In case this helps anyone, we have seen this quite a lot yesterday and today I've changed alpine to 3.20 and the warning is no longer showing

@frederikbosch
Copy link
Contributor Author

@kgogolek That is helpful. Then I am going to apply the 3.20 update too, and see if the error will return, or not. Also, I have reported your findings in the Alpine issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants