Jetty (embedded) rejected warning logs

[kpandey408]

Jetty version(s)
jetty.version>9.4.43.v20210629
Jetty Environment

Java version/vendor (use: java -version)
java-8 (open JDK 1.8.0_232)
OS type/version
OS_NAME="Linux"
OS_VERSION="2.6"
OS_ARCH="amd64"
Description
Recently I was trying to investigate some random 502 (GCP load balancer logs says backend_connection_closed_before_data_sent_to_client)

Found log of following errors in google log console (Note: number of 502 is very low compared to such warnings).

QueuedThreadPool[qtp341239255]@1456e5d7{STARTED,150<=150<=150,i=94,r=-1,q=0}[ReservedThreadExecutor@7092d376{s=0/2,p=2}] rejected CEP:SocketChannelEndPoint@ebb1a67{l=/10.142.64.185:17002,r=/35.191.10.133:50514,OPEN,fill=FI,flush=-,to=152431/620000}{io=1/0,kio=1,kro=1}->HttpConnection@5f666777[p=HttpParser{s=START,0 of -1},g=HttpGenerator@644af3fc{s=START}]=>HttpChannelOverHttp@5f96b225{s=HttpChannelState@70f5fd01{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=60,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING

min=max=150(threads), queue capacity=150
From the log queue is empty and threads are available,
Any suggestions why above warning are in the logs even if there seems to be not too much traffic.

How to reproduce?

[joakime]

Jetty 9.x is now at End of Community Support

• EoCS (End of Community Support) for Jetty 9 - June 2022 #7958

You should be using Jetty 10, Jetty 11, or even Jetty 12 at this point in time.

[kpandey408]

Thanks Will try to upgrade, will monitor and reply back.Regards Kamlesh…On Aug 16, 2023, at 12:07 PM, Joakim Erdfelt ***@***.***> wrote: Jetty 9.x is now at End of Community Support #7958 You should be using Jetty 10, Jetty 11, or even Jetty 12 at this point in time. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[joakime]

@kpandey408 some general advice ...

1. Don't use a BlockingQueue with a bound upper limit with QueuedThreadPool.

2. Monitor your QueuedThreadPool numbers (there's lots of information present in JMX for it), over time you'll want to adjust the configuration of your QueuedThreadPool as usage of your web application changes (especially in your case, as you are not running with a default configuration, and your server seems to want to run with a reduced configuration for some reason)

3. Keep up to date with Jetty. Your version (9.4.43) is vulnerable to several CVEs at this point in time. - https://eclipse.dev/jetty/security_reports.php

4. Keep up to date with Java. Your version (1.8.0_232) has none of the SSL/TLS changes to support the rapidly changing landscape with SSL/TLS/Crypto that the wider internet has already updated to. - https://www.java.com/en/jre-jdk-cryptoroadmap.html

[kpandey408]

Thanks Joakim It seems upgrading to jetty 10/11 will take some time due to other Java8 dependencies Regarding your point #1 above , why is BlockingQueue with upper limit not good here? I thought that should allow us to fail fast when there are too many requests. Thanks Kamlesh...

…

On Wed, Aug 16, 2023 at 12:31 PM Joakim Erdfelt ***@***.***> wrote: @kpandey408 <https://github.com/kpandey408> some general advice ... 1. Don't use a BlockingQueue with a bound upper limit with QueuedThreadPool. 2. Monitor your QueuedThreadPool numbers (there's lots of information present in JMX for it), over time you'll want to adjust the configuration of your QueuedThreadPool as usage of your web application changes (especially in your case, as you are not running with a default configuration, and your server seems to want to run with a reduced configuration for some reason) 3. Keep up to date with Jetty. Your version (9.4.43) is vulnerable to several CVEs at this point in time. - https://eclipse.dev/jetty/security_reports.php 4. Keep up to date with Java. Your version (1.8.0_232) has none of the SSL/TLS changes to support the rapidly changing landscape with SSL/TLS/Crypto that the wider internet has already updated to. - https://www.java.com/en/jre-jdk-cryptoroadmap.html — Reply to this email directly, view it on GitHub <#10327 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA36LB4GX26XMT5OKEZDDTTXVUNYPANCNFSM6AAAAAA3S63DGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[joakime]

Attempting to control limits by tweaking threading is 100% the wrong way to go.
Jetty is a 100% async server, tweaking/limiting threading just means you will break in thousands of unexpected ways that have nothing to do with controlling limits.

There are other techniques to manage too many requests.
Connection limits, DOS handlers, QOS handlers, Low Resource monitors, etc...

[sbordet]

@kpandey408 I have updated the documentation to explain why the thread pool queue should be unbounded.