REST endpoint stops responding after sometime.

[SarthakSahu]

Jetty version
9.4.27.v20200227
https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.4.27.v20200227

Java version
[root@xyz-1 ~]# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

OS type/version
3.10.0-1062.1.1.el7.x86_64 #1 SMP Tue Aug 13 18:39:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Note: This is 24 core machine

Description
After application starts REST endpoint works perfectly for a random duration, then stops working.
[root@xyz-1 ~]# netstat -anp | grep 8182
netstat -anp | grep 8182
tcp 0 0 127.0.0.1:41992 127.0.0.1:8182 ESTABLISHED 28762/curl
tcp6 0 0 127.0.0.1:8182 :::* LISTEN 18513/java
tcp6 105 0 127.0.0.1:8182 127.0.0.1:47516 CLOSE_WAIT 18513/java
tcp6 104 0 127.0.0.1:8182 127.0.0.1:41992 ESTABLISHED 18513/java
tcp6 105 0 127.0.0.1:8182 127.0.0.1:47274 CLOSE_WAIT 18513/java
tcp6 105 0 127.0.0.1:8182 127.0.0.1:47890 CLOSE_WAIT 18513/java

Nonworking_jstack.txt
working_log.txt
working_jstack.txt
Nonworking_log.txt

[sbordet]

What is the configuration of the Jetty thread pool?

[SarthakSahu]

Its 200.

[sbordet]

The (partial) logs show that a connection is being accepted, a task submitted to the thread pool, but the task is never run.

That can only happen if you have run out of threads in the pool, or the log is incomplete and the task is run some time in the future.

The threads dump show that Jetty is using 15 threads: 3 acceptors (1 would be enough) and 12 in select(). All the rest is idle.

Can you please take a server dump as explained here when the problem happens?

[SarthakSahu]

Application deployed in less then 8 core machine where it creates only 1 acceptor thread, it works good with out any issue. Problem comes when it created 3 acceptors in 24 core machine.
Mean while I will provided server dump.

[sbordet]

Problem comes when it created 3 acceptors in 24 core machine.

There are no problems with creating 3 acceptors.

[SarthakSahu]

With -XX:ActiveProcessorCount java command line option, jetty created only one 1 acceptors thread and it working perfectly now.

Here embedded jetty server has used. Could you please provide detailed step to get jetty server dump.

[sbordet]

As I said, I doubt that the issue is creating one or more acceptors.

Follow the link I gave above in the previous message for detailed instructions about how to get a server dump.

[olamy]

more than 1 year without more details so closing it