WAR which uses JNI / a library - LD_LIBRARY_PATH is purged by openjdk

[ArfyFR]

Jetty version
jetty-9.4.24.v20191120 - 20 November 2019

Java version
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)

OS type/version
cat /proc/version
Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Fri Dec 6 15:49:49 UTC 2019

Question
I had to update a WAR on a jetty inside a docker
This WAR now needs a specific library to work

I tried to add the path in LD_LIBRARY_PATH but it faced a NoClassDefFoundError

I added some Java code in another WAR to check what's is LD_LIBRARY_PATH inside jetty => null
As if just before the call to
java -jar "$JETTY_HOME/start.jar" $JETTY_OPTIONS $JETTY_STOP_OPTIONS --module=ext jetty-started.xml
if I echo LD_LIBRARY_PATH, it's well filled

Solution, add this to java -jar call
-Djava.library.path="/path1:/path2:..."

I found this: "JDK-6367077 Purge LD_LIBRARY_PATH usage from the launcher" https://bugs.openjdk.java.net/browse/JDK-6367077
May be add this info in doc ?

[joakime]

First, are you using the official docker images?

• https://hub.docker.com/_/jetty
• https://github.com/eclipse/jetty.docker

Next, if you are not using one of those, are you perhaps using a ${jetty.base} configuration that is causing a forked JVM? (see --exec or output of --list-config)
A forked JVM will not propagate the LD_LIBRARY_PATH environment or the System properties from the parent JVM.

[ArfyFR]

If I compare all environment variables set before the call to
java -jar "$JETTY_HOME/start.jar" $JETTY_OPTIONS $JETTY_STOP_OPTIONS --module=ext jetty-started.xml
and after (I mean inside a java in a War deployed), most of them (too much to check them all) are well set ... but no more LD_LIBRARY_PATH

That's why I searched in Jetty, then openjdk if one may purge it.
And there it was: https://bugs.openjdk.java.net/browse/JDK-6367077 (I suppose)

[janbartel]

@ArfyFR you didn't answer the question of which docker image you are using.

[ArfyFR]

@ArfyFR you didn't answer the question of which docker image you are using.

People from build team just told me: docker from scratch, they don't use official docker images as lot of other tools and backend DEV are included.

And about "forked JVM": I replied, all other environment variables before the
java -jar "$JETTY_HOME/start.jar" $JETTY_OPTIONS $JETTY_STOP_OPTIONS --module=ext jetty-started.xml
are kept ... but LD_LIBRARY_PATH vanished.

Inside a deployed WAR, I put this code to show all known env vars

try {
	String ldlibpath = System.getenv("LD_LIBRARY_PATH");
	String path = System.getenv("PATH");
	FileWriter myWriter = new FileWriter("/tmp/ldlibpath.txt", true);
	
	// Write LD_LIBRARY_PATH and PATH
	myWriter.write("LD_LIBRARY_PATH " + ldlibpath + String.format("%n"));
	myWriter.write("PATH" + path + String.format("%n"));

	// Write all known ENV variables
	Map<String, String> env = System.getenv();
	for (String envName : env.keySet()) {
		myWriter.write(envName + " = " + env.get(envName) + String.format("%n"));
	}

	// Write current java.library.path
	myWriter.write( "Show java.library.path" + String.format("%n"));
	String property = System.getProperty("java.library.path");
	StringTokenizer parser = new StringTokenizer(property, ";");
	while (parser.hasMoreTokens()) {
		myWriter.write( parser.nextToken() + String.format("%n"));
	}
	myWriter.close();
	System.out.println("Successfully wrote to the file.");
} catch (IOException e) {
	System.out.println("An error occurred.");
	e.printStackTrace();
}

[janbartel]

@ArfyFR I've tried setting the LD_LIBRARY_PATH and modifying the standard jetty test webapp to print out both the LD_LIBRARY_PATH and the java.library.path and both are populated. I'm using Ubuntu linux.
I would check what your docker images are doing with LD_LIBRARY_PATH.

[ArfyFR]

@janbartel I also populated the LD_LIBRARY_PATH inside the docker. In bash it's OK
In scripts, it's OK
But inside Jetty/Java, it's reset-ed to NULL

You're also using openjdk ? Another version ?

[joakime]

@ArfyFR is your custom docker image causing a forked JVM with your specific configuration? (the question I asked 14 days ago)
That can easily make it appear to your code that the LD_LIBRARY_PATH is being reset.

If you use the official docker image you'll see that we intentionally work around configurations that cause forking to prevent situations like yours.
First step, verify that you are using a configuration that will cause a forked JVM.
Next step, if you are using such a configuration, you'll need to rework your docker image to do the same things the official docker image does to work around that kind of configuration. (a combination of shell scripts, and --dry-run, and command line massaging, with env variable injection, etc)

See https://github.com/eclipse/jetty.docker/tree/master/9.4-jdk8

[ArfyFR]

@janbartel I asked to the docker team:
The docker base used is a centos:7 official with java-1.8.0-openjdk

How can they test if the "custom docker image causing a forked JVM" ?

[joakime]

It's not docker causing the forked JVM, it's the configuration of your ${jetty.base} that could cause the forked JVM.
Some configurations (in your ${jetty.base}), be it something as mundane as logging, or attempts to configure the JVM (be it memory settings, or agents, or gc tweaking, or the existence of some system properties, etc) will trigger the start.jar to fork a JVM to satisfy that configuration.

However, that forked JVM will not propagate the JVM runtime from the initial JVM to the forked JVM unless you 100% embrace the ${jetty.home} and ${jetty.base} split and configuration (meaning you do absolutely no configuration before the start.jar command.

Example of direct JVM configuration that has no impact on the forked JVM:

# This configuration for max memory will not propagate to the forked JVM
$ java -Xmx1g -jar start.jar

# This environment variable will not propagate to the forked JVM
$ export LD_LIBRARY_PATH=/tmp/lib/
$ java -jar start.jar

If you have a forked JVM due to your configuration, you have a few choices.

1. Use the official Jetty docker image. (see https://hub.docker.com/_/jetty)
2. Use the techniques of the official Jetty docker image to work around the forked JVM. (see https://github.com/eclipse/jetty.docker/tree/master/9.4-jdk8)
3. Change your ${jetty.base} configuration to not fork.
4. 100% embrace the forked JVM and do all of your configuration of the forked JVM using the jvm module in ${jetty.base}, never directly configuring the JVM for anything.

For docker, the options 1 and 2 is the preferred by nearly everyone that uses docker (no point in wasting the memory on the initial JVM that will never be used)

When looking at the official docker image, be aware that it does not use jetty-distribution, as that's wasteful for docker, it uses jetty-home tarball instead. (Note: jetty-distribution is just jetty-home + a demo ${jetty.base} and a transitional start.ini for the deprecated jetty-distribution tarball that no longer exists after Jetty 9. Jetty 10+ only has jetty-home, which is recommended)

[ArfyFR]

@janbartel
Here is the way our "jetty" is build for docker

A Jetty 9 not modified then (jetty-9.4.24.v20191120 in production)
Then

    cd "$JETTY_BASE"

    java -jar "$JETTY_HOME/start.jar" --create-startd
    java -jar "$JETTY_HOME/start.jar" --add-to-start=http,deploy,jsp,websocket,plus
    java -jar "$JETTY_HOME/start.jar" --approve-all-licenses --add-to-start=logging-jul

Nothing else is modified "by hand"
One of these force a forked JVM ?

I had a look at https://github.com/eclipse/jetty.docker/tree/master/9.4-jdk8 but didn't notice something about forked JVM
(the set -- java ... ?)

And once again, other exports I do before the jetty start are kept, only LD_LIBRARY_PATH is purged

[joakime]

Assuming you have no other config, or a pre-existing ${jetty.base}, or an intermediate jetty-dir ...

Then --add-to-start=logging-jul will fork the JVM on Jetty 9.x (it does not fork on Jetty 10.x as the logging layer has been completely rewritten)

This because in order to have JUL logging actually work with Jetty 9.x logging, it has to be enabled from the start of the JVM, practically speaking JUL needs to be initialized first, even before start.jar itself is executed.

Example:

[joakim@hyperion bases]$ mkdir logging-demo
[joakim@hyperion bases]$ cd logging-demo/

[joakim@hyperion logging-demo]$ java -jar ../../jetty-home-9.4.24.v20191120/start.jar --create-startd
MKDIR : ${jetty.base}/start.d
INFO  : Base directory was modified

[joakim@hyperion logging-demo]$ java -jar ../../jetty-home-9.4.24.v20191120/start.jar --add-to-start=http,deploy,jsp,websocket,plusINFO  : webapp          transitively enabled, ini template available with --add-to-start=webapp
INFO  : server          transitively enabled, ini template available with --add-to-start=server
INFO  : mail            transitively enabled
INFO  : servlet         transitively enabled
INFO  : jsp             initialized in ${jetty.base}/start.d/jsp.ini
INFO  : annotations     transitively enabled
INFO  : transactions    transitively enabled
INFO  : threadpool      transitively enabled, ini template available with --add-to-start=threadpool
INFO  : plus            initialized in ${jetty.base}/start.d/plus.ini
INFO  : deploy          initialized in ${jetty.base}/start.d/deploy.ini
INFO  : security        transitively enabled
INFO  : apache-jsp      transitively enabled
INFO  : websocket       initialized in ${jetty.base}/start.d/websocket.ini
INFO  : jndi            transitively enabled
INFO  : http            initialized in ${jetty.base}/start.d/http.ini
INFO  : client          transitively enabled
INFO  : bytebufferpool  transitively enabled, ini template available with --add-to-start=bytebufferpool
MKDIR : ${jetty.base}/webapps
INFO  : Base directory was modified


[joakim@hyperion logging-demo]$ java -jar ../../jetty-home-9.4.24.v20191120/start.jar --approve-all-licenses --add-to-start=logging-jul
INFO  : All Licenses Approved via Command Line Option
INFO  : slf4j-api       transitively enabled
INFO  : jul-impl        transitively enabled
INFO  : slf4j-jul       transitively enabled
INFO  : logging-jul     initialized in ${jetty.base}/start.d/logging-jul.ini
MKDIR : ${jetty.base}/lib/slf4j
COPY  : /home/joakim/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar to ${jetty.base}/lib/slf4j/slf4j-api-1.7.25.jar
MKDIR : ${jetty.base}/etc
COPY  : ${jetty.home}/modules/jul-impl/etc/java-util-logging.properties to ${jetty.base}/etc/java-util-logging.properties
COPY  : /home/joakim/.m2/repository/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar to ${jetty.base}/lib/slf4j/slf4j-jdk14-1.7.25.jar
INFO  : Base directory was modified


[joakim@hyperion logging-demo]$ java -jar ../../jetty-home-9.4.24.v20191120/start.jar --list-config

Java Environment:
-----------------
 java.home = /home/joakim/java/jvm/jdk-11.0.12+7 (null)
 java.vm.vendor = Eclipse Foundation (null)
 java.vm.version = 11.0.12+7 (null)
 java.vm.name = OpenJDK 64-Bit Server VM (null)
 java.vm.info = mixed mode (null)
 java.runtime.name = OpenJDK Runtime Environment (null)
 java.runtime.version = 11.0.12+7 (null)
 java.io.tmpdir = /tmp (null)
 user.dir = /home/joakim/code/jetty/distros/bases/logging-demo (null)
 user.language = en (null)
 user.country = US (null)

Jetty Environment:
-----------------
 jetty.version = 9.4.24.v20191120
 jetty.tag.version = master
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-9.4.24.v20191120
 jetty.base = /home/joakim/code/jetty/distros/bases/logging-demo

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /home/joakim/code/jetty/distros/bases/logging-demo
 ${jetty.home} -> /home/joakim/code/jetty/distros/jetty-home-9.4.24.v20191120


JVM Arguments:
--------------
 -Djava.util.logging.config.file?=${jetty.base}/etc/java-util-logging.properties
 -Dorg.eclipse.jetty.util.log.class?=org.eclipse.jetty.util.log.Slf4jLog

System Properties:
------------------
 (no system properties specified)

Properties:
-----------
 java.version = 11.0.12
 java.version.major = 11
 java.version.micro = 12
 java.version.minor = 0
 java.version.platform = 11
 jetty.base = /home/joakim/code/jetty/distros/bases/logging-demo
 jetty.base.uri = file:///home/joakim/code/jetty/distros/bases/logging-demo
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-9.4.24.v20191120
 jetty.home.uri = file:///home/joakim/code/jetty/distros/jetty-home-9.4.24.v20191120
 jetty.webapp.addServerClasses = ${jetty.base.uri}/lib/slf4j/
 slf4j.version = 1.7.25

Jetty Server Classpath:
-----------------------
Version Information on 36 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:      1.4.1.v201005082020 | ${jetty.home}/lib/mail/javax.mail.glassfish-1.4.1.v201005082020.jar
 1:                   1.7.25 | ${jetty.base}/lib/slf4j/slf4j-api-1.7.25.jar
 2:                   1.7.25 | ${jetty.base}/lib/slf4j/slf4j-jdk14-1.7.25.jar
 3:                    3.1.0 | ${jetty.home}/lib/servlet-api-3.1.jar
 4:                 3.1.0.M0 | ${jetty.home}/lib/jetty-schemas-3.1.jar
 5:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-http-9.4.24.v20191120.jar
 6:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-server-9.4.24.v20191120.jar
 7:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-xml-9.4.24.v20191120.jar
 8:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-util-9.4.24.v20191120.jar
 9:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-io-9.4.24.v20191120.jar
10:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-jndi-9.4.24.v20191120.jar
11:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-security-9.4.24.v20191120.jar
12:                      1.3 | ${jetty.home}/lib/transactions/javax.transaction-api-1.3.jar
13:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-servlet-9.4.24.v20191120.jar
14:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-webapp-9.4.24.v20191120.jar
15:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-plus-9.4.24.v20191120.jar
16:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-annotations-9.4.24.v20191120.jar
17:                      7.2 | ${jetty.home}/lib/annotations/asm-7.2.jar
18:                      7.2 | ${jetty.home}/lib/annotations/asm-analysis-7.2.jar
19:                      7.2 | ${jetty.home}/lib/annotations/asm-commons-7.2.jar
20:                      7.2 | ${jetty.home}/lib/annotations/asm-tree-7.2.jar
21:                      1.3 | ${jetty.home}/lib/annotations/javax.annotation-api-1.3.jar
22:    3.17.0.v20190306-2240 | ${jetty.home}/lib/apache-jsp/org.eclipse.jdt.ecj-3.17.0.jar
23:         9.4.24.v20191120 | ${jetty.home}/lib/apache-jsp/org.eclipse.jetty.apache-jsp-9.4.24.v20191120.jar
24:                   8.5.40 | ${jetty.home}/lib/apache-jsp/org.mortbay.jasper.apache-el-8.5.40.jar
25:                   8.5.40 | ${jetty.home}/lib/apache-jsp/org.mortbay.jasper.apache-jsp-8.5.40.jar
26:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-client-9.4.24.v20191120.jar
27:         9.4.24.v20191120 | ${jetty.home}/lib/jetty-deploy-9.4.24.v20191120.jar
28:                      1.0 | ${jetty.home}/lib/websocket/javax.websocket-api-1.0.jar
29:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/javax-websocket-client-impl-9.4.24.v20191120.jar
30:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/javax-websocket-server-impl-9.4.24.v20191120.jar
31:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/websocket-api-9.4.24.v20191120.jar
32:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/websocket-client-9.4.24.v20191120.jar
33:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/websocket-common-9.4.24.v20191120.jar
34:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/websocket-server-9.4.24.v20191120.jar
35:         9.4.24.v20191120 | ${jetty.home}/lib/websocket/websocket-servlet-9.4.24.v20191120.jar

Jetty Active XMLs:
------------------
 ${jetty.home}/etc/jetty-bytebufferpool.xml
 ${jetty.home}/etc/jetty-threadpool.xml
 ${jetty.home}/etc/jetty.xml
 ${jetty.home}/etc/jetty-webapp.xml
 ${jetty.home}/etc/jetty-plus.xml
 ${jetty.home}/etc/jetty-annotations.xml
 ${jetty.home}/etc/jetty-deploy.xml
 ${jetty.home}/etc/jetty-http.xml

Notice that the --list-config shows the section titled "JVM Arguments" ?
If there are any values in the section "JVM Arguments" or "System Properties", then you have a forked JVM.

Using the openjdk binary jps you can verify this ...

$ jps
1744683 start.jar
1744710 XmlConfiguration
1744772 Jps

$ jps -vvv
1744683 start.jar
1744817 Jps -Dapplication.home=/home/joakim/java/jvm/jdk-11.0.12+7 -Xms8m -Djdk.module.main=jdk.jcmd
1744710 XmlConfiguration -Djava.io.tmpdir=/tmp 
  -Djetty.home=/home/joakim/code/jetty/distros/jetty-home-9.4.24.v20191120 
  -Djetty.base=/home/joakim/code/jetty/distros/bases/logging-demo 
  -Djava.util.logging.config.file=/home/joakim/code/jetty/distros/bases/logging-demo/etc/java-util-logging.properties 
  -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.Slf4jLog

The start.jar is executing as pid 1744683, and the forked JVM is on pid 1744710, complete with the same "JVM Arguments" you see in --list-config.

[joakime]

I had a look at https://github.com/eclipse/jetty.docker/tree/master/9.4-jdk8 but didn't notice something about forked JVM
(the set -- java ... ?)

The whole effort around --dry-run is how the forked JVM is avoided.

If you can upgrade to Jetty 9.4.44.v20210927, then you have a better option (this extension to the --dry-run feature does not exist on 9.4.24)

[logging-demo]$ java -jar ../../jetty-home-9.4.44.v20210927/start.jar --help
...(snip)...
  --dry-run=parts  Print specific parts of the command line. The parts
                   are a comma separated list of
                     o  "java" - the JVM to run
                     o  "opts" - the JVM options (eg -D and -X flags)
                     o  "path" - the JVM class path or JPMS modules options
                     o  "main" - the main class to run
                     o  "args" - the arguments passed to the main class

                   It is possible to decompose the start command:

                     OPTS=$(java -jar start.jar --dry-run=opts,path)
                     MAIN=$(java -jar start.jar --dry-run=main)
                     ARGS=$(java -jar start.jar --dry-run=args)
                     java $OPTS -Dextra=opt $MAIN $ARGS extra=arg

                   Alternatively to create an args file for java:

                     java -jar start.jar --dry-run=opts,path,main,args > /tmp/args
                     java @/tmp/args
...(snip)...

[janbartel]

@joakime @ArfyFR the missing LD_LIBRARY_PATH is nothing to do with jetty, nor with the jvm, but more to do with however @ArfyFR is setting up his docker.

You can verify all this by running the following command, which assumes I have a jetty 9.4 distribution in the working directory:

docker run -it --rm -p 8080:8080 --mount type=bind,source="$(pwd)",target=/tmp/jetty jetty:latest bash -c "cd /tmp/jetty/demo-base && export LD_LIBRARY_PATH=/tmp/jetty && java -jar ../start.jar --exec"

The above command will mount the working directory with the jetty distro onto docker at /tmp/jetty. I happen to be using jetty:latest docker image because it comes already set up with java etc etc, but please notice that I am just running bash command. I export the LD_LIBRARY_PATH then run jetty telling it to exec another process. I modified the HelloWorld servlet in the standard test webapp to print out the values of LD_LIBRARY_PATH and java.library.path like so:

         ServletOutputStream out = response.getOutputStream();
         out.println("<html>");
         out.println("<h1>Hello World</h1>");
+        out.println("LD_LIBRARY_PATH: " + System.getenv("LD_LIBRARY_PATH"));
+        out.println("java.library.path: " + System.getProperty("java.library.path"));
         out.println("</html>");
         out.flush();

The output even with --exec is that the LD_LIBRARY_PATH is available at runtime:

Hello World
LD_LIBRARY_PATH: /tmp/jetty java.library.path: /tmp/jetty:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib

The jvm in use is:

openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)

I observe exactly the same behaviour on my local ubuntu system using openjdk 11.0.2.

[joakime]

Went down a rabbit hole here today.

Started out by just creating a simple ${jetty.base} and including 1 custom XML file.

[env-dump]$ tree -F
.
├── etc/
│   └── dump-env.xml
├── start.ini
└── webapps/

2 directories, 2 files
[env-dump]$ cat start.ini 
--module=http
--module=deploy
etc/dump-env.xml

[env-dump]$ cat etc/dump-env.xml 
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://www.eclipse.org/jetty/configure_9_0.dtd">

<Configure class="org.eclipse.jetty.server.Server">
  <Get class="org.eclipse.jetty.util.log.Log" name="rootLogger">
    <Call name="info">
      <Arg>System.env[LD_LIBRARY_PATH] = <Call class="java.lang.System" name="getenv">
        <Arg>LD_LIBRARY_PATH</Arg>
      </Call></Arg>
    </Call>
  </Get>  
  <Get class="org.eclipse.jetty.util.log.Log" name="rootLogger">
    <Call name="info">
      <Arg>System.property[java.library.path] = <Call class="java.lang.System" name="getProperty">
        <Arg>java.library.path</Arg>
      </Call></Arg>
    </Call>
  </Get>  
</Configure>

When executing you will see the ENV printed out ...

2021-10-08 21:00:07.472:INFO::main: Logging initialized @320ms to org.eclipse.jetty.util.log.StdErrLog
2021-10-08 21:00:07.668:INFO::main: System.env[LD_LIBRARY_PATH] = null
2021-10-08 21:00:07.669:INFO::main: System.property[java.library.path] = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-10-08 21:00:07.672:INFO:oejs.Server:main: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9332; jvm 1.8.0_302-b08

The ENV in your docker image is greatly impacted by the way you create your docker image.

For example, docker-compose have several bugs around how ENV variables are created/populated during the different phases of the docker image creation and runtime.

Using the standard docker command line itself you can setup the image to get ENV from internal to the image (either at build time, or from the ENTRYPOINT behavior), or via external from the image using the docker run command line options (--env or --env-file) when running the image.

I found that docker-compose was the most unreliable way to set and ENV variable.
The only technique that worked reliably was to have the docker entryPoint be a shell script, and that shell script manages the LD_LIBRARY_PATH before calling java.
Using any of the other techniques that docker-compose offers is just unreliable.

Docker plain (not using docker-compose), works find for internal to image ENV, but once you mix in the external to image ENV (such as via command line --env or --env-file) then you have to be super careful not attempt to set a ENV variable that also is being managed internally.

If using the external to docker image ENV options exclusively (no build time declaration, or internal declaration, or script declaration) - Using one of --env and --env-file options, only sets the ENV on the CMD (or ENTRYPOINT), and if a does not propagate to a forked JVM. verified by joining the running docker image and checking with jinfo -sysprops <pid> | grep library, the initial JVM has the correct java.library.path, the forked JVM does not.

The best, most consistent behavior turns out to be to not use the ENV in Dockerfile, or the command line --env or --env-file, but to use the ENTRYPOINT shell script to set the ENV variable just before calling java. That satisfies the ENV on the initial JVM, and also propagates to the forked JVM as well.

In short, the initial JVM (as far as my limited testing shows) is always correct.
The forked JVM can easily be bad for ENV variables (depending on how you setup the ENV variable with docker).
(oh, and don't use docker-compose when dealing with ENV variables.)

I haven't even looked at the extra layer of complexity something like k8s would bring to this mess.

[janbartel]

@ArfyFR I think we've thoroughly discussed the docker ENV issue and you should have enough info to progress, so I'll close this issue now.

[ArfyFR]

@ArfyFR I think we've thoroughly discussed the docker ENV issue and you should have enough info to progress, so I'll close this issue now.

As said on my post on 23 sept 2021:

Solution, add this to java -jar call
-Djava.library.path="/path1:/path2:..."

So I found a way to make it work, but not with LD_LIBRARY_PATH =)