Tomcat – Attempting to restart tomcat 8 with gitlab-runner, pid file created, log empty, server not started

continuous integrationgitlabtomcat

I am trying to restart my tomcat 8 server after deployment.

Setup:

gitlab-runner runs with its own user (called gitlab-runner)

tomcat uses its own user called tomcat

  1. sudoers has an entry so that gitlab-runner can run a script:
    gitlab-runner ALL=(tomcat) NOPASSWD: /home/tomcat/deploy.sh

  2. The deploy script simply copies a file into the webapps directory, shuts down tomact and restarts it:


cd /home/tomcat/bin
sh catalina.sh stop -force
...
sh catalina.sh start

When I run the script manually as the gitlab-runner user everything works fine:

  1. sudo -u tomcat /home/tomcat/deploy.sh

However, when the job is started from the gitlab-ci-multi-runner (with a push to the gitlab repository) the following things are happening. The gitlab-runner basically writes a script.sh which then executes the same command from above.

  1. Usually when I start tomcat I see the following output:
    For shutdown:


Using CATALINA_BASE: /home/tomcat/
Using CATALINA_HOME: /home/tomcat/
Using CATALINA_TMPDIR: /home/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /home/tomcat/bin/bootstrap.jar:/home/tomcat/bin/tomcat-juli.jar
Using CATALINA_PID: /home/tomcat/bin/catalina.pid

For startup:

Using CATALINA_BASE: /home/tomcat/
Using CATALINA_HOME: /home/tomcat/
Using CATALINA_TMPDIR: /home/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /home/tomcat/bin/bootstrap.jar:/home/tomcat/bin/tomcat-juli.jar
Using CATALINA_PID: /home/tomcat/bin/catalina.pid
Existing PID file found during start.
Removing/clearing stale PID file.
Tomcat started.

When I start it with the gitlab-runner there is no output for shutdown but the server shuts down. This is actually working.

For the startup only Tomcat started. shows.

  1. A pid file is created with a pid inside but the process does not exist.

  2. Logfiles are created but they are empty.

If I start the server in debug mode: sh catalina.sh jpda start the file log/catalina.out just conatains one line: tomcat Listening for transport dt_socket at address: 8080

My thought was that the environment is different so I checked with the env cmd. Everything is equal except that TERM=unknown when it's started by the gitlab-runner.

Do you have any clues what might happens in the background or just things I could test?

gitlab-runner is open source and the script that starts the build script is this: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/executors/shell/executor_shell.go#L36

Best Answer

I finally figured out a solution that works.

For the problem of not showing the environment variables I refer to this: https://bz.apache.org/bugzilla/show_bug.cgi?id=37848 The gitlab-runner has no tty, so no output is shown.

The cause of the malfunctioning start is the following: Looking into the source code of the gitlab-runner you can find this: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/executors/shell/executor_shell.go#L84

Basically as soon as the gitlab-runner finishes it kills all its remaining processes. That means in my case the server was actually running for a short amount of time before the process got killed. That's why I got empty log files.

I solved the problem by simply writing an init script to start and shutdown tomcat. The tomcat now has its own user (and group). Now the tomcat process tree is initiated by /sbin/init. Now gitlab-runner is not aware of this process anymore or not able to kill it. Either way the tomcat keeps running after the runner stops.

Related Topic