Docker – Some docker containers fail to start with OCI runtime error after uprade to Virtuozzo 7

dockeropenvzubuntu-18.04virtuozzo

the hosting provider of my KVM (Strato) recently upgraded the version of Virtuozzo they were using from 6 to 7.
Since then a few of my docker containers fail to start with the following error message:

❯ sudo docker start spring_webapp

Error response from daemon: 
OCI runtime create failed: container_linux.go:349: starting container process caused
"process_linux.go:297: applying cgroup configuration for process caused 
\"failed to set memory.kmem.limit_in_bytes, 
because either tasks have already joined this cgroup 
or it has children\"": unknown

Error: failed to start containers: spring_webapp

The only thing the containers that won't start seem to have in common is that they contain a java webapp. Other containers like gitlab or a few mariadb instances seem to start just fine.

A google search for the error message returned some older issues on github but those seem to have been fixed years ago: https://github.com/opencontainers/runc/issues/1083

I am running Ubuntu 18.04 LTS with docker 19.03.12.

Already opened a ticket with my hosting provider but they answered with a canned response that since I have a root server they can't help me, basically saying that it's a problem with my configuration.

Unfortunately I don't know enough about OpenVZ/Virtuozzo to refute them.

Any hint pointing me in the right direction would be highly appreciated 🙂

Here's the output of /proc/user_beancounters

❯ sudo cat /proc/user_beancounters
         
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
    831745: kmemsize                564412416            766132224  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                   16  9223372036854775807  9223372036854775807                    0
            privvmpages               5148863              5444666  9223372036854775807  9223372036854775807                    0
            shmpages                    41758                74651  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       919                  919                 1100                 1100                    0
            physpages                 1972269              2444334              8388608              8388608                    0
            vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
            oomguarpages              2104451              2452167                    0                    0                    0
            numtcpsock                      0                    0  9223372036854775807  9223372036854775807                    0
            numflock                        0                    0  9223372036854775807  9223372036854775807                    0
            numpty                          4                    4  9223372036854775807  9223372036854775807                    0
            numsiginfo                     16                  129  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                       0                    0  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                       0                    0  9223372036854775807  9223372036854775807                    0
            othersockbuf                    0                    0  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0                    0  9223372036854775807  9223372036854775807                    0
            numothersock                    0                    0  9223372036854775807  9223372036854775807                    0
            dcachesize              353423360            552648704  9223372036854775807  9223372036854775807                    0
            numfile                      6327                11230  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                     219                  222                 2000                 2000                    0

Best Answer

Yep, thats a Strato issue... happened after their maintenance 15./16. July 2020. I have already submitted a ticket... lets see if there is a response.

Needless to say, Strato had major performance problems in the past 3..4 months and just got that fixed recently.

Now this new maintenance last week messed up everything again.

EDIT:

Just deleted the container(s) from my server and reinstalled them. Now all is working well again. Maybe thats a solution for others, too?

Related Solutions

Docker – Start and stop services as docker containers

I do use fig (has now been "renamed" to docker-compose) to adress this type of problem. One can denote the individual docker containers involved in a "deployment" (as I will call it for now, please anyone: post a comment if you know better terminology. Bear with me, I'm german). This is done in YAML notation in a file named fig.ylm (docker-compose.yml if you have migrated to docker-compose). You are then able to start, stop, etc. the described set of containers via commands like fig start, fig stop.

The most convenient command to get things running would be fig up (builds all contains and starts all of them like a single application with combined console output).

To do this in scripts, in a deamon like fashion, the -d parameter is of use: fig up -d runs the whole shebang like a single daemon.

Link to Fig documentation: http://www.fig.sh

Same for docker-compose: https://docs.docker.com/compose/

I will now post you a complete example of a Gitlab "deployment" consisting of the services Gitlab, Postgres and Redis. It runs on a Ubuntu host, and is started automatically at system boot by an upstart script:

The fig.yml (at /root/docker_gitlab/ on host):

postgresql:
  image: sameersbn/postgresql:9.1-1
  environment:
    - DB_USER=gitlab
    - DB_PASS=secretpassword
    - DB_NAME=gitlabhq_production
gitlab:
  image: sameersbn/gitlab:latest
  links:
   - redis:redisio
   - postgresql:postgresql
  ports:
   - "10080:80"
   - "10022:22"
  volumes:
   - /opt/gitlab/data:/home/git/data
  environment:
   - SMTP_HOST=smtp.germanprovider.de
   - SMTP_USER=secret@secret.de
   - SMTP_PASS=verysecret
   - GITLAB_HOST=projectserver
   - GITLAB_PORT=10080
   - GITLAB_SSH_PORT=10022
redis:
  image: sameersbn/redis:latest

The upstart script gitlab.conf (at /etc/init/ on host):

description "gitlab service runner"
start on filesystem and started docker
stop on runlevel [!2345]
respawn
chdir /root/docker_gitlab
script
# Wait for docker to finish starting up first.
FILE=/var/run/docker.sock
while [ ! -e $FILE ] ; do
inotifywait -t 2 -e create $(dirname $FILE)
done
/usr/local/bin/fig start
end script

Fig seems to have been so "docker-style", that the docker team incorporated it as docker-compose. So all should be fine in that regard...

Ubuntu – How to auto-start docker containers at system boot

Apparently, the current method to auto-start Docker containers (from Docker 1.2) is to use restart policies. This will control how Docker should handle starting of the container upon startup and re-starting of the container when it exits. I've used the 'always' option so far, and can confirm that it makes Docker auto-start the container at system boot:

sudo docker run --restart=always -d myimage

Documentation Excerpt

Restart Policies Using the --restart flag on Docker run you can specify a restart policy for how a container should or should not be restarted on exit.

no - Do not restart the container when it exits.

on-failure - Restart the container only if it exits with a non zero exit status.

always - Always restart the container regardless of the exit status.

You can also specify the maximum amount of times Docker will try to restart the container when using the on-failure policy. The default is that Docker will try forever to restart the container.
$ sudo docker run --restart=always redis
This will run the redis container with a restart policy of always so that if the container exits, Docker will restart it.
$ sudo docker run --restart=on-failure:10 redis
This will run the redis container with a restart policy of on-failure and a maximum restart count of 10. If the redis container exits with a non-zero exit status more than 10 times in a row Docker will abort trying to restart the container. Providing a maximum restart limit is only valid for the on-failure policy.

Best Answer

Related Solutions

Docker – Start and stop services as docker containers

Ubuntu – How to auto-start docker containers at system boot

Documentation Excerpt

Related Topic