Docker – panic: standard_init_linux.go:175: exec user process caused “permission denied” [recovered]

centos7docker

I am trying to use docker container with overlay.

I am starting the docker daemon with: docker daemon -s overlay

The docker run --rm hello-world command fails with:

panic: standard_init_linux.go:175: exec user process caused "permission denied" [recovered]
        panic: standard_init_linux.go:175: exec user process caused "permission denied"

goroutine 1 [running, locked to thread]:
panic(0x7e9de0, 0xc82011f110)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/urfave/cli.HandleAction.func1(0xc8200f32e8)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e
panic(0x7e9de0, 0xc82011f110)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc8200f2bf8, 0xc82001a0c8, 0xc8200f2d08)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820059630, 0x7fb8ccc34728, 0xc82011f110)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1
main.glob.func8(0xc820076a00, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/main_unix.go:26 +0x68
reflect.Value.call(0x74e5e0, 0x8ffda0, 0x13, 0x846308, 0x4, 0xc8200f3268, 0x1, 0x1, 0x0, 0x0, ...)
        /usr/local/go/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x74e5e0, 0x8ffda0, 0x13, 0xc8200f3268, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:303 +0xb1
github.com/urfave/cli.HandleAction(0x74e5e0, 0x8ffda0, 0xc820076a00, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee
github.com/urfave/cli.Command.Run(0x8491b8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8df0e0, 0x51, 0x0, ...)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec
github.com/urfave/cli.(*App).Run(0xc820001800, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4
main.main()
        /go/src/github.com/opencontainers/runc/main.go:137 +0xe24

If I use the default devicemapper driver it works without problems.

Best Answer

The problem was happening when /var was mounted with noexec. When I remounted it with mount /var/ -o remount,exec and restarted docker daemon, everything started to work as expected.

You need also to edit /etc/fstab and remove noexec option for /var entry.

As alternative is possible to use a different folder (e.g. docker daemon -s overlay -g /new_path/docker) when the fs was mounted with exec option, but you also need to change the SELinux policy for that folder with semanage fcontext and restorecon.