I launch m5.large
(nitro-based) EC2 instance from Ubuntu AMI and attach EBS volume. There is systemd
as a default init system. As AWS documentation "Making an Amazon EBS Volume Available for Use on Linux" stands, I mount EBS volume within user data:
#!/bin/bash
# Sleep gives the SSD drive a chance to mount before the user data script completes.
sleep 15
mkdir /application
mount /dev/nvme1n1 /application
I need Nginx and provide site configuration for it at EBS volume. For default nginx
package with systemd unit file I declare a dependency on the mount with RequiresMountsFor
directive within drop-in:
# /lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/nginx.service.d/override.conf
[Unit]
RequiresMountsFor=/application
[Service]
Restart=always
But this doesn't help to run Nginx only after mount will be completed (in user data) for some reason. I can see the mount unit for /application
path, but I don't see Required=application.mount
as I'd expect:
$ sudo systemctl show -p After,Requires nginx
Requires=system.slice sysinit.target -.mount
After=sysinit.target -.mount systemd-journald.socket basic.target application.mount system.slice network.target
Nginx service still tries to run before cloud-init
completes user data execution, exhausts all attempts to run the service and fails:
Apr 08 15:34:32 hostname nginx[1303]: nginx: [emerg] open() "/application/libexec/etc/nginx/nginx.site.conf" failed (2: No such file or directory) in /etc/nginx/sites-e
Apr 08 15:34:32 hostname nginx[1303]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 08 15:34:32 hostname systemd[1]: nginx.service: Control process exited, code=exited status=1
Apr 08 15:34:32 hostname systemd[1]: Failed to start A high performance web server and a reverse proxy server.
I assume the service should be started by systemd on mount notification for the specified path /application
. What am I missing?
What is the most flexible and correct way to mount EBS volumes at Ubuntu + systemd?
Best Answer
Here is my take on a solution that attempts to honour the constraints and limitations mentioned in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#identify-nvme-ebs-device. Not currently used in production but that's the aim...
The basic idea is to mimic the systemd
Requires=foo.mount
behaviour by wrapping the EBS mount procedure in a systemdservice
. Other services that depend on the EBS mount can simply specify that they must be startedAfter
the mountservice
is available.The functionality builds upon the
udev
module that sym-links the physical device with the requested device specified when the volume is attached. For example/dev/sdf
links to/dev/nvme...
). See https://github.com/oogali/ebs-automatic-nvme-mapping and the above guide for more details.The
data-mount.service
waits in a sleep loop for the sym-link to appear (allowing you time to attach the volume), then mounts the volume at the defined mount point, formatting it if necessary.Service to mount the EBS volume
/sbin/ec2-boot-mount-ebs-volume
/etc/systemd/system/data-mount.service
Other Services
/etc/systemd/system/other-service.service.d/override.conf