AWS EC2 – Can m5d’s Local SSD Be Used as a Root Device?

amazon ec2amazon-web-services

I've been using the Amazon ECS AMI on m5 EC2 instances, and I noticed that there's a new m5d instance type which is similar but comes with a 70 GB local SSD. Since these are throwaway instances, faster semi-cheap SSD's seem perfect, but I can't figure out how to actually use this local storage.

I initially tried to replace the /dev/xvda or /dev/xvdcz device with ephemeral0 in the block device mapping:

BlockDeviceMappings:
  - DeviceName: /dev/xvdcz
    VirtualName: ephemeral0

.. but this doesn't seem to do anything (the logs on the box say something about ephemeral0 not existing), and I found in the documentation that:

[NVMe instance store volumes] These volumes are automatically enumerated and assigned a device name; including them in your block device mapping has no effect.

Since the ECS AMI uses an LVM partition, I could potentially create a small EBS volume for /dev/xvdcz and then add the local partition (which is visible in lsblk) to the LVM store, but the fact that NVMe partitions have arbitrary names makes this hard to do in a safe way.

So this is the point I'm stuck at. How do you actually use the local storage if you can't use it in a block device mapping and the device names are semi-random? Does Amazon expect you to manually log in and edit the fstab every time you boot one of these instances? I feel like I must be missing something.

I realize there's probably special rules that prevent me from replacing the root device with an ephemeral device, but I can't figure out how I'm supposed to use this thing if it can't even replace the data volume.

Best Answer

You can't use the Instance storage as your root device. That used to be possible many years ago but as far as I know it's no longer supported with new AMIs. So your root will have to be on EBS and you will have to format and mount the ephemeral device during boot and use it only for your temporary data, cache folders, tmp, etc.

And no, you are not expected to SSH to the instance and do it manually. Instead put the required commands into the UserData script that's executed when the instance boots for the first time. To learn more about UserData refer to Running Commands on Your Linux Instance at Launch

Update: One of the issues is that the Instance storage device name is not quite predictable. It may be /dev/nvme1n1 or /dev/nvme2n1 or who knows what. Fortunately all disk devices also have more descriptive aliases under /dev/disk/by-id/ and we can use those for our purpose:

[ec2-user@ip-172-31-10-228 ~]$ ls -l /dev/disk/by-id/
nvme-Amazon_EC2_NVMe_Instance_Storage_AWS12B74F473B3456789       << instance storage
nvme-Amazon_EC2_NVMe_Instance_Storage_AWS12B74F473B3456789-ns-1  << instance storage
nvme-Amazon_Elastic_Block_Store_vol0199d8c18d345678a
nvme-Amazon_Elastic_Block_Store_vol0199d8c18d345678a-...
...

The first two lines are the instance storage. I'm not sure what's the difference between the /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS12B74F473B3456789 and the ...-ns-1 alias, they refer to the same block device so we can use either of them.

All we need to do now is to find the actual /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_... device name during boot, format it, add it to /etc/fstab and mount it. Note: we add it to /etc/fstab so that it's mounted again after reboot.

Now we put all the pieces together into a Cloud Formation template. Something like should do:

Instance:
  Type: 'AWS::EC2::Instance'
  Properties:
    [...]
    UserData:
      Fn::Base64: |
        #!/bin/bash -x
        IDX=1
        for DEV in /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_*-ns-1; do
          mkfs.xfs ${DEV}
          mkdir -p /local${IDX}
          echo ${DEV} /local${IDX} xfs defaults,noatime 1 2 >> /etc/fstab
          IDX=$((${IDX} + 1))
        done
        mount -a

Once the instance is created and booted you should have your Instance / Ephemeral / SSD storage mounted under /local1:

[ec2-user@ip-172-31-10-228 ~]$ mount | grep /local1
/dev/nvme1n1 on /local1 type xfs (rw,noatime,attr2,inode64,noquota)

Here the canonical device name /dev/nvme1n1 is displayed, not the /dev/disk/by-id/.. device alias. But it's still the same disk.

Hope that helps :)

Related Topic