Lvm – Encrypted storage for clustered MySQL

clusterencryptionlvm

I have a requirement to provide a highly available MySQL database with encryption-at-rest on a 2.6.32 linux kernel. The 'highly available' part isn't that hard, but the 'with encryption-at-rest' is proving to be a challenge when used in combination with HA.

The key problem comes with mounting the encrypted storage. On all of our other encrypted-at-rest systems there is a command that needs to be run by a human, who is then prompted for the encryption key. This model has a fairly obvious flaw when it comes to a cluster arrangement where services will have to start automatically.

I am currently at a loss over how to provide encryption-at-rest in an HA environment and not also store key passphrases on the same system.

I can see two possible scenarios, either of which will work with my environment but I'm not sure of the details to make them work. Or even if it is possible.

Scenario 1: CLVM & Cluster

  • A volume is shared between my cluster members.
  • This volume is set up roughly:
    • cryptsetup stuff on the physical device
    • LVM stuff on the new crypt-device
  • Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
  • Cluster Services are started via command run by a human, which supplies the decryption key, which in turn activates the CLVM stuff.

This way running nodes have access to the CLVM volume so they can start services when told to by the cluster manager. Reboots of nodes will still require a human, and the crypt passphrase is never kept on disk anywhere.

Scenario 2: DRBD & Cluster

  • A volume is created on each cluster member
  • cryptsetup stuff is run on the physical device
  • drbd is configured on top of the crypted device, to replicate it between each node
  • LVM or a filesystem is placed on top of the drbd volume
  • Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
  • Cluster services are started by a human who provides the decryption key, which in turn makes the LVM (or filesystem) visible-but-not-mounted.

As with the CLVM setup, nodes don't join the cluster until they have visibility into the presumably-shared storage.

The thing is, I'm not sure if either of the above work that way. Both assume it's possible to layer an LVM PV on top of an encrypted volume (e.g. pvcreate /dev/mapper/cryptmysql). This may not be possible.

Best Answer

The main challenge seems to be the human intervention for key entry. There is some help to that: dm-crypt has support for the TPM which might be available with your platform. See this IBM blueprint for configuration details. Also LUKS/cryptsetup supports reading a slot key from file / from stdin. If you can store your key safely somewhere (like on a smartcard), this might be a viable option.

As for your question whether you can have an LVM PV on a dm-crypt volume: you can, you just would need a run of pvscan / vgchange -a -y after slot unlocking. We've run this kind of setup with much older Kernels a couple of years ago. In the end, we've abandoned it in favor of the use of SEDs for applications with data-at-rest encryption requirements due to performance reasons (dm-crypt used to employ a single thread per encrypted device at that time, which led to a CPU bottleneck in our setup).

Related Topic