Linux – Is it possible to hardware accelerate LUKS encryption

The general consensus seems to be that the answer to your question comes in two parts:

How do we find the source of the funny burning smell?

You've got the "How" pretty well nailed down:

• The "Sniff Test"
• Look for visible smoke/haze
• Walk the room with a thermal (IR) camera to find hot spots
• Check monitoring and device panels for alerts

You can improve your chances of finding the problem quickly in a number of ways - improved monitoring is often the easiest. Some questions to ask:

• Do you get temperature and other health alerts from your equipment?
• Are your UPS systems reporting faults to your monitoring system?
• Do you get current-draw alarms from your power distribution equipment?
• Are the room smoke detectors reporting to the monitoring system? (and can they?)

When should we troubleshoot versus hitting the Big Red Switch?

This is a more interesting question.
Hitting the big red switch can cost your company a huge amount of money in a hurry: Clean agent releases can be into the tens of thousands of dollars, and the outage / recovery costs after an emergency power off (EPO, "dropping the room") can be devastating.
You do not want to drop a datacenter because a capacitor in a power supply popped and made the room smell.

Conversely, a fire in a server room can cost your company its data/equipment, and more importantly your staff's lives.
Troubleshooting "that funny burning smell" should never take precedence over safety, so it's important to have some clear rules about troubleshooting "pre-fire" conditions.

The guidelines that follow are my personal limitations that I apply in absence of (or in addition to) any other clearly defined procedure/rules - they've served me well and they may help you, but they could just as easily get me killed or fired tomorrow, so apply them at your own risk.

1. If you see smoke or fire, drop the room
   This should go without saying but let's say it anyway: If there is an active fire (or smoke indicating that there soon will be) you evacuate the room, cut the power, and discharge the fire suppression system.
   Exceptions may exist (exercise some common sense), but this is almost always the correct action.

2. If you're proceeding to troubleshoot, always have at least one other person involved
   This is for two reasons. First, you do not want to be wandering around in a datacenter and all of a sudden have a rack go up in the row you're walking down and nobody knows you're there. Second, the other person is your sanity check on troubleshooting versus dropping the room, and should you make the call to hit the Big Red Switch you have the benefit of having a second person concur with the decision (helps to avoid the career-limiting aspects of such a decision if someone questions it later).

3. Exercise prudent safety measures while troubleshooting
   Make sure you always have an escape path (an open end of a row and a clear path to an exit).
   Keep someone stationed at the EPO / fire suppression release.
   Carry a fire extinguisher with you (Halon or other clean-agent, please).
   Remember rule #1 above.
   When in doubt, leave the room. Take care about your breathing: use a respirator or an oxygen mask. This might save your health in case of chemical fire.

4. Set a limit and stick to it
   More accurately, set two limits:

   • Condition ("How much worse will I let this get?"), and
   • Time ("How long will I keep trying to find the problem before its too risky?").

   The limits you set can also be used to let your team begin an orderly shutdown of the affected area, so when you DO pull power you're not crashing a bunch of active machines, and your recovery time will be much shorter, but remember that if the orderly shutdown is taking too long you may have to let a few systems crash in the name of safety.

5. Trust your gut
   If you are concerned about safety at any time, call the troubleshooting off and clear the room.
   You may or may not drop the room based on a gut feeling, but regrouping outside the room in (relative) safety is prudent.

If there isn't imminent danger you may elect bring in the local fire department before taking any drastic actions like an EPO or clean-agent release. (They may tell you to do so anyway: Their mandate is to protect people, then property, but they're obviously the experts in dealing with fires so you should do what they say!)

We've addressed this in comments, but it may as well get summarized in an answer too -- @DeerHunter, @Chris, @Sirex, and many others contributed to the discussion

One of the servers that I administrate runs the type of configuration that you describe. It has six 1TB hard drives with a LUKS-encrypted RAIDZ pool on it. I also have two 3TB hard drives in a LUKS-encrypted ZFS mirror that are swapped out every week to be taken off-site. The server has been using this configuration for about three years, and I've never had a problem with it.

If you have a need for ZFS with encryption on Linux then I recommend this setup. I'm using ZFS-Fuse, not ZFS on Linux. However, I believe that would have no bearing on the result other than ZFS on Linux will probably have better performance than the setup that I am using.

In this setup redundant data is encrypted several times because LUKS is not "aware" of Z-RAID. In LUKS-on-mdadm solution data is encrypted once and merely written to disks multiple times.

Keep in mind that LUKS isn't aware of RAID. It only knows that it's sitting on top of a block device. If you use mdadm to create a RAID device and then luksformat it, it is mdadm that is replicating the encrypted data to the underlying storage devices, not LUKS.

Question 2.8 of the LUKS FAQ addresses whether encryption should be on top of RAID or the other way around. It provides the following diagram.

Filesystem     <- top
|
Encryption
|
RAID
|
Raw partitions
|
Raw disks      <- bottom

Because ZFS combines the RAID and filesystem functionality, your solution will need to look like the following.

RAID-Z and ZFS Filesystem  <-top
|
Encryption
|
Raw partitions (optional)
|
Raw disks                  <- bottom

I've listed the raw partitions as optional as ZFS expects that it will use raw block storage rather than a partition. While you could create your zpool using partitions, it's not recommended because it'll add a useless level of management, and it will need to be taken into account when calculating what your offset will be for partition block alignment.

Wouldn't it significantly impede write performance? [...] My CPU supports Intel AES-NI.

There shouldn't be a performance problem as long as you choose an encryption method that's supported by your AES-NI driver. If you have cryptsetup 1.6.0 or newer you can run cryptsetup benchmark and see which algorithm will provide the best performance.

This question on recommended options for LUKS may also be of value.

Given that you have hardware encryption support, you are more likely to face performance issues due to partition misalignment.

ZFS on Linux has added the ashift property to the zfs command to allow you to specify the sector size for your hard drives. According to the linked FAQ, ashift=12 would tell it that you are using drives with a 4K block size.

The LUKS FAQ states that a LUKS partition has an alignment of 1 MB. Questions 6.12 and 6.13 discuss this in detail and also provide advice on how to make the LUKS partition header larger. However, I'm not sure it's possible to make it large enough to ensure that your ZFS filesystem will be created on a 4K boundary. I'd be interested in hearing how this works out for you if this is a problem you need to solve. Since you are using 2TB drives, you might not face this problem.

Will ZFS be aware of disk failures when operating on device-mapper LUKS containers as opposed to physical devices?

ZFS will be aware of disk failures insofar as it can read and write to them without problems. ZFS requires block storage and doesn't care or know about the specifics of that storage and where it comes from. It only keeps track of any read, write or checksum errors that it encounters. It's up to you to monitor the health of the underlying storage devices.

The ZFS documentation has a section on troubleshooting which is worth reading. The section on replacing or repairing a damaged device describes what you might encounter during a failure scenario and how you might resolve it. You'd do the same thing here that you would for devices that don't have ZFS. Check the syslog for messages from your SCSI driver, HBA or HD controller, and/or SMART monitoring software and then act accordingly.

How about deduplication and other ZFS features?

All of the ZFS features will work the same regardless of whether the underlying block storage is encrypted or not.

Summary

1. ZFS on LUKS-encrypted devices works well.
2. If you have hardware encryption, you won't see a performance hit as long as you use an encryption method that's supported by your hardware. Use cryptsetup benchmark to see what will work best on your hardware.
3. Think of ZFS as RAID and filesystem combined into a single entity. See the ASCII diagram above for where it fits into the storage stack.
4. You'll need to unlock each LUKS-encrypted block device that the ZFS filesystem uses.
5. Monitor the health of the storage hardware the same way you do now.
6. Be mindful of the filesystem's block alignment if you are using drives with 4K blocks. You may need to experiment with luksformat options or other settings to get the alignment you need for acceptable speed.

February 2020 Update

It's been six years since I wrote this answer. ZFS on Linux v0.8.0 supports native encryption, which you should consider if you don't have a specific need for LUKS.