High speed network writes with large capacity storage

network-attached-storageperformancestoragezfs

I have a NAS running Samba with a 20T ZFS pool with one raid1 vdev with two spinning rust drives. I have 16G RAM in the machine right now. The storage is used for continuously growing, permanent backup archive of video footage. It's write once, read once for processing and then possibly backup restore.

I regularly fling 40GiB files to this NAS. I'm going to upgrade my gigabit network to 10GbE in order to make this process less painful. However I'm suspecting I'm going to become limited by the write speed of the underlying drives.

My understanding is that a ZIL and SLOG only accelerate synchronous writes so adding an nvme SSD as SLOG wouldn't affect my use case as I believe Samba is using asynchronous writes by default.

I'm not sure if configuring samba for synchronous writes and adding a SLOG on a nvme SSD would do what I need. I understand this comes with the risk of data loss if the drive fails or power cuts out. This is acceptable as I retain the files on the source machine long enough to retransfer in case of near term data loss. Wear and tear on the SSD is a concern but typical drives have 300 TBW or there about which is enough to fill my never-delete NAS 15 times over, or in 75 years at current data generation rate, I'm ok with that and to buy a new SSD if/when the SSD breaks. These are acceptable caveats. Normally I would just try and benchmark but in the current everything-shortage I'd like to know ahead of time what I need to purchase for this.

I know I can add more raid 1 vdevs to the pool to get a raid 10 pool but this is too expensive, the midtower chassis cannot support that many drives, it grossly over provisions the pool together with the existing drives and would use more energy over time to keep all that rust spinning.

What are my options for achieving write speeds in excess of 10Gbps to this zfs pool for at least 40GiB worth of data, aside from adding more spinning rust to the pool in a raid 10 fashion?

Best Answer

Synchronous writing mode ensures that the writes end up in a persistent location immediately. With asynchronous writes, data is cached in RAM and the write call finishes right away. The filesystem will schedule the actual writes to final location (hard disk).

In ZFS case, the point of ZIL / SLOG is to act as a fast interim persistent storage, that allows synchronous mode, that is, ensuring writing client that the writes are final. Otherwise the filesystem would need to write the blocks to the hard disk directly, which makes synchronous mode slow.

In your case, if you want to ensure full speed writing of 40 GB of data, then you should increase your RAM size to cover the size of the file.

However, since the FS starts writing to hard disks immediately, you don't need 40GB memory to get full speed for your writes. For example, when the client has written 20GB of data, 10GB could be in RAM cache and the rest 10GB already in hard drive.

So, you need to do some benchmarking to see how much RAM you need in order to get the full speed writes.

Related Topic