Linux – Home-Brew Automatic Tiered Storage Solutions

archivelinuxssdstorageunix

Related to this question about using an SSD for system and HDD for data, except I would like my system to do this automatically…

Is it possible to have several layers of storage and push items automatically between them, using preferably free, open-source software?

I know of hugely expensive enterprise-class solutions like the EMC SAN->EMC Centera automatic archiving, but I was wondering if this sort of staged storage is possible to do automatically.

It would be nice to be able to have several layers in this : Memory->SSD->HDD->slower HDD or tape or some other archive solution.

Are there any filesystems which can do this automatically? (ZFS, Btrfs, HAMMER?)

Any Unix-variants are fine, as I'm interested in how this works and whether it's likely to be portable to Linux or other flavours (BSD etc).

Best Answer

Well, ZFS uses a storage layering called Hybrid Storage Pool (HSP):

  1. Layer: memory
  2. Layer: SSD-based read-caches (L2ARC) and write-caches (separate intent log aka slog).
  3. Layer: (cheap) harddisks

With HSP its easy to automatically benefit from the advantages of SSDs compared to a harddisk-only solution. A system using HSP can be both faster and cheaper than the latter. See this link for some nice examples and more details.

I think there are plans regarding hierarchical storage management (HSM) for ZFS (see for example the Automatic Data Migration (ADM) OpenSolaris project) but I don't know its current status.

Related Topic