I'm in the field of computer science and I do research in bioinformatics. Currently 746 on Biostars :)
I have been operating the bioinformatics compute facilities for 3 years at a university (about 40 Linux servers, 300 CPUs, 100TB disk space + backups, about 1T RAM total - servers ranging 16 to 256GB of RAM). Our cluster has 32 8-core compute nodes, 2 head nodes, and we are expanding it with 2 more 48-core compute node. We serve the files to the compute nodes over NFS.
I would recommend switching to NFS for your situation.
We considered switching to Gluster, Lustre, and Samba but decided not to use those.
NFS
I have a few main tips about NFS:
- Have a dedicated NFS server. Give it 4 cores and 16GB RAM. A dedicated server is more secure and easier to maintain. It's a much more stable setup. For example, sometimes you need to reboot the NFS server - a dedicated server will not fail your disk accessing computations - they will simply freeze and proceed once NFS server is back.
- Serve to your compute and head nodes only. No workstations. No public network.
- Use NFS version 3. From my experience NFSv4 was more fragile - more crashes - harder to debug. We switched the cluster from NFSv3 to NFSv4 and back several times before settling. It's a local network so you don't need the security (integrity and/or privacy) of NFSv4.
Storage Hardware
Our current cluster was bought 3 years ago so it's not using SAS, but rather has an expansive FiberChannel drives and san controllers. This is changing, all the new storage that we are buying is SAS.
I would suggest considering a SAS storage. SAS is replacing FiberChannel as a cheaper, faster and a better solution. Recently I did research on the different solutions offered. Conveniently the options that we looked at are documented of Server Fault:
What are SAS external storage options (Promise, Infortrend, SuperMircro, ...)?
We recently ordered a 24TB 6Gb SAS - 6Gb SAS storage system from RAID Incorporated. Just for the storage we payed $12k. The order should come in a couple of weeks. This is a no-single-point-of-failure system - all components are redundant and automatically fail over if any components fail. It's attached to 2 servers each using a different partition of the array. It is a turn-key solution so once it's shipped we just need to connect it, power it on, and it will work (RAID6 partitions will be mounted on Linux). The order also included servers and RAID Incorporated are setting-up Linux Debian on those for no extra cost.
Other considerations
Unfortunately, if you do bioinformatics infrastructure operations you probably need to become a storage guru.
For your 10TB partition, pick RAID6 - 2 drives can fail without losing you data. Rebuilding a 2TB drive onto a hot spare takes 24 hours, another drives can fail during that time. I had 2 drives fail simultaneously in a 16 drive array.
Consider dedicating one drive to be a hot spare in the array. When you have more then 16 drives then I would say a hot spare is a must.
Think of a plan of action if hardware fails on the dedicated NFS server. I would keep a twin as a compute node as a potential replacement for the original NFS server.
Finally, I have to mention our file server is running OpenSolaris (sounds unusual - I know). OpenSolaris (as it turned out for us) has excellent server hardware support (FiberChannel, IniniBand, ...). Setting up an NFS server ground up takes 1 hour - all steps a completely straight forward: install os, update through a NAT, setup network, create a zfs pool, create zfs filesystems, share NFS. Sun were the ones who developed NFS in 1984, not surprisingly OpenSolaris is very good at serving NFS. The main reason to use OpenSolaris was ZFS - a good filesystem for bioinformatics. Some features that I like:
- Integrity (all writes are checksumed)
- Pooled storage, snapshots
- NFS exports are configure in the served filesystem
- Online compression
- Reservations (space guarantees)
- Block level Deduplication
- Efficient backups (see
zfs send).
Using Linux for your NFS server would be fine - in that case stick to XFS or Ext4.
Best Answer
Ok, so let me get this straight. You have direct-attached storage on a server but you want to share files on it with two other Windows 2008 servers? Why not use Samba and be done with it? Samba basically turns your Solaris server into a domain-member file server. Just install Samba, set up your file shares, and let the Windows 2008 servers access the files from there.
If you are wanting to do some kind of shared-cluster solution, you'll need to use a clustering filesystem. Yes, iSCSI is meant to provide block-level sharing. However, with a clustering filesystem, you would use something like iSCSI as a layer beneath it to share physical disk space.
I guess the issue is that you need to clarify what the 2008 servers will use the Solaris install for. This would at least give an idea about what kind of sharing is involved.