We are looking into building an iSCSI SAN for virtualization and general storage needs. At a prior job I had built an iSCSI SAN using opensolaris (back when it was SUN) and utilized snapshot send/recv for sending data to a dr site. This setup is still working well and has not required much attention.
Since the company I work for is growing we are now hitting our limitations and we are now looking at either purchasing a SAN or building it ourselves. At this point, we are wanting three independent nodes, one at our internal site and one at each of our DR locations.
We have narrowed it down to two options as we have attempted to look into the netapp/equallogic/hp msa series side and the cost is ridiculous for what you get.
Option one:
Purchase Oracle hardware (a 7410) for each location and utilize the built in replication software (Is this AVS?) to replicate between sites. It looks like it has 12TB RAW which means we may have 4TB after implementing raidz accross the volumes. This is an issue as we will basically be back where we started in terms of maxed out disk space. Yet we now have redundancy. We could add more shelves to get more storage, however at this point we are stuck using Oracle supplied hardware and we are locked in.
Option two:
Use Dell hardware, r710s and a dell md1200 arrays at each location. Purchase Solaris through Oracle ($1,000 per processor, so $3k). Where my issue is, is with the replication piece. I used to work with SUN AVS when it was opensource and it was amazing what it could do. I also used to work with snapshot send/recv (inefficient), are you still able to purchase AVS? If so does anyone have any figures on cost of this product? I see on Oracle's website That it is available as a download, however they state you must pay per TB transmitted. Any ideas on cost?
Does the above idea make sense i.e. sanity check? I just want to get the community's opinion and maybe see if others have gone down this path and if there are obstacles I should know about in a larger setup like this.
I have looked at zfsbuild.com and have scoured the internet looking for ideas, my only issue that I cannot find is the replication piece. Thanks in advance for any answers.
Best Answer
I feel like we answer variants of this question often on Server Fault.
You can do all of this with NexentaStor's commercial offering, using either asynchronous replication (ZFS Send/Receive over ssh/netcat or rsync) or synchronous replication (with another commercial plugin).
There are vendors who package and certify these setups with custom hardware. The easiest example is the Pogo Linux Storage Director series.
I personally own a Sun x4540 that used to run OpenSolaris, but is now running Nexenta Enterprise. The Sun hardware is a little sketchy after only two years though, and I've seen found more value in building my own. I'm currently using HP hardware to build single storage nodes, but I can expand to external storage with LSI enclosures. People have documented SuperMicro-based solutions as well. In the HP systems, I'm replacing the Smart Array controllers with LSI 6GB SAS controllers; 9211-8i for internal use, and 9200 for external. You would need to do something similar in the Dell case.