So, few months ago my boss throws at me "what happens we needed to scale to X clients, in a reasonably short period of time." sort of scenarios, somewhat of an engineering challenge if anything else, but of course more clients means more income.
So, right now we have a setup similar to below. Redundant Load Balancers, serving data from two webservers (sessions shared via memcache), and two backend fully redundant File Servers. Each file server has the whole shabang, RAID6 Array across the drives, hot-space drive, dual raid controllers, dual nics, multipathing yada yada yada.
The design is one that I feel is solid, High Availability through no single-point-of failure. High Performance by splitting the load to multiple web servers, high scalability in the sense that we should be able to just keeping adding more and more machines horizontally to scale to more and more clients. The primary bottleneck being how much storage each file server could hold, and how much space is allotted to each client. That part of this would scale out faster then the rest of the system. FileServer/Client "Pools" route was chosen because it scales horizontally and is cheaper then saying "okay we need to buy an even BIGGER SAN."
So, this is all very easy, more and more online storage, means more and more NFS mounts. And that's where my second-guessing comes into play. I want to make sure I've ironed out all the potential issues in this design before putting anything in place. As long as each piece of this puzzle is properly monitored, I think its controllable, but I wanted to get some other opinions first, maybe from people who've been down this road before.
I know of a few key issues that will have to be watched for.
- Hotspots on the file servers, or particular machines working harder then the rest of the pool.
- Bandwidth & Backend Switching. being there will be a lot of talking between webservers to NFS devices, A high quality switch will have to be in place with a high switch fabric capacity.
Unknown issues…
- NFS Mounts on Webservers, Will there be any issues with having each webserver have 2… 5… 10… 100 NFS Mounts open at any given time? Is there any way to make this easier or more friendly? Maybe a NFS Proxy of some kind? (that'd create an internal bottle neck, which makes me frown). I thought about a Distributed file system, and have the webservers just mount that, However it seems that Non-Proprietary, POSIX Compliant, Expandable without downtime, Internally Redundant file systems are either too immature for production work, are outrageously expensive, or are really good at hiding from Google.
Let me know what you guys think, and if you see any suggestions & optimizations, it would be greatly appreciated.
((Due it it being an open-ended question where there is no one specific "right answer," question is a community Wiki))
Best Answer
In 90% of the hosting situations you will run into, you'll have far more webservers than storage servers which alters your network design quite a bit. You'll run storage servers in pairs, as many primary/primary fileservers don't support more than mirrored replication. Large NFS servers will likely handle a dozen or so webservers if you have a 10g backbone. Your connecting lines will be virtual connections as you'll run web lan on one vlan, storage lan on a separate vlan, gigE for the web lan, 10g for the storage lan depending on budget. You mention dual primary storage servers, and then mention NFS mounts which are somewhat exclusionary. Are they truly running shared nothing or is it a dual head/dual shelf/single fcal setup?
Run your load balancers in dual primary to reduce the cutover time and increase potential throughput. Don't use more than 40% of their capacity so that you have enough spare once you have a failure.
You also need to account for MySQL/PostgreSQL/Cassandra/etc clusters as well -- they don't particularly like NFS mounts.
Lefthand Networks has a distributed filesystem product. GlusterFS is somewhat mature and would work depending on your workload. MogileFS/Hadoop is another possibility. Also, take a look at Ceph, GFS and OCFS.
As for the number of mounts, you'll mount each NFS server once or possibly twice per webserver. Dozens of mounts wouldn't be unheard of, but, you might end up segregating your web clusters to eliminate having hundreds of mounts. Hadoop can present itself to your network as a single global filesystem across a distributed network.