Nfs – DRBD vs. GlusterFS for replication

drbdgitglusterfsnfsshared-storage

I need to build a solution to host internal git repositories. It needs to supports hundreds of thousands (or more) repositories.

I plan on using multiple "dumb" servers with a shared storage, so basically when a client is trying to access a repository – it will be redirected by the load-balancer to any of the available servers. Any change to the repository – will be replicated across all nodes.

My first thought was to use GlusterFS for that, but I've read it doesn't handle well with small files. I'm also thinking of replicating everything myself using DRBD, but this requires more setup and seems more complicated when comparing to GlusterFS.

Which one of the two provides better performances? Basically the problem I'm trying to solve is that when any of the servers goes down – I want others to still be able to serve the data.

Best Answer

This is a classic scale-out use case, and IMO GlusterFS should fit the bill. You can give it a try - just bring a few VMs up, set up a few bricks to be used for repository storage and run a stress test.

DRBD is not an option here anyway - it doesn't scale. If anything, I'd look at other object storage projects (Swift for example), if Gluster doesn't work well enough, but none of them are extremely performance oriented

Related Solutions

Mysql – STONITH with a DRBD/Pacemaker/Corosync 2 node cluster

The best thing to do is test what actually happens under different failure modes, to make sure there is no single failure that could cause both MySQL servers to try to become masters.

Test disabling the internet connection on one server. See what happens on both servers, and watch what happens when you bring it back up.

Do the same for any redundant connections. Then do the same with disabling ALL network connections at once.

One reason for not doing STONITH on a two node cluster is it is fairly easy to end up with both nodes trying to kill the other, and actually succeeding. You need to test your setup to make sure that they don't either both shutdown, or both keep running as masters and get your database out of sync.

One other thing I recommend, while you are testing it, before it goes into production: Intentionally break it. Do something that will cause mysql and drbd to get out of sync, and learn how to fix it. Write down what you needed to do to fix it. Because it's much better to know how to do that BEFORE you really need to.

Linux – Setting up DRBD with only one node

You can bring DRBD up without a peer and without starting the DRBD service:

# modprobe drbd
# drbdadm create-md <res>
# drbdadm up <res> 
# drbdadm primary <res> --force
-- now you can use /dev/drbdX --

You could use DRBD to replicate from the existing server to the new server by installing DRBD on the existing storage.

If you're data is currently sitting on LVM you can unmount the logical volume, grow the logical volume (don't grow the filesystem) to make room for DRBD's metadata at the end of the device, set the disk option in the DRBD configuration to the logical volume, create the DRBD's metadata, and then force it to become Primary (with or without a peer). The rule of thumb for how much room DRBD needs for metadata is 32MB per 1TB of data. It would look something like this (assuming a 1TB LV):

# umount /dev/vg_name/lv_name
# lvextend -L +32M /dev/vg_name/lv_name
# drbdadm create-md <res>
-- will warn you if it finds existing data at the end of the device --
# drbdadm up <res> 
# drbdadm primary <res> --force

If you're not using LVM, you'd need to either shrink the filesystem to make room for DRBD's metadata, or use external metadata (described in the user's guide: https://www.drbd.org/en/doc/users-guide-84/ch-internals#s-metadata).

Once you bring DRBD up on the new server, it should start syncing your data. Just make sure the new server's backing disk is equal or larger in size.

Best Answer

Related Solutions

Mysql – STONITH with a DRBD/Pacemaker/Corosync 2 node cluster

Linux – Setting up DRBD with only one node

Related Topic