Why are GlusterFS replicated volumes not recommended for Hosts in different datacenters

glusterfsiredmailopenvzreplicationvirtualhost

Any tutorial I can find about GlusterFS replicated volumes assumes that both (all) bricks are on the same private network which then also leads to the conclusion they must be in the same datacenter.

e.g. "The problem is when the storage you would like to replicate to is on a remote network, possibly in a different location, GlusterFS does not work very well. This is because GlusterFS is not designed to work when there is a high latency between replication nodes." is a quote from https://github.com/GlusterFS/Notes

Also https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Geo%20Replication/ says that replicated volumes are not meant for geo-replication however the real "geo-replication" mechanism in GlusterFS only creates read-only slaves which won't work in every scenario.

So the question is, why isn't it recommended in general, since I haven't found a single example of replicated volumes for hosts on different networks, yet even different datacenters.

I can also explain why I want to use replicated volumes. I have a vServer (OpenVZ) in a datacenter in Frankfurt, Germany and a second in Nuremburg, Germany. Both have multiple peerings with DE-CIX, Deutsche Telekom, and so on and the latency between the vServers is < 4 ms which in my opinion cannot be considered high latency, whatever the definition for that might be in the case of GlusterFS.

I am running iRedmail services on both servers and MariaDB is replicated in Master-Master replication storing only the mail configuration. The mail storage is on disk and I am using GlusterFS replicated volumes to replicate it. I see no issues so far (mail storage is about 20 GB of e-mails including attachments) and am wondering if I am just lucky or if there are problems I just didn't detect yet. Anyway, I prefer to follow best practices which I didn't do in this case and I am wondering what you think about GlusterFS replicated volumens for hosts in different datacenters and what "high latency" actually means.

Best Answer

This issue applies to many types of data stores not just GlusterFS. This is because increased distance increases latency. The recommendation to be on the same subnet is also to reduce latency due to network hops.

In order to maintain data synchronization, the various servers must ensure that all servers have the same view of the data. For data reads, the latency effect is usually not an issue. However, serious data corruption can occur if multiple servers write the same block before they are synchronized. When a data block is being updated it is possible to loose changes, if the block being updated was read before a subsequent update on a different server data will likely be lost.

Locking mechanisms can be used to reduce the risk of corruption. However, distributed locks take longer to obtain and release as latency increases. In this case, latency it the time to complete a round-trip between servers. There are three contributing factors when communicating between data centers.

Mail data stores tend to be relatively read mostly. Normally, it is unlikely that multiple clients attached to different servers would be updating the same file or directory. There may be some contention between incoming email messages and clients reading them, but the latency should not be a significant issue. Maildir format stores should have relatively lower contention that other formats. However, they have relatively high rename and move activity which may cause issues if your nodes become disconnected.

  • Distance: Wire data travels over wire at about 30 cm in a nanosecond, 300 meters in a a microsecond, or 300 kilometers in a millisecond. This adds significant latency as distance increases.
  • Switching time: Each switch a packet passes through need to be examine, route, queue and transmit the packet. This adds additional latency which increases as the switch gets busier.
  • Network congestion: Networks can get congested causing additional delays as traffic is queued longer and possibly re-routed. If congestion is bad, the delays may be long enough to trigger packet re-transmission.
Related Topic