Linux bidirectional file sync over a WAN with immediate/incremental syncing

fileslinuxrsyncsynchronizationunison

Does anyone know of a solution that lets you sync files bidirectionally over a WAN, but also syncs immediately/incrementally?

Basically Unison is perfect for the file syncing, but it doesn't sync incrementally – it will wait for a file to finish changing before syncing, and unfortunately we need the sync process to start immediately.

Essentially the workflow is:

  1. User uploads file to Server A
  2. Server A starts syncing the file to Server B as it is being uploaded
  3. Upload finishes
  4. Server B does some processing, and writes some results to a directory
  5. Server B syncs the results back to Server A
  6. User downloads the results from Server A

Server A also does processing, but since it writes back to Server A it is not important for syncing purposes.

My first thought was to perhaps use DRBD in dual-primary with a clustered filesystem, but a) the WAN latency is a good 300ms and bandwidth can sometimes be poor (500K/s) and b) we can't change the current filesystems, and Server B is using a NAS (NFS) as its storage – I don't believe that combination (and using loopback devices) would be practical (correct me if I'm wrong).

Best Answer

I think GlusterFS can solve your problem.

It is a distributed filesystem that operates in userspace, so you don't have to change your current filesystem. With a gluster replicated volume you have a single virtual FS, which you can mount on your servers using FUSE module or NFS. This virtual FS is immediately synced between the nodes on your cluster.

Related Topic