Centos – Can Apache Read The GlusterFS Brick Directly But Write To The GlusterFS Mount

While afs looks to be an obvious solution, I'd previously looked at this in some depth for a highly custom web application - and the fastest / most efficient / reliable solution was to implement to replication within the app - using rsync when bringing nodes back online. I did have a longer term plan to implement demand-based resynchronisations using inotify as the trigger (but never found the time).

As a solution, I was going to mount a RAM drive (tmpfs) that mirrors the GlusterFS web files but get the webserver to use the RAM drive

Yes, if GlusterFS does not support synchronization of cache invalidations, then it might be slightly faster (at the cost of not updating in real-time) although unless you have a very high data turnover, you'll probably find it faster to use a optimized file-system on a conventional device rather than a RAM drive.

You're probably going to see similar problems with any shared disk file systems (but this is mostly guesswork on my part).

A better solution would be to use a database (cluster?) to store any data - clustering is much more manageable (and easier to implement). See also mysql replication and Cassandra.

/me wonders if you could use overlay filesystems (unionfs) - putting the local copy on top and the remote system underneath then run rsync periodically from top to bottom - although I suspect it may prove difficult to delete files.

HTH

C.

Let's say you have two machines, A and B. On each machine, you export /opt/files as a Gluster brick, and set up client-side replication. We then mount the resulting directory as /mnt/gluster-files on both machines. This is important!

Using that mount point, we now have a highly available file system across the two machines.

When you write a file - let's say /mnt/gluster-files/example on machine A, it will cause two things to happen:

1. Write a copy to /opt/files
2. Send a copy over the network to be written to /opt/files on machine B.

This is good, because we want to have redundancy, which means we have to have more than one copy of the data.

Next up, let's say we want to read the same file. Again on machine A:

1. You issue a read for /mnt/gluster-files/example
2. GlusterFS says "I need to check all the replica nodes to find out who has the most recent version of this file"
3. GlusterFS checks every node
4. It turns out that all copies are the same, because replication is working nicely
5. You are returned the file from your local disk. §

(§ There is a read-subvolume client option, and it is sensible to set it to the local volume on any machine that is a Gluster client and server, as in this case. Otherwise, step 5 could be 'you are sent the file from a random node'.)

Behind the scenes, GlusterFS keeps /opt/files on both machines in sync. Checking every node, especially for a large number of small files, adds a not-insignificant performance penalty.

The question is therefore raised: if I am running a process on one of these two machines, and I know the files are in sync, why can't I just read the files from the local share?

It's not recommended, but you can do this. Read the files from /opt/files. Manually keep track of if you get out sync, and if you do, do something like a ls -laR in /mnt/gluster-files which will trigger a synchronization.

So, what happens if you write to /opt/files on machine A?

The file sits there unnoticed by GlusterFS. Gluster doesn't work that way. It doesn't get onto machine B unless you happen to do something which makes Gluster notice it on machine A.

Therefore, you can't just tell Apache to read and write to /opt/files. What seems like a good compromise is telling it to read from /opt/files but write to /mnt/gluster-files. This is only possible if your application lets you specify a different path for reading and writing files, which not many do.