Configuring RHEL6.1 GFS cluster

I have worked a little with clusters. These are my opinions on the subject.

could have simply added force_unmount="1",  but I would like to know what is
the reason behind the default behavior. Why is it not unmounted? 

If you choose to configure gfs as a clustered resource, and add the clvmd and gfs disk as resources, then when you failover with rgmanager it will try to umount the disk, so what I'd do in your case first is check logs (or lsof/fuser etc) for an indication why the umounting might have failed. Likely there is a process having a file open or something like that, preventing a "clean" umount.

Can it be because you don't use rgmanager to start your clustered application? I don't see it in your cluster.conf. That would if true explain the behaviour.

If you choose to force_unmount, what rgmanager will do when failing over/recovering is forcefully killing any recourse using the disk before umounting the disk. Weather that is a good idea or not depends.

clvm is stopped, but GFS is not unmounted, so a node cannot alter LVM structure 
on shared storage anymore, but can still access data. And even though a node can 
do it quite safely (dlm is still running), [...]  
Moreover if I later try to stop cman on that node, it will find a dlm locking,
produced by GFS, and fail to stop.

If you want to change LVM structure in this scenario you can start the clvmd daemon again manually. if you umount the gfs disk before stopping cman, that should work. On the other hand, in a production scenario I rarely find myself in a situation where I'd want to stop CMAN on a clustered node.

My preference is to go with option 4.

If I understand the latest approach correctly, such cluster only controls 
whether nodes are still alive and can fence errant ones, but such cluster
has no control over the status of its resources.

It is true that if you don't add gfs2 and clvmd resource as a cluster resource, rgmanager won't be able to control it. What I usually do when setting upp A/A clusters (depending on the case of course) is that I'd add the start script for my service as the clustered resource. (rgmanager will then call the script with status argument on a regular basis to determine weather it needs to take configured action). Since my script has a dependency on the gfs file system it will fail unless it is mounted.

The 4 approach implies manually enabeling clvmd, cman and gfs2 (and possibly other daemons too depending on the situation).

Since the GFS fileystem sits on top of a iSCSI device, adding _netdev option to the mount in /etc/fstab is a requirement for it to work.

• This way I don't get an over-complicated cluster configuration, adding more services later will be less of a headache (say for example two services using the same disk or what ever)
• when something does happen, my experience is that manual intervention is a lot easier with resources not managed by rgmanager
• in my experience, it's not the gfs2 or clvmd services which goes wrong most in a cluster, but the services on top, so re-starting/mounting them often will only take you extra time.

There are a few disadvantages I can think of too:

• Like you said, rgmanager will not manage these resources, and will take no action if for example the gfs filesystem would somehow fail/get umounted
• having a gfs filesystem mounted a lot can generate unnecessary load on the device by for example updatedb and other jobs which might want to traverse the filesystem, thereby causing drive latency (locking traffic)

No matter what you decide

I would add the init script as a clustered resource, and if you chose to add gfs and clvm to the cluster as resources, I'd consider adding the __independent_subtree attribute to it, so if it fails, rgmanager won't re-mount the gfs filesystem. This depends of course on your particular situation. Note the nested configuration in the link, marking a sort of dependency tree.

For your case I'd still stick with a network-filesystem. You've already run into the scaling problems with NFS, so it's time to look at something else. There are a couple of good options out there:

• Gluster. It's a RH project, so is extremely well supported on CentOS, and scales up to "way lots". Hundreds/thousands of client is entirely doable, especially in a read-heavy environment like you seem to be in. I'm currently using it with Ubuntu, so there is support in debian-land.
• Ceph. Like Gluster, it's a next-gen network file-system that also provides a direct-mount capability. Also designed for "way lots" scaling. It isn't tied to Red Hat like Gluster.

Both are currently being used in cloud-scale infrastructure.

Direct-mount filesystems like gfs2 and ocfs really, really do have scaling bottlenecks. The cross-system file-locking problem, not to mention cross-host cache-coherency, is rather hard to solve and is the major scaling problem. Other clustered file-systems like VMware's VMFS also have "small tens" maximal mount limitations for the same reason.

Network-filesystems that aren't NFS have been designed to scale to very large concurrencies. The options I mention above both have duplication and even distributed volume support to help prevent single-points-of-failure.