DRBD – Why read-only on secondary resource would cause cache coherency issue

cachedrbd

According to the DRBD9 documentation (https://www.drbd.org/en/doc/users-guide-90/s-resource-roles), reading the device would cause cache coherency issue but i'm not sure about why. (I know i can't access a resource in secondary role but i want understand the reasons)

My hypothesis is (assuming we have DRBD-1 primary and DRBD-2 secondary, filesystem ext4 and "file_a" is a file on the drbd device):

DRBD-2 access file_a and places the file_a offset (or something like that) in his cache. (DRBD-2 have read-only rights, so the system thinks the file can not be modified).
DRBD-1 writes file_b content at the old file_a offset.
DRBD-2 wants to access file_a again, so the system uses the offset he stored in his cache, expecting to find file_a data while accessing file_b data.
We have cache coherency issue.

Thanks in advance for shed light me on about this point.

Best Answer

Yes, what you described it is the exact scenario that can lead to cache coherency issues.

To solve that problem, you had to use a filesystem designed to re-validate any cached entries accessed by other hosts. These are called clustered filesystem, with GFS2 and OCFS being primary examples.

But please note that for this very same reason, clustered filesystems are generally slower than traditional ones.

Related Solutions

Linux – DRBD Dual Primary Revisited

The kernel has an in-memory image of the state it thinks the file system is in and it doesn't check the disk to see if it might have changed, because this can't happen, as only the local kernel is allowed to change the file system and it knows what it does and doesn't need to check. If you make changes on the second node, the on-disk structures will be different from what the kernel expects and data-loss is nearly guaranteed.

And since cluster-aware file systems add quite a lot of synchronization and checks to the picture to avoid all kind of problems, it's not as easy as letting the kernel read the file system before every operation to make e.g. ext4 cluster capable.

Linux – drbd service not startable / WFConnection

I know this is an old post, but I thought I would provide a solution for those who want/need SELinux enabled. Do this by creating a local SELinux policy for DRBD.

If you grep the '/var/log/audit/audit.log' for "drbd_t", you should see some errors; piping those errors into 'audit2allow' will allow you to generate a new SELinux policy package that permits the previously denied actions. You can then install that policy with 'semodule -i':

# grep drbd_t /var/log/audit/audit.log | audit2allow -M drbdlocal
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i drbdlocal.pp

# semodule -i drbdlocal.pp

If you run into another SELinux error, you can rerun the above two commands to create/install a new policy package until you have no more errors.

Best Answer

Related Solutions

Linux – DRBD Dual Primary Revisited

Linux – drbd service not startable / WFConnection

Related Topic