CentOS 7 DRBD – Can’t Run drbdadm up with DRBD 8.4

centos7drbdfdiskpartition

Nodes:

  • node1
  • node2

Use this way installed DRBD on both nodes:

rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
yum -y install drbd84-utils kmod-drbd84

Added new hard disk and do partition on both nodes:

fdisk /dev/sdb

Then format partition:

mkfs.ext4 /dev/sdb

(Failed on node2, so removed it and added a new disk become /dev/sdc. Formated it.)

Follow official guide to config the basic resource on both nodes:

http://docs.linbit.com/docs/users-guide-8.4/p-build-install-configure/#ch-configure

Created /etc/drbd.d/r0.res on both nodes:

resource r0 {
    protocol C;
    on node1 {
            device /dev/drbd0;
            disk /dev/sdb;
            address 192.168.0.1:7789;
            meta-disk internal;
    }
    on node2 {
            device /dev/drbd0;
            disk /dev/sdc;
            address 192.168.0.2:7789;
            meta-disk internal;
    }
}

Edit iptables on both nodes:

iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 7788 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 7799 -j ACCEPT
service iptables save

Initialize data on both nodes:

drbdadm create-md r0

All OK.

Put the resource up on both nodes: drbdadm up r0

node1:

Device '0' is configured!
Command 'drbdmeta 0 v08 /dev/sdb internal apply-al' terminated with exit code 20

node2:

??: Failure: (162) Invalid configuration request
additional info from kernel:
minor exists in different resource
Command 'drbdsetup-84 new-minor r0 0 0' terminated with exit code 10

Both of them happened issues.

Run cat /proc/drbd on both nodes:

node1:

version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
    ns:0 nr:0 dw:0 dr:912 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2097052

node2:

version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48
 0: cs:WFConnection ro:Secondary/Unknown ds:Diskless/DUnknown C r----s
    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Primary and Secondary unknown.

Run drbdadm connect all on node1:

r0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup-84 connect r0 ipv4:192.168.0.1:7789 ipv4:192.168.0.2:7789 --protocol=C' terminated with exit code 10

Run drbdadm -- --discard-my-data connect all on node2:

r0: Failure: (102) Local address(port) already in use.
Command 'drbdsetup-84 connect r0 ipv4:192.168.0.1:7789 ipv4:192.168.0.2:7789 --protocol=C --discard-my-data' terminated with exit code 10

What's wrong of using it?

Edit 1

After ran dd if=/dev/zero of=/dev/sdb(sdc) bs=1M oflag=direct status=progress on both nodes, they also can't ran drbdadm up r0 successfully:

node1

No valid meta data found
Command 'drbdmeta 0 v08 /dev/sdb internal apply-al' terminated with exit code 255

node2

??: Failure: (162) Invalid configuration request
additional info from kernel:
minor exists in different resource
Command 'drbdsetup-84 new-minor r0 0 0' terminated with exit code 10

Edit 2

At this time, ran systemctl start brdb on both nodes, all of them got pending. Wait for a long time returned no result.

And ran lsblk -a now:

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0   10G  0 disk
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0    9G  0 part
  ├─cl-root 253:0    0    8G  0 lvm  /
  └─cl-swap 253:1    0    1G  0 lvm  [SWAP]
sdb           8:32   0   16G  0 disk
sr0          11:0    1 1024M  0 rom
drbd0       147:0    0    2G  0 disk

Best Answer

You don't want to create the filesystem on your backing disks for DRBD (sdb and sdc); you want to create the DRBD first, and then format the resulting DRBD device with your filesystem. It can be done the other way, but then you'd need to grow the backing partition, or reduce the size of the filesystem to make room for DRBD's metadata (which lives at the end of the block device when using internal metadata).

The process should be something like this:

-Install all your software like you have done above.

-Zero out your partition on both nodes (optional, but do it):

# dd if=/dev/zero of=/dev/sdb bs=1M oflag=direct status=progress
# dd if=/dev/zero of=/dev/sdc bs=1M oflag=direct status=progress

-Create the configuration file for your DRBD resources like you have done above.

-Add the firewall rules like you've done above (except you don't really need the port 7799, as that's not used in your configuration).

-Create DRBD's metadata on the backing block devices, and bring up r0 on both nodes:

# drbdadm create-md r0
# drbdadm up r0
# cat /proc/drbd

-Check the output of cat /proc/drbd, you should see your device is Connected, Secondary/Secondary, and Inconsistent/Inconsistent. If any of those things are NOT true, STOP, something isn't right.

-Then, pick either node (NOT BOTH), force it to be primary (DRBD will not let you go primary on a node with Inconsistent data) and create the FS:

# drbdadm primary r0 --force
# mkfs.ext4 /dev/drbd0

Then you'd use /dev/drbd0 just like you would have used /dev/sda or /dev/sdc; don't touch /dev/sdb or /dev/sdc ever again unless you're confident in what you're doing. Touching DRBD's backing disks can introduce inconsistencies that DRBD will be unaware of (until you run a verify or otherwise overwrite the block).

Keep reading LINBIT's documentation. DRBD is obviously a great tool, and is easy to use once you understand the basics, but you're dealing with storage so there is a lot of room to mess things up in a permanent way.

Related Topic