Unable to add initial monitor to Ceph in Ubuntu

ceph

I am trying to set up a Ceph cluster. I have 4 nodes – 1 admin-node, 1 monitor and 2 object storage devices. The installation guide I am using is given at the following location: http://ceph.com/docs/master/start/quick-ceph-deploy/.

When I am trying to add the initial monitor (step 5 in the guide), I am getting the following error:

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cloud-user/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.21): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts worker-1-full
[ceph_deploy.mon][DEBUG ] detecting platform for host worker-1-full ...
[worker-1-full][DEBUG ] connection detected need for sudo
[worker-1-full][DEBUG ] connected to host: worker-1-full 
[worker-1-full][DEBUG ] detect platform information from remote host
[worker-1-full][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
[worker-1-full][DEBUG ] determining if provided host has same hostname in remote
[worker-1-full][DEBUG ] get remote short hostname
[worker-1-full][DEBUG ] deploying mon to worker-1-full
[worker-1-full][DEBUG ] get remote short hostname
[worker-1-full][DEBUG ] remote hostname: worker-1-full
[worker-1-full][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[worker-1-full][DEBUG ] create the mon path if it does not exist
[worker-1-full][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-worker-1-full/done
[worker-1-full][DEBUG ] create a done file to avoid re-doing the mon deployment
[worker-1-full][DEBUG ] create the init path if it does not exist
[worker-1-full][DEBUG ] locating the `service` executable...
[worker-1-full][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=worker-1-full
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[worker-1-full][WARNIN] monitor: mon.worker-1-full, might not be running yet
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[worker-1-full][WARNIN] monitor worker-1-full does not exist in monmap
[worker-1-full][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[worker-1-full][WARNIN] monitors may not be able to form quorum
[ceph_deploy.mon][INFO  ] processing monitor mon.worker-1-full
[worker-1-full][DEBUG ] connection detected need for sudo
[worker-1-full][DEBUG ] connected to host: worker-1-full 
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 5
[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 4
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 3
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 2
[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 1
[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
[ceph_deploy.mon][ERROR ] worker-1-full

"worker-1-full" is the node I am trying to set up as my monitor. The command I used is:

"ceph-deploy mon create-initial". Please help. Thanks in advance!

Best Answer

I faced the same errors was able to resolve the issue by adding my other ceph node's hostname & IpAdrress and by adding "public_network ="

The sections which I tweaked in ceph.conf are:

mon_initial_members =
mon_host =
public_network =

cat /etc/ceph/ceph.conf

[global]
fsid = 33cb5c76-a685-469e-8cdd-fee7c98c3f4d
mon_initial_members = ceph1,ceph2
mon_host = 192.168.61.39,192.168.61.40
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public_network = 192.168.61.0/24

And the running the command:

$ ceph-deploy --overwrite-conf mon create <ceph-node>

Related Solutions

Ceph OSD always ‘down’ in Ubuntu 14.04.1

I experienced the same issue in very much the same environment. I finally tracked down the problem to a messed-up OSD UUID. What gave it away was the following line in the MON log (not the OSD log!):

... mon.minion-001@0(leader).osd e75 preprocess_boot from osd.0 10.208.66.2:6800/3427 clashes with existing osd: different fsid (ours: 71b33e7f-b464-4ba9-96b3-8c814921fea2 ; theirs: 5401be6f-b4ff-42ef-8531-78ee73772d5b)

I resolved the problem by first manually removing the OSD, destroying its file system and manually re-creating it from scratch. How the problem came into existence is something I will subsequently have to track down.

Given the fact that I used puppet to set up the OSDs and the reason for it to mess up is probably related to something particular to my environment means that the issue you are experiencing is likely to be a different one, but maybe you can check your MON log anyway. You will have to enable debugging on the MON, though, by stating something like this in ceph.conf:

[mon]
        debug mon = 9

The message in question is logged at level 7, so this gives you some more details without making everything terribly chatty.

@LoicDachary: wouldn't it make sense to log this error/warning message at level 0? I would certainly have spotted this issue earlier had it been logged right away.

Slow fsync() with ceph (cephfs)

Is ceph waiting the until the next journal commit when it hits an fsync call?

Yes, mostly. But it acts a little differently depending on the backend.

Under FileStore, there's a small journal buffer that can act as a small write-burst cache, but it's tiny. And yes, once it fills, it blocks to flush - across the entire cluster or PG.

Under BlueStore, there's no such buffer. And yes, bluestore blocks on every write to fsync to the Journal - all Journals in the PG. This is how BlueStore can remain very consistent and predictable in IOPS and writes. Under Bluestore, you want to move at least the write-ahead-log (WAL) off to an Enterprise SSD - because BlueStore will move the Journal and DB off to the same WAL partition - if there's enough room (you don't even have to specify them, just the WAL).

Enterprise SSDs as WAL/DB/Journals because they ignore `fsync`

But the real issue in this cluster is that you are using sub-optimum HDDs as Journals that are blocking on very slow fsyncs when they get flushed.

Even Consumer-grade SSDs have serious issues with Ceph's fsync frequency as journals/WAL, as consumer SSDs only have transactional logs and no real power-backup.

It's Enterprise SSDs that have large capacitors that allow the drive to continue operating after a power-loss. Thus, they can guarantee a successful write under a power-loss event.

The added benefit is that Enterprise SSDs typically ignore fsync commands from the OS! Because they can guarantee the success of the write, they immediately return the fsync request from the OS.

Thus, you get major performance gains when using an Enterprise-grade SSD as your WAL/DB/Journal.

Under FileStore, you'll see those delays go away, but you'll see inconsistent bursts of cache and then back down.

This is where BlueStore comes in, as BlueStore will guarantee a consistent IOPS and write across the board. But, you need WAL/DB/Journal on an Enterprise SSD to ignore those fsyncs.

At this time, Intel S3700s can be had in the used market for about $40/ea. Tiny investment for massive performance gains of unblocking fsyncs.

Some quotes (https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Bluestore_vs_Filestore):

Filestore writes everything to the journal and only starts to flush it to the data device when the journal fills up to the configured percent. This is very convenient because it makes journal act as a «temporary buffer» that absorbs random write bursts.

Bluestore can’t do the same even when you put its WAL+DB on SSD. It also has sort of a «journal» which is called «deferred write queue», but it’s very small (only 64 requests) and it lacks any kind of background flush threads. So you actually can increase the maximum number of deferred requests, but after the queue fills up the performance will drop until OSD restarts.

And: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/

The BlueStore journal will always be placed on the fastest device available, so using a DB device will provide the same benefit that the WAL device would while also allowing additional metadata to be stored there (if it will fit). This means that if a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device.

Best Answer

Related Solutions

Ceph OSD always ‘down’ in Ubuntu 14.04.1

Slow fsync() with ceph (cephfs)

Enterprise SSDs as WAL/DB/Journals because they ignore fsync

Related Topic

Enterprise SSDs as WAL/DB/Journals because they ignore `fsync`