cephclusterUbuntuubuntu-15.10
I've rebooted the server (one of ceph's hosts). Started the cluster, but the osd, that's on the host, which was rebooted is down. The osd's # is 2, so when I try:
sudo /etc/init.d/ceph start osd.2
it shows:
Starting ceph (via systemctl): ceph.service.2
, but:
ceph osd tree
shows, that it's down.
When I try:
sudo start ceph-osd id=2
, it errors out:
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
How can I start that osd?
I experienced the same issue in very much the same environment. I finally tracked down the problem to a messed-up OSD UUID. What gave it away was the following line in the MON log (not the OSD log!):
... mon.minion-001@0(leader).osd e75 preprocess_boot from osd.0 10.208.66.2:6800/3427 clashes with existing osd: different fsid (ours: 71b33e7f-b464-4ba9-96b3-8c814921fea2 ; theirs: 5401be6f-b4ff-42ef-8531-78ee73772d5b)
I resolved the problem by first manually removing the OSD, destroying its file system and manually re-creating it from scratch. How the problem came into existence is something I will subsequently have to track down.
Given the fact that I used puppet to set up the OSDs and the reason for it to mess up is probably related to something particular to my environment means that the issue you are experiencing is likely to be a different one, but maybe you can check your MON log anyway. You will have to enable debugging on the MON, though, by stating something like this in ceph.conf:
[mon]
debug mon = 9
The message in question is logged at level 7, so this gives you some more details without making everything terribly chatty.
@LoicDachary: wouldn't it make sense to log this error/warning message at level 0? I would certainly have spotted this issue earlier had it been logged right away.
I did eventually work out what was wrong. I had to manually change 'type host' to 'type osd' in our crushmap, which is different from Spongman's suggestion.
after booting rgw, I find that the owner of radosgw process is "root", not "ceph". command "ceph -s" also show that "100.000% pgs not active".
I search the clue "100.000% pgs not active", the post "https://www.cnblogs.com/boshen-hzb/p/13305560.html" tell how to solve it - change 'type host' to 'type osd' , as result, "ceph -s" show "HEALTH_OK" and the owner of radosgw process become "ceph", and rgw web service(7480) is listening.
Best Answer
After about 2 days of trying to resolve this issue and banging my head against the wall, an other person's question to the similar issue on ceph's IRC channel, has led me to a solution:
where # is the number of osd on the host, that was rebooted, so I've used:
Remember to log into and run this command on the node that is down.