Mongo server does not start on Centos 6.4

centos6.4mongodb

I have a mongo server installed and when I try to start it with, always fail.

OS: CentOS release 6.4 x86_64
Mongo version: 2.4.8

I have other machines working with the same versión and puppetized working fine.

Error:

/home$>service mongod start
Starting mongod: Thu Dec 5 13:48:47.840
terminate() called, printing stack (if implemented for platform):
0xde05e1 0x6cf3de 0x291d0a34be6 0x291d0a34c13 0x291d0a34d0e 0xdf7bb1
0xdd6a51 0xdd7309 0xdd59a5 0xdd5b09 0x70088c 0x700d2c 0x7010d2
0x701111 0x701563 0x6dd695 0x6df459 0x291d0168d1d 0x6cf1c9
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo11myterminateEv+0x3e) [0x6cf3de]
/usr/lib64/libstdc++.so.6(+0xbcbe6) [0x291d0a34be6]
/usr/lib64/libstdc++.so.6(+0xbcc13) [0x291d0a34c13]
/usr/lib64/libstdc++.so.6(+0xbcd0e) [0x291d0a34d0e]
/usr/bin/mongod(_ZN5boost11filesystem36detail6statusERKNS0_4pathEPNS_6system10error_codeE+0x231)
[0xdf7bb1]
/usr/bin/mongod(_ZN5mongo11ProcessInfo16checkNumaEnabledEv+0x41)
[0xdd6a51]
/usr/bin/mongod(_ZN5mongo11ProcessInfo10SystemInfo17collectSystemInfoEv+0x3a9)
[0xdd7309]
/usr/bin/mongod(_ZN5mongo11ProcessInfo20initializeSystemInfoEv+0xa5)
[0xdd59a5]
/usr/bin/mongod(_ZN5mongo36_mongoInitializerFunction_SystemInfoEPNS_18InitializerContextE+0x9)
[0xdd5b09]
/usr/bin/mongod(_ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_+0xc)
[0x70088c]
/usr/bin/mongod(_ZNK5mongo11Initializer7executeERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x13c)
[0x700d2c]
/usr/bin/mongod(_ZN5mongo21runGlobalInitializersERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x32)
[0x7010d2]
/usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x21)
[0x701111]
/usr/bin/mongod(ZN5mongo26runGlobalInitializersOrDieEiPKPKcS3+0x3b3)
[0x701563] /usr/bin/mongod() [0x6dd695] /usr/bin/mongod(main+0x9)
[0x6df459] /lib64/libc.so.6(__libc_start_main+0xfd) [0x291d0168d1d]
/usr/bin/mongod(__gxx_personality_v0+0x499) [0x6cf1c9] Thu Dec 5
13:48:47.847 Got signal: 6 (Aborted).

Thu Dec 5 13:48:47.853 Backtrace: 0xde05e1 0x6d0559 0x291d017c9a0
0x291d017c925 0x291d017e105 0x6cf3e3 0x291d0a34be6 0x291d0a34c13
0x291d0a34d0e 0xdf7bb1 0xdd6a51 0xdd7309 0xdd59a5 0xdd5b09 0x70088c
0x700d2c 0x7010d2 0x701111 0x701563 0x6dd695
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
/lib64/libc.so.6(+0x329a0) [0x291d017c9a0]
/lib64/libc.so.6(gsignal+0x35) [0x291d017c925]
/lib64/libc.so.6(abort+0x175) [0x291d017e105]
/usr/bin/mongod(_ZN5mongo11myterminateEv+0x43) [0x6cf3e3]
/usr/lib64/libstdc++.so.6(+0xbcbe6) [0x291d0a34be6]
/usr/lib64/libstdc++.so.6(+0xbcc13) [0x291d0a34c13]
/usr/lib64/libstdc++.so.6(+0xbcd0e) [0x291d0a34d0e]
/usr/bin/mongod(_ZN5boost11filesystem36detail6statusERKNS0_4pathEPNS_6system10error_codeE+0x231)
[0xdf7bb1]
/usr/bin/mongod(_ZN5mongo11ProcessInfo16checkNumaEnabledEv+0x41)
[0xdd6a51]
/usr/bin/mongod(_ZN5mongo11ProcessInfo10SystemInfo17collectSystemInfoEv+0x3a9)
[0xdd7309]
/usr/bin/mongod(_ZN5mongo11ProcessInfo20initializeSystemInfoEv+0xa5)
[0xdd59a5]
/usr/bin/mongod(_ZN5mongo36_mongoInitializerFunction_SystemInfoEPNS_18InitializerContextE+0x9)
[0xdd5b09]
/usr/bin/mongod(_ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_+0xc)
[0x70088c]
/usr/bin/mongod(_ZNK5mongo11Initializer7executeERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x13c)
[0x700d2c]
/usr/bin/mongod(_ZN5mongo21runGlobalInitializersERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x32)
[0x7010d2]
/usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x21)
[0x701111]
/usr/bin/mongod(ZN5mongo26runGlobalInitializersOrDieEiPKPKcS3+0x3b3)
[0x701563] /usr/bin/mongod() [0x6dd695]
                                                       [FAILED]

I found similar problems but their fixes don't work.

MongoDB server crashed, showing the following error

When I try to start with

/usr/bin/mongod -f /etc/mongo.conf

Mongo starts without problems.

The file /etc/init.d/mongod is the standard file from mongo repos.

dbpath exists and has the right permissions

/home$>ll mongo

total 81936

drwxr-xr-x 2 mongod mongod 4096 Dec 5 12:54 db

drwxr-xr-x 2 mongod mongod 4096 Dec 5 15:25 journal

-rw——- 1 mongod mongod 67108864 Dec 5 15:25 local.0

-rw——- 1 mongod mongod 16777216 Dec 5 15:25 local.ns

-rwxr-xr-x 1 mongod mongod 4 Dec 5 15:25 mongod.lock

Best Answer

Check permissions under /sys/devices/system

You may be hitting the issue mentioned here (the new 2.6 release has a fix already checked in for this):

https://jira.mongodb.org/browse/SERVER-10623

The most common cause is that /sys/devices/system/node is not +rx for the user running the mongod process, but you can also hit it for others, not to mention the permissions on the parent folders.

Related Solutions

Openvpn – Setup openvpn client on centos 6.4

The problem seems to be the security context of some files.

The steps you need to take are:

check if trying to start the service generates AVC denials
```
# ausearch -m avc -ts today
```
it could be the case that there were no output. You could try to disable dontaudit rules temporarily
```
# semodule -DB
```
check the permissions and security labels of the affected files and/or directories
```
# ls -lrtZ /path/to/file
# ls -lrtdZ /path/to/dir
```
query the current policy to see what is the expected security label
```
# matchpathcon /path/to/file
# matchpathcon /path/to/dir
```
if the current security label and the expected security label don't match, restore it. Note that you can restore directories recursively
```
# restorecon -v /path/lo/file
# restorecon -v -R /path/to/dir
```
try again to start the service and repeat the steps if necessary

Changing Disk Encryption key in CentOS 6.4

The default encryption method uses LUKS, so issue:

# cryptsetup luksDump /dev/sda
LUKS header information for /dev/sda2

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4096
MK bits:        512
MK digest:      ec 80 b6 a6 f4 b6 23 29 d6 22 53 2c af c6 a6 06 01 ca 7c 34
MK salt:        97 4c c5 fe 39 e5 ec f7 3c 7a a5 bd 98 34 56 b4
                0d c4 cc c2 ab 74 fa 5c bd d3 0f e3 ae 65 c3 d7
MK iterations:  59625
UUID:           d8bfb76f-fec3-4d6e-8801-5ab24517c081

Key Slot 0: ENABLED
        Iterations:             238801
        Salt:                   42 37 13 4d 6f d5 eb e6 10 dc 78 c8 04 f9 10 74
                                48 15 4d e0 68 d5 dc 7f 6a 14 66 a2 88 83 4f 17
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

to see how many slots you are currently using. Replace /dev/sda with the appropriate block device.

Add a new key to a free slot using cryptsetup luksAddKey, check the possible options in the cryptsetup(8) manpage.

Afterwards, reboot and ensure you can access the machine using the new key. Then you can delete the old one (cryptsetup luksRemoveKey).

Best Answer

Related Solutions

Openvpn – Setup openvpn client on centos 6.4

Changing Disk Encryption key in CentOS 6.4

Related Topic