I'm seeing some strange behaviour when starting a process with an init script. Most likely I am doing something stupid, but perhaps someone could let me know what might cause this behaviour:
[Background: the process is mongodb's monogod running as a config server (2.4), and I'm running on Debian 6. The process runs under user 'mongodb', and I'm issuing commands via SSH as a non-root user]
- When the server is booted, the process appears to start normally (I can see it via ps -ef, and there is a mongodb.pid file generated).
- If I try to stop the process via
sudo /etc/init.d/mongodb stop
, then the weirdness begins : I getmongodb apparently not running
. (The process and the pid are still there though). - Of course, if I try to
sudo /etc/init.d/mongodb start
now I get 'failed' and an error in the log about the port already in use, which is to be expected as there is actually already an instance running. - If I kill -15 the process and then try
sudo /etc/init.d/mongodb start
I get more weirdness: In the console I getStarting database: mongodb failed!
(no more info). But… the process is now running and in the logs I don't see any errors, it just looks as if the process has started normally. - If I try to stop this process with
sudo /etc/init.d/mongodb stop
I get themongodb apparently not running
.
The script in question is here.
Any ideas? Thanks!
EDIT: In case it's something mongo related, here is my mongodb.conf for the config server:
configsvr=true
dbpath=/data/configdb
logpath=/var/log/mongodb/mongodb.log
logappend=true
port = 28222
bind_ip = a_public_ip,127.0.0.1,a_private_ip
auth = true
keyFile = /srv/mongodb/keyfile
nohttpinterface = true
fork = true
Additionally: I'm seeing Starting database: mongodb failed!
during the boot screen (but still there is a process seemingly running normally after boot).
Best Answer
If you have more than one instance running on that server, or something else that might be putting another value here:
That would cause the problem you are seeing here, it will be looking for the process with the ID that is in that file to determine if
mongod
and what PID to stop when shutting it down.You should also take a look at the permissions on that file to make sure that the process can access it and update it when it is starting up.
Basically, we need to have separate $NAME values for each init script you are using to avoid collisions, and make sure that the right PID value is making it into that file for the service to work as expected.
If everything looks good, then I would guess you are hitting the issues described here:
https://jira.mongodb.org/browse/SERVER-7254
numactl being invoked (as it should when installed) seems to cause the incorrect PID to be written to the PID file. We're looking for feedback troubleshooting this one, so if you could add your experience (and whether you have numactl installed) that would be appreciated.