I would suggest using sar
from the sysstat
(or atsar
) package. This run every 10 minutes as a cron job and makes a note of your server's vital statistics -- memory usage, cpu utilization, disk activity, network activity, etc.
You use it like this:
Show processor activity (the default)
sar -p
(or just sar
)
Show memory ("ram") statistics
sar -r
Show the memory statistics from the 27th
sar -r -f /var/log/sysstat/sa27
Note that the path varies based on your installation. On redhat-based systems, the files are usually in /var/log/sa/
, while if you have the atsar
package installed, they'll be in /var/log/atsar/
-- but the pattern is that the file will end in a number that represents the day of the month when the data was gathered.
Some versions (like atsar
) allow you to simply specify the day: sar -n 27
. Check the manpage that came with your installation to find out the correct syntax and what data you can retrieve.
Once you have this installed and running (and you probably do already!) you can use the information it gathers to get an idea of what was going on immediately before the crash. For example, if the report shows your memory to be exhaused and free swap space counting down to zero, then you'll have a pretty good idea of what to look for.
With the information in hand, you can set up additional reporting to give you a better idea of what's wrong: for example, you can write a short bash script that examines certain system statistics (such as the contents of /proc/meminfo
or /proc/loadavg
) and if the trigger conditions are met, perhaps appends the appropriate debugging information (like the output of ps auwwxf
) to a file, or emails the information to you.
You might want to remove exec from your script. It does the same thing as putting the script brackets. Should look like
script
/usr/local/bin/orbited --config=/etc/orbited.cfg
end script
Hope this helps
Best Answer
D'oh, alright I figured it out... dbus isn't installed for some reason although services were working before..? Not sure what happened, anyways
sudo apt-get install dbus
worked.