Linux – centos 6 ps aux hangs up

kvm-virtualizationlinux

I have problem with my server. Server is running centos 6 (CloudLinux Server release 6.2). uname -a = 2.6.32-320.4.1.lve1.1.4.el6.x86_64
That is a kvm guest. On host is debian 6.

If i run command ps aux, it stuck on random process (shows some processes only), top command is working fine. htop doesn't work too (black screen).

top - 12:11:51 up 34 min,  1 user,  load average: 4.26, 6.71, 16.15
Tasks: 201 total,   7 running, 192 sleeping,   0 stopped,   2 zombie
Cpu(s):  7.9%us,  2.8%sy,  0.0%ni, 87.5%id,  1.6%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   9862044k total,  2359484k used,  7502560k free,   171720k buffers
Swap: 10485720k total,        0k used, 10485720k free,  1336872k cached

server has one Intel(R) Xeon(R) CPU E5606 @ 2.13GHz,

free -m

             total       used       free     shared    buffers     cached
Mem:          9630       2336       7293          0        170       1324
-/+ buffers/cache:        841       8789
Swap:        10239          0      10239

php -v

PHP 5.3.19 (cli) (built: Nov 28 2012 10:03:07)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
    with the ionCube PHP Loader v4.2.2, Copyright (c) 2002-2012, by ionCube Ltd., and
    with Zend Guard Loader v3.3, Copyright (c) 1998-2010, by Zend Technologies
    with Suhosin v0.9.33, Copyright (c) 2007-2012, by SektionEins GmbH

mysql Server version: 5.1.63-cll

php -i

disable_functions => apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_e
xec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, openlog, passthru, php
_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_set
sid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_de
code, xmlrpc_server_create, putenv, show_source,mail => apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval,
 exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore,
inject_code, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, pos
ix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exe
c, syslog, system, xmlrpc_entity_decode, xmlrpc_server_create, putenv, show_source,mail
...
suhosin.executor.disable_eval => Off => Off
suhosin.executor.eval.blacklist => include,include_once,require,require_once,curl_init,fpassthru,base64_encode,base64_decode,mail,exec,system,proc_open,leak,
syslog,pfsockopen,shell_exec,ini_restore,symlink,stream_socket_server,proc_nice,popen,proc_get_status,dl, pcntl_exec, pcntl_fork, pcntl_signal,pcntl_waitpid,
 pcntl_wexitstatus, pcntl_wifexited, pcntl_wifsignaled,pcntl_wifstopped, pcntl_wstopsig, pcntl_wtermsig, socket_accept,socket_bind, socket_connect, socket_cr
eate, socket_create_listen,socket_create_pair,link,register_shutdown_function,register_tick_function,gzinflate => include,include_once,require,require_once,c
url_init,fpassthru,base64_encode,base64_decode,mail,exec,system,proc_open,leak,syslog,pfsockopen,shell_exec,ini_restore,symlink,stream_socket_server,proc_nic
e,popen,proc_get_status,dl, pcntl_exec, pcntl_fork, pcntl_signal,pcntl_waitpid, pcntl_wexitstatus, pcntl_wifexited, pcntl_wifsignaled,pcntl_wifstopped, pcntl
_wstopsig, pcntl_wtermsig, socket_accept,socket_bind, socket_connect, socket_create, socket_create_listen,socket_create_pair,link,register_shutdown_function,
register_tick_function,gzinflate

Sometimes i cannot kill httpd process. I run kill -9 PID even several times, and nothing happens. php runs via suphp. I learned somewhere that it can be trojan.
I ran strace ps aux and it stops on

open("/proc/216456/cmdline", O_RDONLY)  = 5
read(5,

If i reboot server, problem is gone but after some time it is back again .. 🙁

Thanks.

Best Answer

Problem was with audit: backlog limit exceeded in /etc/audit/audit.rules i set -b from 320 to 9216 and problem is gone. Thanks everyone for help :)

Related Solutions

Linux Centos 6.2 “hangs” until date is synchronized

Do not use ntpdate on any scheduled basis to keep your clock synchronized. It was not meant for that, it causes problems, it's abusive to the server you're pulling time from. Don't do it.

If you need to keep a server's time synchronized, use ntpd. It is really not that hard to setup, there's no excuse for using anything else on *nix (and I run ntpd on Windows too).

If you have multiple machines on-site that need their clocks synchronized. Pick one to be the "ntp server" (the one that pulls time from an external source). You can pick two or even three if you have a lot of servers and need some degree of reliability/availability.

Pop this into the selected server(s) /etc/ntpd.conf

server 0.us.pool.ntp.org minpoll 7 maxpoll 12 iburst
server 1.us.pool.ntp.org minpoll 8 maxpoll 14
server 2.us.pool.ntp.org minpoll 8 maxpoll 14
server 3.us.pool.ntp.org minpoll 9 maxpoll 15

driftfile /var/db/drift.ntp

logfile /var/log/ntp.log
logconfig +sysall +syncall

Those last two lines about logging are entirely optional. Enable ntpd, how will depend on your system.

Now, the rest of the servers in your site: Configure them with a file similar to this:

server 192.168.1.1

driftfile /var/db/drift.ntp

logfile /var/log/ntp.log
logconfig +sysall +syncall

Again the logging bits are optional. Change the IP to that of your "selected" server. If you picked more than one server, duplicate that first line for each server. So if you picked two servers to be your time keepers:

server 192.168.1.1
server 192.168.1.2

Nothing to it. There are millions of more advanced configurations you could setup. I highly recommend starting with a simply configuration. Just get one server syncing to an external source. Once you've got that, set the other servers to sync to it. If you find you need more accuracy or higher availability just ask, and we can churn out the config.

Misconfiguring time synchronization is a pet-peeve of mine. This stuff has been around for many years and really isn't complicated. Running a public NTP Pool server isn't even that bad...

Linux – Random CentOS 5.x Server Hangs

this link is usefull concerning logs on centos : http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-logfiles.html

35.1. Locating Log Files

Most log files are located in the /var/log/ directory. Some applications such as httpd and samba have a directory within /var/log/ for their log files.

You may notice multiple files in the log file directory with numbers after them. These are created when the log files are rotated. Log files are rotated so their file sizes do not become too large. The logrotate package contains a cron task that automatically rotates log files according to the /etc/logrotate.conf configuration file and the configuration files in the /etc/logrotate.d/ directory. By default, it is configured to rotate every week and keep four weeks worth of previous log files.

read /var/logs/messages for general stuff look for access_log / error_log in apache directory / virtualhost for your website

you should read them all and try to get a clue to what is going wrong look in /var/log/messages and do some research on what you found and add it to your question so people may have more info to help you

note that logs have to be read from bottom to top as the last entries contains the last (error) data don't post everything if the file is too big , just the last 20 lines for example will be a good start (ex. tail -n 20 /var/log/messages )

Best Answer

Related Solutions

Linux Centos 6.2 “hangs” until date is synchronized

Linux – Random CentOS 5.x Server Hangs

Related Topic