We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file)
Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:
[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
Below core files had generated at these segmentation faults.
-rw------- 1 http http 65130496 Jun 20 01:28 core.6520
-rw------- 1 http http 66134016 Jun 20 03:27 core.6462
We are running on below Apache http version:
Server version: Apache/2.2.29 (Unix)
Server built: Mar 4 2016 15:21:54
I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?
Please let me know if further information is required help get more understanding pertinent to this issue.
can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.
UPDATE: Common Pattern
The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/
Below is the output of 'gdb full' during debugging:
#0 0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>) at pod.c:54
c = <value optimized out>
fd = 6
rc = <value optimized out>
#2 0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
threads = 0xc2bab0
rv = <value optimized out>
ts = 0xc079f8
thread_attr = 0xc07a18
start_thread_id = <value optimized out>
#3 0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
pid = 0
#4 0x000000000044efc0 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
j = <value optimized out>
free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137, 11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
total_non_dead = <value optimized out>
active_thread_count = <value optimized out>
idle_thread_count = <value optimized out>
free_length = 1
i = <value optimized out>
ps = <value optimized out>
totally_free_length = <value optimized out>
last_non_dead = <value optimized out>
#5 server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
status = 11
pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
i = <value optimized out>
child_slot = <value optimized out>
exitwhy = 6
processed_status = <value optimized out>
#6 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1782
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#7 0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
c = 0 '\000'
configtestonly = <value optimized out>
confname = 0x458a64 "conf/httpd.conf"
def_server_root = 0x458a4f "/app/***"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb44870
server_conf = 0xb44870
pglobal = 0xb3b128
pconf = 0xb3d138
plog = 0xb45178
ptemp = 0xb41158
pcommands = 0xb3f148
opt = 0xb3f238
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
signal_server = <value optimized out>'
Below modules are being used in httpd.conf file.
`cat httpd.conf | grep LoadMo
# have to place corresponding LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbd_module modules/mod_authn_dbd.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Below is the output of httpd -V (capital V)
./httpd -V
Server version: Apache/2.2.29 (Unix)
Server built: Jan 13 2016 15:10:57
Server's Module Magic Number: 20051115:36
Server loaded: APR 1.5.1, APR-Util 1.5.3
Compiled using: APR 1.5.1, APR-Util 1.5.3
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/app/****"
-D SUEXEC_BIN="/app/****/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
**OUTPUT of file * **
httpd.exp: script text executable for .
mod_actions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_advertise.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_basic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_digest.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_kerb.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_anon.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_file.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_host.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_owner.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_user.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_autoindex.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cern_meta.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgid.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav_fs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_deflate.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dumpio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_expires.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ext_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_headers.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ident.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_imagemap.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_include.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_info.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_config.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_forensic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_logio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_manager.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime_magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_negotiation.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ajp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_balancer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_cluster.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_connect.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ftp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_http.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_scgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_reqtimeout.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_rewrite.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_setenvif.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_slotmem.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_speling.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ssl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_status.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_substitute.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_unique_id.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_userdir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_usertrack.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_version.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_vhost_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
Best Answer
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.