UPDATE:
If I do:
yum upgrade
I see:
Skip Packages With Broken Dependencies When
How do I fix this?
Older post:
I found the file erl_crash.dump here:
/var/lib/rabbitmq/erl_crash.dump
If I do this:
head -n 50 /var/lib/rabbitmq/erl_crash.dump
I get:
=erl_crash_dump:0.5
Tue Mar 1 20:00:36 2022
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<185,
System version: Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
Compiled: Fri Sep 6 18:47:55 2019
Taints: crypto
Atoms: 13923
Calling Thread: scheduler:1
=scheduler:1
Scheduler Sleep Info Flags:
Scheduler Sleep Info Aux Work: THR_PRGR_LATER_OP
Current Port:
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | EXEC
Current Process: <0.0.0>
Current Process State: Running
Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
Current Process Program counter: 0x0000ffff5cd5d020 (init:printable_list/1 + 24)
Current Process CP: 0x0000ffff5cd5dcc8 (init:boot_loop/2 + 1944)
Current Process Limited Stack Trace:
0x0000ffff5a93b9f0:SReturn addr 0xACD16DC0 (<terminate process normally>)
I'm not sure how to read this. Is failed_to_start_child the error? If yes, what does it mean? How do I fix it?
I've got a new EC2 instance, and a fresh install of Redhat8, with everything just installed with yum or dnf but I keep getting errors on RabbitMQ, which tries to start but fails instantly. If I do:
tail -n 500 /var/log/messages
Then I see this message, repeating over and over again:
Mar 1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: {"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,84,120,32,178,183,28,70,220,34,20,13,145,69,180,207,145,18,30,76,255,205,110,198,166,173,68,94,39,2,201,5,116,139,211,47,32,125,29,33,161,98,58,78,127,162,252,146,192,110,153,88,149,58,190,136,74,66,191,238,168,79,190,2,120,197,45,101,48,216,168,33,102,142,44,160,25,38,229,27,132,117,91,23,171,84,244,84,230,129,67,131,57,152,36,238,207,222,137,164,79,90,128,114,75,111,48,205,252,239,253,68,193,32,61,190,149,117,145,224,144,191,59,109,247,145,102,240>>,[<<167,114,26,126,33,160,146,70,65,253,176,24,5,41,155,193>>,<<0,0,0,1>>],32],[]},{credentials_obfuscation_pbe,xor_sum,7,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,160}]},{credentials_obfuscation_pbe,blocks,10,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,152}]},{credentials_obfuscation_pbe,make_key,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,94}]},{credentials_obfuscation_pbe,encrypt,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,76}]},{credentials_obfuscation_svc,check,3,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,163}]},{credentials_obfuscation_svc,init_state,0,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,133}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}"}
Mar 1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,8
Mar 1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: #015
Mar 1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Crash dump is being written to: erl_crash.dump...done
Mar 1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Mar 1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar 1 18:23:47 ip-172-31-56-22 systemd[1]: Failed to start RabbitMQ broker.
I'm struggling to think what the problem is. I believe I've opened up all the ports necessary.
Crash dump is being written to: erl_crash.dump...done
I have read this:
I have opened up port 4369 on the security group, but that did not help.
If I run:
epmd -names
I get:
epmd: up and running on port 4369 with data
So that is working, but RabbitMQ is not.
If I do this:
rabbitmqctl start_app
I get:
Starting node rabbit@ip-172-31-56-22 ...
Error: unable to perform an operation on node 'rabbit@ip-172-31-56-22'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@ip-172-31-56-22
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: ['rabbit@ip-172-31-56-22']
rabbit@ip-172-31-56-22:
* connected to epmd (port 4369) on ip-172-31-56-22
* epmd reports: node 'rabbit' not running at all
no other nodes on ip-172-31-56-22
* suggestion: start the node
Current node details:
* node name: 'rabbitmqcli-871-rabbit@ip-12-131-56-22'
* effective user's home directory: /var/lib/rabbitmq
* Erlang cookie hash: Gug6Jfak+f3/k+ww6vhA==
I have the impression that something failed to install. Whenever I use, I see something about RabbitMQ, even when my command has nothing to do with RabbitMQ. For instance, I do this:
sudo yum install screen
And part of the output is:
rabbitmq_rabbitmq-server 5.2 kB/s | 1.8 kB 00:00
rabbitmq_rabbitmq-server-source 3.4 kB/s | 951 B 00:00
I see those 2 lines whenever I use yum install and it makes me uneasy that RabbitMQ is mentioned with every use of yum install as if there is some weird failed cache issue.
I also see this:
[/usr/lib/tmpfiles.d/rabbitmq-server.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly.
Best Answer
Based on a couple of things I found:
it seems that upgrading RabbitMQ might solve this: https://www.rabbitmq.com/install-rpm.html