Linux – How to find faulty memory module from MCE message

linuxmemoryUbuntu

I am trying to understand MCE message to find which memory module is bad on a server. This message appears in /var/log/kern.log in one server that freezes two times today.

Apr 13 22:39:22 mbox kernel: [36247975.116860] sbridge: HANDLING MCE MEMORY ERROR
Apr 13 22:39:22 mbox kernel: [36247975.116867] CPU 0: Machine Check Exception: 0 Bank 5: 8c00004000010090
Apr 13 22:39:22 mbox kernel: [36247975.116869] TSC 0 ADDR 4a0d75900 MISC 21405cdc86 PROCESSOR 0:206d7 TIME 1428957562 SOCKET 0 APIC 0
Apr 13 22:39:22 mbox kernel: [36247975.951013] EDAC MC0: 1 CE memory read error

I suspect a bad memory module. The server is a 2x Xeon E5-2650 with 8x8Go memory modules (8 memory slots for each cpu)

Here is the memory module population from lshw:

 *-memory:0
      description: System Memory
      physical id: 2d
      slot: System board or motherboard
    *-bank:0
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-197.A
         vendor: Kingston
         physical id: 0
         serial: B83AE5C2
         slot: P1_DIMMA1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:1
         description: DIMM Synchronous [empty]
         product: Dimm1_PartNum
         vendor: Dimm1_Manufacturer
         physical id: 1
         serial: Dimm1_SerNum
         slot: P1_DIMMA2
         width: 64 bits
    *-bank:2
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 2
         serial: EC309238
         slot: P1_DIMMB1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:3
         description: DIMM Synchronous [empty]
         product: Dimm4_PartNum
         vendor: Dimm4_Manufacturer
         physical id: 3
         serial: Dimm4_SerNum
         slot: P1_DIMMB2
         width: 64 bits
    *-bank:4
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 4
         serial: E9305438
         slot: P1_DIMMC1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:5
         description: DIMM Synchronous [empty]
         product: Dimm7_PartNum
         vendor: Dimm7_Manufacturer
         physical id: 5
         serial: Dimm7_SerNum
         slot: P1_DIMMC2
         width: 64 bits
    *-bank:6
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 6
         serial: E7305738
         slot: P1_DIMMD1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:7
         description: DIMM Synchronous [empty]
         product: Dimm10_PartNum
         vendor: Dimm10_Manufacturer
         physical id: 7
         serial: Dimm10_SerNum
         slot: P1_DIMMD2
         width: 64 bits
 *-memory:1
      description: System Memory
      physical id: 3f
      slot: System board or motherboard
    *-bank:0
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-197.A
         vendor: Kingston
         physical id: 0
         serial: B63A08C3
         slot: P2_DIMME1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:1
         description: DIMM Synchronous [empty]
         product: Dimm1_PartNum
         vendor: Dimm1_Manufacturer
         physical id: 1
         serial: Dimm1_SerNum
         slot: P2_DIMME2
         width: 64 bits
    *-bank:2
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 2
         serial: EA309638
         slot: P2_DIMMF1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:3
         description: DIMM Synchronous [empty]
         product: Dimm4_PartNum
         vendor: Dimm4_Manufacturer
         physical id: 3
         serial: Dimm4_SerNum
         slot: P2_DIMMF2
         width: 64 bits
    *-bank:4
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 4
         serial: E7305938
         slot: P2_DIMMG1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:5
         description: DIMM Synchronous [empty]
         product: Dimm7_PartNum
         vendor: Dimm7_Manufacturer
         physical id: 5
         serial: Dimm7_SerNum
         slot: P2_DIMMG2
         width: 64 bits
    *-bank:6
         description: DIMM DDR3 1333 MHz (0,8 ns)
         product: 9965516-048.A
         vendor: Kingston
         physical id: 6
         serial: E7305B38
         slot: P2_DIMMH1
         size: 8GiB
         width: 64 bits
         clock: 1333MHz (0.8ns)
    *-bank:7
         description: DIMM Synchronous [empty]
         product: Dimm10_PartNum
         vendor: Dimm10_Manufacturer
         physical id: 7
         serial: Dimm10_SerNum
         slot: P2_DIMMH2
         width: 64 bits
 *-memory:2 UNCLAIMED
      physical id: 7
 *-memory:3 UNCLAIMED
      physical id: 9

As you can notice, there is no memory module on the bank #5 which. So my question is : do you agree this message is about memory failure? And if so, how can I find which module as to be replaced?

Best Answer

These errors are coming from the EDAC - Error Detection And Correction.

edac_mc class of the device.

The events that you are receiving are CE events (Correctible Errors). These are indications that a DIMM is beginning to fail.

EDAC has not reported any specific information about which memory row or channel it refers to so it's difficult to tell which one to replace until that one fail.

but have a look at /sys/devices/system/edac/mc/mc* and this might tell you a little bit more about which row / DIMM that might be the faulty one.

For example

ls -s /sys/devices/system/edac/mc/mc0
total 0
0 ce_count 0 csrow1 0 csrow4 0 csrow7 0 reset_counters 0 size_mb
0 ce_noinfo_count 0 csrow2 0 csrow5 0 device 0 sdram_scrub_rate 0 ue_count
0 csrow0 0 csrow3 0 csrow6 0 mc_name 0 seconds_since_reset 0 ue_noinfo_count

look at the ce_count field.

On a side note:

The system can still continue to operate, but with less safety. Preventive maintenance and proactive part replacement of memory DIMMs exhibiting CEs can reduce the likelihood of the dreaded UE (uncorrectible error) events and system 'panics'.

More info on EDAC here:

https://www.kernel.org/doc/Documentation/edac.txt

Related Solutions

Windows Memory – How to Find Memory Usage of Individual Windows Services

There is an easy way to get the information you are asking for (but it does require a slight change to your system):

Split each service to run in its own SVCHOST.EXE process and the service consuming the CPU cycles will be easily visible in Task Manager or Process Explorer (the space after "=" is required):

SC Config Servicename Type= own

Do this in a command line window or put it into a BAT script. Administrative privileges are required and a restart of the computer is required before it takes effect.

The original state can be restored by:

SC Config Servicename Type= share

Example: to make Windows Management Instrumentation run in a separate SVCHOST.EXE:

SC Config winmgmt Type= own

This technique has no ill effects, except perhaps increasing memory consumption slightly. And apart from observing CPU usage for each service it also makes it easy to observe page faults delta, disk I/O read rate and disk I/O write rate for each service. For Process Explorer, menu View/Select Columns: tab Process Memory/Page Fault Delta, tab Process Performance/IO Delta Write Bytes, tab Process Performance/IO Delta Read Bytes, respectively.

On most systems there is only one SVCHOST.EXE process that has a lot of services. I have used this sequence (it can be pasted directly into a command line window):

rem  1. "Automatic Updates"
SC Config wuauserv Type= own

rem  2. "COM+ Event System"
SC Config EventSystem Type= own

rem  3. "Computer Browser"
SC Config Browser Type= own

rem  4. "Cryptographic Services"
SC Config CryptSvc Type= own

rem  5. "Distributed Link Tracking"
SC Config TrkWks Type= own

rem  6. "Help and Support"
SC Config helpsvc Type= own

rem  7. "Logical Disk Manager"
SC Config dmserver Type= own

rem  8. "Network Connections"
SC Config Netman Type= own

rem  9. "Network Location Awareness"
SC Config NLA Type= own

rem 10. "Remote Access Connection Manager"
SC Config RasMan Type= own

rem 11. "Secondary Logon"
SC Config seclogon Type= own

rem 12. "Server"
SC Config lanmanserver Type= own

rem 13. "Shell Hardware Detection"
SC Config ShellHWDetection Type= own

rem 14. "System Event Notification"
SC Config SENS Type= own

rem 15. "System Restore Service"
SC Config srservice Type= own

rem 16. "Task Scheduler"
SC Config Schedule Type= own

rem 17. "Telephony"
SC Config TapiSrv Type= own

rem 18. "Terminal Services"
SC Config TermService Type= own

rem 19. "Themes"
SC Config Themes Type= own

rem 20. "Windows Audio"
SC Config AudioSrv Type= own

rem 21. "Windows Firewall/Internet Connection Sharing (ICS)"
SC Config SharedAccess Type= own

rem 22. "Windows Management Instrumentation"
SC Config winmgmt Type= own

rem 23. "Wireless Configuration"
SC Config WZCSVC Type= own

rem 24. "Workstation"
SC Config lanmanworkstation Type= own

rem End.

Linux – “No irq handler for vector (irq -1)” after hibernation

Try the kernel-option noirqdebug. This might help here...

BTW: What devices use which irqs before and after hybernation?

Is anything else wrong - are some devices disabled after hybernate?

After googling around I found this: This looks like your problem

Best Answer

Related Solutions

Windows Memory – How to Find Memory Usage of Individual Windows Services

Linux – “No irq handler for vector (irq -1)” after hibernation

Related Topic