I need your opinion if the drive below is failing.
When I run "smartctl -a /dev/sda -d megaraid,1", 2 errors are posted at the end of the output, stating "Error: WP at LBA". I don't see anything suspicious in the SMART parameters.
Here is the "smartctl -a /dev/sda -d megaraid,1" full output.
This HDD is one of the two HDDs from a RAID 1 (mirrored) hardware configuration, on a Dell H330 controller on a Dell PowerEdge server.
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-957.21.3.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MG03ACAxxx(Y) Enterprise HDD
Device Model: TOSHIBA MG03ACA300
Serial Number: 73VCK8GDF
LU WWN Device Id: 5 000039 4ebc82c58
Firmware Version: FL1A
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Feb 27 23:05:39 2020 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: ATA return descriptor not supported by controller firmware
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 510) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8874
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 27
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 068 068 000 Old_age Always - 12964
10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 27
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 6
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 25
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 42
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 31 (Min/Max 11/48)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 068 068 000 Old_age Always - 12994
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 103
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
ATA Error Count: 2
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 2 occurred at disk power-on lifetime: 12901 hours (537 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 10 0e fb 74 40 Error: WP at LBA = 0x0074fb0e = 7666446
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 08 00 48 7a e0 40 00 42d+20:47:35.187 WRITE FPDMA QUEUED
61 08 20 58 89 8a 40 00 42d+20:47:35.187 WRITE FPDMA QUEUED
61 10 20 48 89 8a 40 00 42d+20:47:35.187 WRITE FPDMA QUEUED
61 08 20 48 7a e0 40 00 42d+20:47:35.183 WRITE FPDMA QUEUED
61 08 20 40 89 8a 40 00 42d+20:47:35.183 WRITE FPDMA QUEUED
Error 1 occurred at disk power-on lifetime: 12901 hours (537 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 00 0e fb 74 40 Error: WP at LBA = 0x0074fb0e = 7666446
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 10 10 18 94 de 40 00 42d+20:47:32.312 WRITE FPDMA QUEUED
60 00 08 00 fc 74 40 00 42d+20:47:32.311 READ FPDMA QUEUED
60 00 00 00 fb 74 40 00 42d+20:47:32.311 READ FPDMA QUEUED
60 00 00 00 fa 74 40 00 42d+20:47:32.284 READ FPDMA QUEUED
60 00 00 00 f9 74 40 00 42d+20:47:32.264 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Later edit 1:
I have checked also the iDRAC on the PowerEdge server and in the Storage menu > Summary > Recently Logged Storage Events I have found the event coresponding to the occurance of the 2 SMART errors.
The Event States: "A disk media error on Disk 1 in Backplane 1 of RAID Controller in Slot 1 was corrected during recovery". Please find below the screenshot.
Image from iDRAC > Storage menu > Summary > Recently Logged Storage Events
Later edit 2:
After a few days, Current_Pending_Sector increased to 1 for a few hours and then decreased to 0.
Reallocated_Sector_Ct, Reallocated_Event_Count and Offline_Uncorrectable remained 0 all the time.
Also another error appeared in SMART error log: "Error: UNC at LBA".
Although, no other errors appeared in the iDRAC.
We decided to change the drive with a new one, because we no longer trusted this drive.
Thank you!
Best Answer
The two logged errors state that your HDD failed read/writing a specific LBA. However, no
Reallocated_Sector_Ct / Reallocated_Event_Count / Current_Pending_Sectorwhere logged, which seems to indicate no problem on platter side.However, this does not means that you can discard the errors as software-induced ones: after all some LBA were not read/written properly, so you have a real issue. When such kind of errors pop-up without a corresponding bad sector, they can often be caused by:
On a proper PowerEdge server, you should not have cable problems (ie: you are using a SATA backplan). While occasional issues can occur, they are quite rare.
On the other side, you have a non-zero
G-Sense_Error_Rate, so maybe the failed read/write is related to a strong vibration of the server/disk.I would monitor both SATA and
dmesglogs to be sure that the issue does not recur. If you see it again, please take note of the affected LBA range and compare it with the one above (LBA = 0x0074fb0e = 7666446): if it continue to be the same, you have a genuinely bad sector which for some reason is not reported by SMART data (a relatively common behavior on consumer disks, albeit surprising for an enterprise HDD).UPDATE: based on your iDRAC logs, it seems that a Media Error was corrected using the value stored in the other mirror leg by a Patrol Read. This really seems a genuine bad block; however, the fact that the relative SMART counters where not increased is puzzling. In the past I saw a disk which reallocated sectors only when a read/write error was reported twice for the affected sector, but it would be strange for a Toshiba enterprise disk.
Another possible explanation is that something (maybe a strong vibration) caused a bad/unaligned/torn write which made a sector unreadable. However, as that sector was not really damaged, it was overwritten successfully by the Patrol Read without any reallocation happening.
Finally, it can be a genuine case of bitrot: the written data did non match the HDD internal ECC checksum. In these cases, HDDs are designed to return a read error; however, this does not explain the write error reported above.
Anyway, occasional media correction are to be expected. However, as stated above, I would monitor the situation and substitute the disk if such reports become more frequent.