SATA hdd errors

hard drivesata

I have one server with WDC WD3202ABYS … There are 100 virtual hosts. Server is working about 5 years and in this time period I have change 4 disks. All with same reason: sata error. The last one:

ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:60:57:7b:b6/00:01:06:00:00/e0 tag 0 dma 180224 out
         res 51/10:60:57:7b:b6/10:01:06:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found

Descriptor sense data with sense descriptors (in hex):
        72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        06 b6 7b 57 
end_request: I/O error, dev sdb, sector 112622423
Buffer I/O error on device dm-8, logical block 14077747
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077748
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077749
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077750
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077751
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077756
lost page write due to I/O error on dm-8
ata2: EH complete
SCSI device sdb: 625142448 512-byte hdwr sectors (320073 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:90:17:30:b7/00:02:08:00:00/e0 tag 0 dma 335872 out
         res 51/10:90:17:30:b7/10:02:08:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found

Descriptor sense data with sense descriptors (in hex):
        72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        08 b7 30 17 
end_request: I/O error, dev sdb, sector 146223127
printk: 34 messages suppressed.
Buffer I/O error on device dm-8, logical block 18277835

Looks like some software error…

but in short time after that (maybe when I started fsck) following error:

EXT3-fs error (device dm-8): ext3_put_super: Couldn't clean up the journal
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/01:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x1 (device error)
ata2.00: status: { DRDY ERR }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)

Is it possible that this errors are also "software" … I mean this HDD is only 9000 hours old … where are no extra load on HDD… temperature is 29 Celsius… Do I need to replace hdd? or check disk is enough? 

EXT3-fs error (device dm-8): ext3_put_super: Couldn't clean up the journal
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/01:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x1 (device error)
ata2.00: status: { DRDY ERR }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)

How to find out reason?

Here are errors from smart:

Error 36 occurred at disk power-on lifetime: 9160 hours (381 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 22 09 80 e3  Error: UNC at LBA = 0x03800922 = 58722594

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 1f 09 80 03 0a  47d+13:38:13.534  READ DMA
  ec 00 00 00 00 00 00 0a  47d+13:38:13.530  IDENTIFY DEVICE
  ef 03 46 00 00 00 00 0a  47d+13:38:13.528  SET FEATURES [Set transfer mode]

Ok. Is it possible the following scenario:
1. Disk was on 9000 without fsck.
2. There are some errors
3. In dmesg there started errors like:

ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:60:57:7b:b6/00:01:06:00:00/e0 tag 0 dma 180224 out
         res 51/10:60:57:7b:b6/10:01:06:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found
  1. And errors like inode error and so on…
  2. I tried to umount this parition, and error come from hdd like it cannot find such inode and so on… ?

If so.. I do not understand. Do I need to change disk every year? Just to prevent this error? Does some one have same problem? Not only with one disk…

Best Answer

In my experience the errors you're seeing are actually hardware errors reflected in software. The 'lost page write due to I/O error' message is one I've seen with bad hard-drives, and it behaves similar to how you describe when attempting to fsck it. This is almost definitely a true hardware fault.

You should check the output of smartctl to see what it says could be problem. 

smartctl --attributes /dev/sdb

It'll give you output similar to this:

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED     RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   212   186   021    Pre-fail  Always       -       4358
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       97
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   200   200   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   066   066   000    Old_age   Always       -       25420
 10 Spin_Retry_Count        0x0013   100   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0013   100   253   051    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       86
194 Temperature_Celsius     0x0022   104   001   000    Old_age   Always       -       46
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      -       0

The output can be arcane, but the one I'd pay close attention to would be Reallocated_Sector_Ct, since that tells you what the HD has for known bad sectors. The command 'smartctl -a' will give a lot more data. On the bad HD I had a while back, the bottom of that output is the 'SMART Error Log' which had a few entries.

Related Topic