Smart long test – what is the performance impact on busy server

hard driveraidraid1smartsmartctl

I have a busy server with a RAID 1 setup. The application (runs in PHP) is very read/write intensive to the database (MariaDB).

A cronjob runs smartctl short test every day and checks the output of smartctl -H and mdadm -D.

I would like to run a long test sometimes, but I am concerned about its performance impact. I read that it can take several hours to complete. If it causes server performance to degrade while it runs, my users will be affected for 5+ hours.

So, a few questions here:

1) Do long smart tests usually impact performance that can be significant to users?

2) Since I have RAID 1 and do short tests, are long tests still necessary?

3) Is there a way to stop a long test if I find it is causing trouble on server performance?

Best Answer

It depends. (muhahaha) On what? How much your application uses the disk and how much caching can your application take advantage of. There's no magic here - if the drive is being tested, it cannot serve at the same maximum speed and low latency as when it is not. However, if your minimum latency requirements are less than the impact of the tests, then it is a wash in terms of application impact.
Probably not. Big Enterprise storage companies (EMC, IBM, NetApp, etc.) replace drives based on their M(ean) T(ime) B(etween) F(ailures), not merely waiting for the drive's firmware to report a problem. The short tests test everything that is likely to fail first. The long tests do the same tests as the short tests, except they do not have a time limit. Just assume that all drives will fail, but their likelyhood is higher after the warranty expires.
Yes, PROVIDED that the drive supports aborting (or "suspending") offline collection.

From the 'smartctl(8)' manual page:

       -X, --abort
              Aborts  non-captive  SMART  Self  Tests.  Note that this command
              will abort the Offline Immediate Test routine only if your  disk
              has the "Abort Offline collection upon new command" capability.

(I think you can also abort if you see "Suspend Offline collection upon new command"; I think the man page needs to be updated.) You can check for that capability via:

smartctl -x <device>

Related Solutions

Linux – 1TB Seagate Barracuda ST31000528AS – Unexpected “Offline uncorrectable sectors”

Yes, it definitely looks like a hardware fault.
I have had multiple Barracuda ES / ES.2 drives go..including 12 out of 16 across two servers.
I don't think the smartmontools is sufficient for warranty replacement. Go to Seagate's website and download their SeaTools bootable CD. When/if it determines the drive has a fault, it will give you a warranty code.

To me, it looks like you have a dying drive, with sectors that can't be re-mapped. Ditch the drive, get a new one, rebuild the RAID array. The drives are so cheap it's really not worth thinking about. Especially if you have any attachment to any of this data.

edit: Additionally, SeaTools will give you a list of the bad sectors that it finds. Also, output of smartctl -a /dev/sdb would be helpful.

edit2: From looking at your smartctl -a output...

You have 2 "Offline Unrecoverable Sectors" --but what does this mean?

An offline uncorrectable sector is a disk sector which was not readable during
an off-line scan or a self-test. This is important to know, because if you have
data stored in this disk sector, and you need to read it, the read will fail.
Please see the previous '-C' option for more details.

citation: http://smartmontools.sourceforge.net/man/smartd.8.html

There are also 2 "Current Pending Sectors" --and what are they?

A pending sector is a disk sector (containing 512 bytes of your data) which the
device would like to mark as ``bad" and reallocate. Typically this is because
your computer tried to read that sector, and the read failed because the data on
it has been corrupted and has inconsistent Error Checking and Correction (ECC)
codes. This is important to know, because it means that there is some unreadable
data on the disk. The problem of figuring out what file this data belongs to is
operating system and file system specific. You can typically force the sector to
reallocate by writing to it (translation: make the device substitute a spare
good sector for the bad one) but at the price of losing the 512 bytes of data
stored there.

So, your disk was unable to read from two sectors, could not, and now wants to mark them as faulty so they can be reallocated.

Personally, I tend to toss drives (especially in RAID arrays) once they start having bad sectors. But, I also don't pay for them...

Alternatively, maybe you could break the mirror, remove that drive, and give it a scan with SeaTools? If it is able to remap the sectors, it will. It shouldn't matter if/when it destroys, because the mirror will be rebuilt when you re-insert the disk and rebuild the array. However, I've not done this, and would wait for someone to chime in.

Inexplicably slow Western Digital WD15EARS disk

Possibly the cause for significantly decreased performance would be the Automatic Acoustic Management (AAM) - check its status on both drives using hdparm -M and disable it entirely by setting the value to 254.

Also, enabling the write cache on the drives using hdparm -W should be worth trying.

Since you are using Caviar Green drives (which you should not be doing for a Linux RAID setup, honestly) and already are at it, make sure to disable the power saving features, especially the IntelliPark feature.

If nothing helps, it might be a hardware problem indeed - open up an RMA and return the drive to WD.

Best Answer

Related Solutions

Linux – 1TB Seagate Barracuda ST31000528AS – Unexpected “Offline uncorrectable sectors”

Inexplicably slow Western Digital WD15EARS disk

Related Topic