CentOS, Xeon 1230, 16 Gb RAM, 2x1TB SSD in raid1 (mdadm).
Read from raid1 vary from 130 to 250 MB/s depends of files etc.
But write speed very slow: 15-20 MB/s (iotop, mc etc.)! You know the real write speed to SSD. CPU load not more than 23-25% (only one core). So situation isn't normal.
What should I do to reach hiest speed?
I try increase speed_limit_min:
# sysctl -w dev.raid.speed_limit_min=20000
but without result.
What should I do in order to increase write speed?
Update 1:
Raid details
# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Mon Dec 7 14:23:49 2015
Raid Level : raid1
Array Size : 999547904 (953.24 GiB 1023.54 GB)
Used Dev Size : 999547904 (953.24 GiB 1023.54 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Feb 12 13:26:17 2017
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : localhost:pv00
UUID : 0fe1355e:edd9d104:e2db49ec:bc3bb9b6
Events : 7865
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 33 1 active sync /dev/sdc1
Parted output for sda:
# parted /dev/sda print
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sda: 1024GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 538MB 537MB primary ext4 boot
2 538MB 1024GB 1024GB primary raid
Parted output for sdc
# parted /dev/sdc print
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sdc: 1024GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 1024GB 1024GB primary raid
mdstat file contents:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[1] sdb1[0]
976630464 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
md127 : active raid1 sdc1[1] sda2[0]
999547904 blocks super 1.2 [2/2] [UU]
bitmap: 3/8 pages [12KB], 65536KB chunk
It is impossible…
dmesg | grep -i ahci
return nothing… the end…
Can it lead to my problem?
Update 1:
Turn on (bios, initramfs). No affect. Copying 20 Gbytes: firstly 130-160 MB/s and liearly slowing down to 17 MB/s after minute or two.
Update 2:
# hdparm -I /dev/sda
/dev/sda:
ATA device, with non-removable media
Model Number: Samsung SSD 850 PRO 1TB
Serial Number: S252NXAG905316P
Firmware Revision: EXM02B6Q
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Used: unknown (minor revision code 0x0039)
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 2000409264
Logical Sector size: 512 bytes
Physical Sector size: 512 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 976762 MBytes
device size with M = 1000*1000: 1024209 MBytes (1024 GB)
cache/buffer size = unknown
Nominal Media Rotation Rate: Solid State Device
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 1 Current = 1
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
Write-Read-Verify feature set
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Phy event counters
* unknown 76[15]
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Asynchronous notification (eg. media change)
* Software settings preservation
unknown 78[8]
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
* reserved 69[4]
* DOWNLOAD MICROCODE DMA command
* SET MAX SETPASSWORD/UNLOCK DMA commands
* WRITE BUFFER DMA command
* READ BUFFER DMA command
* Data Set Management TRIM supported (limit 8 blocks)
Security:
Master password revision code = 65534
supported
not enabled
not locked
frozen
not expired: security count
supported: enhanced erase
2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 500253884009b0f0
NAA : 5
IEEE OUI : 002538
Unique ID : 84009b0f0
Checksum: correct
# hdparm -I /dev/sdc
/dev/sdc:
ATA device, with non-removable media
Model Number: WDC WD1004FBYZ-01YCBB1
Serial Number: WD-WMC6M0D492LN
Firmware Revision: RR04
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Used: unknown (minor revision code 0x006d)
Supported: 10 9 8 7 6 5
Likely used: 10
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 1953525168
Logical Sector size: 512 bytes
Physical Sector size: 512 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 953869 MBytes
device size with M = 1000*1000: 1000204 MBytes (1000 GB)
cache/buffer size = unknown
Form Factor: 3.5 inch
Nominal Media Rotation Rate: 7200
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: 96
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
* Advanced Power Management feature set
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
* 48-bit Address feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
* IDLE_IMMEDIATE with UNLOAD
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
unknown 119[6]
unknown 119[7]
unknown 119[8]
unknown 119[9]
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Phy event counters
* Idle-Unload when NCQ is active
* NCQ priority information
* unknown 76[14]
* unknown 76[15]
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
unknown 78[7]
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
* reserved 69[3]
* DOWNLOAD MICROCODE DMA command
* Long physical sector diagnostics
Security:
Master password revision code = 65534
supported
not enabled
not locked
frozen
not expired: security count
supported: enhanced erase
118min for SECURITY ERASE UNIT. 118min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee0aef2a46c
NAA : 5
IEEE OUI : 0014ee
Unique ID : 0aef2a46c
Checksum: correct
Update 3: smartctl
# smartctl -a /dev/sda SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 15006 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 34 177 Wear_Leveling_Count 0x0013 099 099 000 Pre-fail Always - 18 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0032 067 054 000 Old_age Always - 33 195 Hardware_ECC_Recovered 0x001a 200 200 000 Old_age Always - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 235 Unknown_Attribute 0x0012 099 099 000 Old_age Always - 12 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 22922238691
Update 4:
Think I've found. Option "discard" in /etc/fstab. It increase write speed on testing server up to 70-80 MB/s, on target server I can't check it right now.
Thank you all, I think problem is solved.
Best Answer
This variable you are referring to is related to RAID rebuild speed. If your array is in normal mode (not rebuilding), it should not affect the system performance.
I can notice slow HD (or too busy one) by looking at running processes. You can look for processes that you know are accessing disk with
Dstate (uninterruptible sleep). If you got some, this may indicate a slow disk. Otherwise, your processes may not be doing IO at expected rate.