LSI 9260-8i limited to max 200 MB/s (independently from the drives)

bandwidthlsi

LSI MegaRAID 9260-8i controller is limited to max 200 MB/s transfer rates.
The server is a HP DL180 G6, with CentOS 7 (64 bits) and we are testing 4TB SAS drives (Model: WD4001FYYG).
The controller is using iBBU08 (512 cache).
We have tested enabling/disabling cache and direct I/O, but it doesn't solve the problem.

According to our tests, writing concurrently to two different virtual disks (a RAID10 drive of 6 disks and a RAID0 drive of a single disk) we get max. 200 MB/s when reading and max. 200 MB/s when writing.

We verified that the performance decreases when operating concurrently on a different drive because the bandwith (aprox. 200 MB/s) is shared among different independent operating disk drives operations (bottleneck).

Conclusion:

The LSI controller is limiting the bandwidth to max 200 MB/s.

Why is this happening?
How can we fix it?
May it be related with the PCI card?
Can we measure the transfer rate?

PS: Issue was filed in support ticket SR # P00117431, but we stopped getting answers from AVAGOTECH (ex- LSI) after sending them detailed info.

Thanks

This are our IO tests:

— 1) Single drive IO test —

Write test:

# sync
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync

1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 46.7041 s, 184 MB/s

Read test:

# sync
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync

1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 47.1691 s, 182 MB/s

— 2) Two drives concurrent IO tests —

We will repeat the previous test, but running the same IO operations on a second independent drive at the same time.
As a result, the same drive now only performs 50%, which proves that the IO's on the second drive (/mnt/sdb/test) are sharing some limited resources on the LSI controller.

Write test:

Process 1:

[root@hp ~]# sync
[root@hp ~]# echo 3 > /proc/sys/vm/drop_caches
[root@hp ~]# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 87.8613 s, 97.8 MB/s

Process 2:

[root@hp ~]# dd if=/dev/zero of=/mnt/sdb/test bs=8k count=1M conv=fsync
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 86.3504 s, 99.5 MB/s

Read test:

Process 1:

[root@hp ~]# dd if=/tmp/test of=/dev/null bs=8k count=1M
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 81.5574 s, 105 MB/s

Process 2:

[root@hp ~]# dd if=/mnt/sdb/test of=/dev/null bs=8k count=1M
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 84.2258 s, 102 MB/s

Best Answer

This can easily happen because of the 4k sectors on the drive are 512e (you didn't specify the drive model, so it's a bit of the wild guess, but considering it's the 4Tb drive, I'd say it's Advanced Format). So I would check if your OS is aware about the size of your sector, unless you want read-modify-write cycling. This means proper partition alignment and the block size of the filesystems you are using.

And yeah, there's no such thing as HP DL 180 G7, gen6 was the last, then the model index changed from 180.

Just in case, there's pretty decent article for you (yeah, you are using CentOS, but... you know, it's basically the same stuff when it comes to internals).

Another thing you should probaby check and enable - it's the controller write cache, if you have a BBU, of course.

Related Topic