Revisiting an older vSphere setup with two Dell 1950s, 4 NICs per host:
esxi1
and esxi2
for simplicity sake.
SAN is a Dell MD3000i, two controllers, two NICs per controller:
rc00
and rc01
; rc10
and rc11
Only one LUN 0/virtual disk configured right now; RAID 10, 300GB SAS 15K, 6 spindles. Controllers/channels are as follows:
rc00
: 192.168.130.101/24
rc01
: 192.168.131.101/24
rc10
: 192.168.130.102/24
rc11
: 192.168.131.102/24
Switches (sw-1
and sw-2
) are Dell PowerConnect 5424s; iSCSI "optimization" (QoS) is not enabled as there's no other traffic on the two switches. Jumbo Frames enabled, 9000 MTU, Flow Control on, MDIX auto.
Wanted to do some benchmarking while this setup is empty and I have some time on my hands.
Couldn't quite remember how to setup multipathing as it has been a while so, Googling around and reading a couple of older 4.1 white papers from Dell and vmware, I'm seeing really two ways of doing it:
one vSwitch with multiple VMKernel Ports and physical NICs:
rc00:192.168.130.101
—sw-1
—-esxi1:vSwitch1:vmk1:eth1:192.168.130.11
rc01:192.168.131.101
—sw-2
—-esxi1:vSwitch1:vmk2:eth2:192.168.131.11
… or two vSwitches with one VMKernel Port and one physical NIC:
rc00:192.168.130.101
—sw-1
—-esxi1:vSwitch1:vmk1:eth1:192.168.130.11
rc01:192.168.131.101
—sw-2
—-esxi1:vSwitch2:vmk1:eth2:192.168.131.11
Question #1: Are there any practical differences in performance or a reason to choose one over the other? Everything else look ok?
Question #2: I've actually staggered the VMKernel Ports across NIC controllers so that one of the VMKernel Port/physical NICs (eth1) is bound to one of the built-in Broadcom NICs, and the other (eth2) is bound to one of the Intel NICs.
I figured if one of the NICs/NIC controllers goes south, then there's still a path available through the second NIC/NIC controller. Wondering though if this will causing multipathing performance issues or general flakyness; didn't see anything out there to indicate one way or the other.
Perhaps I'm anticipating a failure that will likely never fail that "nicely" (i.e. if there's a NIC failure, chances are the host will just freak out anyways).
NOTE: the "one vSwitch, multiple VMKernel Ports" method actually seems to freak out the ESXi host. Takes an abnormally long time to reboot, some times the paths/LUNs are not showing active/active I/O or showing up at all, requiring a Rescan and/or and up/down of the VMKernel to get it to see the LUNs again. It looks odd for a configuration anyways as you're putting two different subnets on the same vSwitch/broadcast domain, and I believe that the vSwitches function as a layer 2 switch.
Benchmark #1: doesn't this seem kind of terrible?
Running ubuntu 10.04.2 LTS with "typical" settings (1 vCPU, 1024 MB RAM, 8 GB disk, defaults for filesystem, ext4 with LVM) and bonnie++
:
gravyface@testubu:~$ bonnie++ -f -d /tmp
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testubu 2G 96131 57 33783 16 98930 17 444.6 13
Latency 623ms 645ms 111ms 503ms
Version 1.96 ------Sequential Create------ --------Random Create--------
testubu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 16509 79 +++++ +++ 25608 88 19044 86 +++++ +++ 25079 86
Latency 10289us 1398us 8288us 509us 442us 12159us
Take 2:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testubu 2G 97240 54 32974 17 93371 17 420.6 14
Latency 291ms 1421ms 1266ms 616ms
Version 1.96 ------Sequential Create------ --------Random Create--------
testubu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14410 71 +++++ +++ 22082 86 18109 88 +++++ +++ 22054 88
Latency 108ms 1324us 2400us 814us 88us 4835us
1.96,1.96,testubu,1,1336168050,2G,,,,97240,54,32974,17,,,93371,17,420.6,14,16,,,,,14410,71, +++++,+++,22082,86,18109,88,+++++,+++,22054,88,,291ms,1421ms,,1266ms,616ms,108ms,1324us,2400us,814us,88us,4835us
Take 3: with --iops=3
set from esxcli
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testubu 2G 115663 61 35594 18 103602 21 440.0 17
Latency 285ms 571ms 52049us 477ms
Version 1.96 ------Sequential Create------ --------Random Create--------
testubu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14206 73 +++++ +++ 22753 90 18424 91 +++++ +++ 22367 90
Latency 108ms 1951us 1827us 6200us 326us 6127us
1.96,1.96,testubu,1,1336168752,2G,,,,115663,61,35594,18,,,103602,21,440.0,17,16,,,,,14206,73,+++++,+++,22753,90,18424,91,+++++,+++,22367,90,,285ms,571ms,,52049us,477ms,108ms,1951us,1827us,6200us,326us,6127us
Best Answer
Q1: One vSwitch per vmkernel port is the usual way of doing it, but I'm not sure if anything gets jerky if you do it any other way. vSphere 5 has a pretty strict compliance test that you must pass in order to bind a adapter to the iSCSI initiator, and it could fail if you're using a single vSwitch. But these are just my thoughts, not actual facts :)
Q2: I also use different NIC's for each vmkernel, as I've seen NIC's go down before.. you really do not want to loose all connectivity against your storage.. but then again, the likelyhood for something like that to happen isn't exactly big. It's also quite common in FC enviroments to use dual single-port HBA's instead of single dual-port HBA's. Better safe than sorry?
Either way - you should not experience any performance issues since all modern NIC's have offloading built in. I would actually guess that you get better performance with dual NIC's, as you get different interrupts and a separate PCIe lane..