Ubuntu – IOmeter IOPS on Win2k3,Win2008, and Ubuntu 10.04

iopsUbuntuwindows-server-2003windows-server-2008

Summary: Given the same hardware, Ubuntu outperforms Win2k3 by about 50 percent when it comes to IOPS. Win2008R2 initially performs the same as Win2k3, but it gradually gains performance until it is on par with Ubuntu (takes about 20-30 seconds of continuous activity).

Questions: Can anyone explain the Win2008R2 behavior?

Details: I'll only give details I think are relevant, but hit me up for more if I miss something. We've got a SAN with an 11x600GB SAS RAID5 group. Everything is connected by 4Gbps fiber. I created a 500GB LUN and shared it with a physical server running Win2k3Entx64. I created another 500GB LUN and shared it with a server running ESXi. I created a Win2k3Entx64 VM, an Ubuntu 10.04 x64, and a Win2008R2x64 VM in ESXi.
The physical Win2k3 server outperformed the Win2k3 VM by a small amount. The Ubuntu VM smoked both Win2k3 servers. The Win2008 VM acted like Win2k3 for about 5-10 seconds then started inching up performance until it was about the same as the Ubuntu Server.

IOmeter Setup and Results:
32k 50% read 0% random 1 Worker on a 1.5GB test file.

Win2k3x64 (Physical): Avg I/O response (ms) – 1.0789
Total I/O per second – 926.28

Win2k3x64 (VM) : Avg I/O response (ms) – 1.1786
Total I/O per second – 847.81

Ubuntu (VM) : Avg I/O response (ms) – .7849
Total I/O per second – 1273.00

Win2008R2(VM)Intial : Avg I/O response (ms) – 1.0959
Total I/O per second – 910.00

Win2008R2(VM)30 Sec.: Avg I/O response (ms) – .8810
Total I/O per second – 1133.66

You may be saying, switch to Linux! Half of our applications rely on MS-SQL and Win2k3 so it is not an option. Switching to 2008 server may be an option, but not before I figure out why I'm getting these results.

PART 2

Alignment was definately the problem. It turns out the 2008 and Ubuntu are automatically aligned. Now for problem 2. For one IOmeter worker process, it is fine. However, the performance on the VM gets worse compared to the physical for every worker I add. I've added up to 8 (1 per CPU) and again I'm down by 50 percent compared to the physical server.
I have tried 4k,16k,and 64k disk allocations on Win2k3. The SCSI drivers are LSI Logic PCI-X Ultra 320 SCSI host adapter (version 5.2.3790.3959). The CPU usage is about the same as the physical server so its not a cpu resource issue.

You guys solved the first problem, hopefully you have some advice for this too. Thank you

Best Answer

Looks to me that you haven't aligned the test volume on Win2K3. By default Win2K3 doesn't align partitions so the MBR causes an offset that results in a penalty on writes that cross stripe boundaries. Win2K8 automatically aligns on 1Meg which generally matches most RAID stripe boundaries. Recent Ubuntu builds also automatically align partitions starting at a 1Meg offset.

With your 32K IO size you are likely to be hitting a lot of RAID stripe boundaries. The 50% penalty that you're seeing is higher than anything I've seen with this but the exact penalty depends on your RAID controller's stripe size.

I've no good explanation for the W2K8 ramp up behaviour.

Related Solutions

Ubuntu – Horrible PHP/MySQL Query Performance On Ubuntu Server 10.04

Sounds like your MySQL config needs tuning. A base MySQL config on any distro will be on the conservative side, assuming they're not the only service running on the server. You should spend some time and tune it. If your tables are in InnoDB format, make sure you tune the innodb_buffer_pool_size variable to approximately 80% of your system memory (assuming it's the only service to be run).

That's likely the first place to look when comparing the OSes. Each distro has their own standards. You should match the configs if you want to run a fair test between them.

Hope this helps!

Linux – How to Reduce Number of Sockets in TIME_WAIT

One thing you should do to start is to fix the net.ipv4.tcp_fin_timeout=1. That is way to low, you should probably not take that much lower than 30.

Since this is behind nginx. Does that mean nginx is acting as a reverse proxy? If that is the case then your connections are 2x (one to client, one to your web servers). Do you know which end these sockets belong to?

Update:
fin_timeout is how long they stay in FIN-WAIT-2 (From networking/ip-sysctl.txt in the kernel documentation):

tcp_fin_timeout - INTEGER
        Time to hold socket in state FIN-WAIT-2, if it was closed
        by our side. Peer can be broken and never close its side,
        or even died unexpectedly. Default value is 60sec.
        Usual value used in 2.2 was 180 seconds, you may restore
        it, but remember that if your machine is even underloaded WEB server,
        you risk to overflow memory with kilotons of dead sockets,
        FIN-WAIT-2 sockets are less dangerous than FIN-WAIT-1,
        because they eat maximum 1.5K of memory, but they tend
        to live longer. Cf. tcp_max_orphans.

I think you maybe just have to let Linux keep the TIME_WAIT socket number up against what looks like maybe 32k cap on them and this is where Linux recycles them. This 32k is alluded to in this link:

Also, I find the /proc/sys/net/ipv4/tcp_max_tw_buckets confusing. Although the default is set at 180000, I see a TCP disruption when I have 32K TIME_WAIT sockets on my system, regardless of the max tw buckets.

This link also suggests that the TIME_WAIT state is 60 seconds and can not be tuned via proc.

Random fun fact:
You can see the timers on the timewait with netstat for each socket with netstat -on | grep TIME_WAIT | less

Reuse Vs Recycle:
These are kind of interesting, it reads like reuse enable the reuse of time_Wait sockets, and recycle puts it into TURBO mode:

tcp_tw_recycle - BOOLEAN
        Enable fast recycling TIME-WAIT sockets. Default value is 0.
        It should not be changed without advice/request of technical
        experts.

tcp_tw_reuse - BOOLEAN
        Allow to reuse TIME-WAIT sockets for new connections when it is
        safe from protocol viewpoint. Default value is 0.
        It should not be changed without advice/request of technical
        experts.

I wouldn't recommend using net.ipv4.tcp_tw_recycle as it causes problems with NAT clients.

Maybe you might try not having both of those switched on and see what effect it has (Try one at a time and see how they work on their own)? I would use netstat -n | grep TIME_WAIT | wc -l for faster feedback than Munin.

Best Answer

Related Solutions

Ubuntu – Horrible PHP/MySQL Query Performance On Ubuntu Server 10.04

Linux – How to Reduce Number of Sockets in TIME_WAIT

Related Topic