High Physical Disk Queue on Exchange 2003 mail store

exchange-2003

I have a raid 10 array with 10 7200 SATA rpm disks. My disk queue length during business hours is averaging around 100. The following is true of the setup:

  1. The array has one mail Store with 95 active mailboxes. (This is the only thing, no logs or system files)
  2. Avg Mailbox size of ~ 400 MegaBytes
  3. The array is one large 1.3 TB partition that was aligned to the raid stripe
  4. The Mail store is ~ 48 GB ( for both etm and stm files )
  5. The mail store was just defragged
  6. The transaction logs are on another array that has a less than 1 avg disk queue

Does this seem high? If so, does anything seem wrong with this setup? Should I look at some other counters?

Updates after comments:

  1. The array itself seems okay, just the other weeks it got good results from several days of jetstress testing
  2. There is no pagefile on the array 🙂
  3. There is symantec AV software. The Highest two of IO Reads and IO Writes from looking at Task Manager are conduit.exe ( symantec anti spam / virus ) and store.exe. conduit is at 18 million reads and 25 million writes, store is 144 million reads and 9 million writes. As of now, since I have gateway servers, so I am looking into taking AV off of the backend server.

Best Answer

That's beyond "rather high"-- that's exceedingly, mind-blowingly high. I have boxes with over double that number of mailboxes running on RAID-5 on old-as-dirt 7,200RPM Ultra160 SCSI drives with much lower disk queues.

Something else besides Exchange is thrashing your disks. I'd throw open Perfmon and graph the "IO Data Operations / sec" in the "Process" object for each individual process and see what process is causing so much IO.

Edit:

The article you linked in your comment to Jim B has some very good perfmon counters to have a look at. I'm wondering, too, if you've put a virtual memory pagefile onto those disks and are seeing excessive paging.

I do have some suspicions, after reading the article and linked articles re: Entourage that you might be having some issues associted with those clients. Outlook Anywhere (aka RPC over HTTP) isn't going to cause the same problems that Entourage will, though-- that's a different thing entirely (MAPI over HTTP, versus WebDAV which the Entourage clients use).

It goes w/o asking, but are you seeing anything odd in the event logs?

Edit after your update:

The total number of reads / writes isn't really what you're looking for. You're really looking for the delta in reads / writes on a per-interval basis. Throw open Perfmon, clear the default counters, and add some counters for:

  • Object: Process - Counter: Data Operations/sec - Instance: conduit.exe
  • Object: Process - Counter: Data Operations/sec - Instance: store.exe

You might also have a look at the Microsoft Exchange User Monitor (nice little article about its usage available at http://www.msexchange.org/tutorials/Microsoft-Exchange-Server-User-Monitor.html). This won't show WebDAV sessions, but it might give you some insight into what your traditional MAPI-based users are doing.

Related Topic