I've found that when I've had to tune for lower latency vs throughput, I've tuned nr_requests down from it's default (to as low as 32). The idea being smaller batches equals lower latency.
Also for read_ahead_kb I've found that for sequential reads/writes, increasing this value offers better throughput, but I've found that this option really depends on your workload and IO pattern. For example on a database system that I've recently tuned I changed this value to match a single db page size which helped to reduce read latency. Increasing or decreasing beyond this value proved to hurt performance in my case.
As for other options or settings for block device queues:
max_sectors_kb = I've set this value to match what the hardware allows for a single transfer (check the value of the max_hw_sectors_kb (RO) file in sysfs to see what's allowed)
nomerges = this lets you disable or adjust lookup logic for merging io requests. (turning this off can save you some cpu cycles, but I haven't seen any benefit when changing this for my systems, so I left it default)
rq_affinity = I haven't tried this yet, but here is the explanation behind it from the kernel docs
If this option is '1', the block layer will migrate request completions to the
cpu "group" that originally submitted the request. For some workloads this
provides a significant reduction in CPU cycles due to caching effects.
For storage configurations that need to maximize distribution of completion
processing setting this option to '2' forces the completion to run on the
requesting cpu (bypassing the "group" aggregation logic)"
scheduler = you said that you tried deadline and noop. I've tested both noop and deadline, but have found deadline win's out for the testing I've done most recently for a database server.
NOOP performed well, but for our database server I was still able to achieve better performance adjusting the deadline scheduler.
Options for deadline scheduler located under /sys/block/{sd,cciss,dm-}*/queue/iosched/ :
fifo_batch = kind of like nr_requests, but specific to the scheduler. Rule of thumb is tune this down for lower latency or up for throughput. Controls the batch size of read and write requests.
write_expire = sets the expire time for write batches default is 5000ms. Once again decrease this value decreases your write latency while increase the value increases throughput.
read_expire = sets the expire time for read batches default is 500ms. Same rules apply here.
front_merges = I tend to turn this off, and it's on by default. I don't see the need for the scheduler to waste cpu cycles trying to front merge IO requests.
writes_starved = since deadline is geared toward reads the default here is to process 2 read batches before a write batch is processed. I found the default of 2 to be good for my workload.
Whenever a Windows application is writing data to disk, this data is written to the host memory first.
A "normal" write request returns immediately after the data is in memory and a queue entry indicating that this data needs flushing to persistent storage is created. A mechanism called the Lazy Writer ensures that this queue is being processed periodically (1/8th of the queue is flushed by the lazy writer every second by default). This is the mechanism you are disabling by unchecking "enable write caching on the disk" - every write request would need to wait until it has been acknowledged as "written" by the storage device before it returns.
Applications with specific requirements for data integrity (databases, filesystem drivers) do have options for a more intelligent approach to caching. For writes which need immediate persistence (NTFS journal, database transaction logs) FILE_FLAG_WRITE_THROUGH can be specified with the write. In this case, the write call would not return before the data is actually committed to persistent storage. Unless you activate the "Enable advanced performance" checkbox which causes the cache manager to ignore FILE_FLAG_WRITE_THROUGH, return the call immediately and pass it to the lazy writer as every other "normal" write.
As you have two additional layers of caching1 (numer one is your host operating system running the KVM hypervisor, number two is your storage controller with a BBWC/FBWC), things are getting more complicated. Each of these layers would provide you with similar choices and as every write request has to pass through all of them, the weakest link of the chain will be effective to your data's integrity.
Application developers at large do know and understand the effects of caching and write through calls. So really critical data parts are written with FILE_FLAG_WRITE_THROUGH while everything not written with this flag can be considered as safe to be cached in volatile memory. The trouble starts when FILE_FLAG_WRITE_THROUGH is being ignored at any layer and the data is actually lost in the case of a power outage or a software failure. Such conditions usually result in corruptions in filesystems and transaction logs, leading to unpredictable results and maybe even requiring you to restore from a backup, so this obviously should be avoided. If your storage controller's cache is "battery-backed" or "flash-backed", it can be considered "non-volatile" to a certain degree, so it generally is considered safe to use its write-back cache even for write-through requests2.
The bottom line: it generally is safe to "Enable write caching on disk" unless you are dealing with broken applications which are not using FILE_FLAG_WRITE_THROUGH but need every write to be persistent. Disabling this would not hurt too much in your case as most calls should be handled by the storage controller's write cache and return nearly immediately (but you likely would have additional overhead from this though and the cache size would be limited by the controller's DRAM). You never should "Enable advanced performance" or "Turn off Windows write-cache buffer flushing on the device" on a system where you value uptime or require data integrity.
Further reading:
MSDN Libary - Windows File Caching
Smallvoid blog - description of the hard disk cache
1 actually there even is another layer of caching at the hard disk itself, but in most cases the write through request is honored no matter what the drive's cache settings are. Some flash drives are notable (read: broken) exceptions to this rule though - with flash SSDs, writes are usually cached and reported as written immediately but only committed to volatile cache - not just for performance reasons but also to coalesce writes and prolong the life time of the flash cells. The "enterprise" versions of flash SSDs usually have capacitors which would ensure the drive has enough power to flush the cache to flash cells, the "consumer" versions often don't - beware of those.
2 it obviously is not safe under all circumstances - if the battery is defective and it goes undetected, if there is a bug in the controller's logic handling the power failure case, if the power outage period exceeds the time the battery is able to provide power, if the supercaps or the flash cells of the FBWC go bust, data is going to be lost. But these occurrences are commonly rare enough to take the risk.
Best Answer
These systems are designed to just plug in and go. Here's how each tier handles I/O.
OS
Writes are cached briefly (dirty pages) in RAM while the I/O subsystem actually commits things. Once a write is committed, the page is then cached in case it is immediately read again. The OS Cache does not maintain a pool of uncomitted writes, it maintains a pool of already comitted writes that may need to be read again. It is, in effect, a 100% read cache.
RAID Controller
The BBC of the RAID controller receives the Write from the OS. Depending on the cache policy of the volume being written to (write-thru vs write-back), the RAID controller may report the write as Comitted at this time. It will then queue the write for comitting to actual disk
Disk
Some RAID cards actually do disable the HD cache. Others, don't. I don't remember how HP does theirs, but would not be surprised if the HD cache is disabled and the write-optimization logic is pushed up into the RAID controller itself; there is a reason HP uses custom firmware on their drives.
Operating systems, and the filesystems they support, know very well that sudden power-loss is a failure mode that can kill writes between the time the OS determines that it needs to happen and when the storage system reports it is done. We've been doing this a while now, and we're pretty good at defending against it.
The XFS filesystem has a bad reputation for survivability in sudden power-loss situations due to how it handles metadata writes. But then, it's intended environment is one where power is presumed to be adequately redundant. Other filesystems, the ext series, btrfs, and of course zfs, survive that just fine as well.
If you're operating in an environment with known bad power, to ensure no data loss during power outages:
And that's it. The BBC on the RAID card ensures the RAID cache is preserved until power is restored. The disk caches are likely disabled. No need to tune the RAID card cache to be all-read. No need to disable the OS block caches.
Really.