Manage HDD Behind Dell PERC H330 Controller

Is it possible to mix SATA and SAS drives on the same controller?

Yes.

These are intended to be used these in server 2012 R2 software raid?

Is that a question? ;-)

Obviously not mixed in the same array but maybe on the same controller, depending on how many disk controllers the server comes with.

SATA drives on a SAS controller / SAS hostadapter / SAS based RAID card should work just fine. You probably want to avoid port multipliers though because if a SATA drives dies then that might lock up the multiplier. If you just use direct connections then mixing them should be fine.

My original intention was a layout like this: 2 x SATA 250GB mirrored - for Hyper-V partition 4 x SAS software Raid 1+0 - for virtual machines and all data 1 x SATA - for backup purposes

Backups are good. But I recommend also storing backups off-line or at least elsewhere. This to prevent loss of both originals and backups in the case of fire, flooding, lightning strike, theft, ...

Problem is now it comes to ordering the hardware it seems there is some question as to whether it can support SATA and SAS at the same time.

It can. I am writing the answer from a desktop with a 3ware 9750 RAID card with both SAS and SATA drives attached to the same card.

FYI: Link to a similar question over on [SU], which was answered by a regular on [SF]. The same answer is true for software RAID. (In fact, if you are going to use SW based RAID and have a HW RAID card, then reflash the RAID card to a target/initiator mode).

+1 to all of Evan Anderson's answer (though I'm not familiar with opROM execution problems).

You may also want to investigate the specifications of your server to find how many PCIe cards of each type are supported, and whether there is any "slot priority" that you should be following (these factors exist even with Dell servers). Just because you have a PCIe slot for the card doesn't mean it's getting the necessary power and bandwidth to function properly.

As a work-around (if you can't figure out how to get the PERC card working) you may want to consider using a SAS HBA (e.g. Dell's SAS 6/E card). This would be presenting each individual hard drive directly to the OS though, which would require something akin to software RAID (like Storage Spaces in Server 2012, which is actually validated for use with that type of setup).

Edit 2014-09-11

Any advice on where to obtain the H800 drivers for Windows 2012? (Or equivalent LSI drivers?) I can only find Windows 2008 R2 64-bit PERC drivers so far.

The Server 2012 drivers for the PERC H800 card are provided by Microsoft via Windows Update. If you can get far enough to see the card listed in device manager, "Update Driver Software..." should handle the rest almost flawlessly.

I've looked in the BIOS previously regarding PCI-e settings, and there was no signs of slot priority that I recall. Also, this is the first expansion card we've added to the server - so regardless of any slot priority, I'd think it would allow at least one card without modification.

Slot priority (if any) is typically determined by the server manufacturer, and is not normally a configurable option in the BIOS settings. You should reference the documentation of the server (e.g. an installation, administration, or user guide) for details on which slots should be used first for specific types of cards. It could be that the slot you're using won't support a single external-storage card alone, even if no others are present.

Try testing in alternate slots with the PERC card. If that still doesn't work for you, look into purchasing a SAS6/E HBA or the LSI 9207-8e - I believe that either of these are supported for the MD12xx enclosures for use with Storage Spaces (I've tested both in lab with no problems, but cannot find supporting documentation for the MD12xx series enclosures at the moment).