vCenter Server is basically a management tool that will allow you to manage VM's across multiple VMware hosts at once. You'll connect to vCenter Server with a vSphere client and instead of one host you'll see multiple hosts at once.
You can migrate VM's from one host to another via vMotion (although with Essentials you can't do it while the VM is running, and it requires shared storage). vCenter is especially useful for the more expensive VMware licenses, to manage things like DRS, HA, clustering and vMotion. With Essentials the use is more limited, but I also have Essentials and I've still found it to be quite useful to see my entire VM collection at once.
vCenter Server runs well in a virtual Windows box. It needs SQL Server Express or better and a 64-bit Windows OS.
vSphere's memory management is pretty decent, though the terms used often cause a lot of confusion.
In general, memory over-commit should be avoided as it creates exactly this type of problem. However, there are times when it cannot be avoided, so forewarned is forearmed!
What are the downsides of overcommitting and over-configuring resources
(specifically RAM) in vSphere environments?
The major downside of over-committing resources is that should you have contention, your hosts would be forced to balloon, swap or intelligently schedule/de-duplicate behind the scenes in order to give each VM the RAM it needs.
For ballooning, vSphere will inflate a "balloon" of RAM within a chosen VM, then give that ballooned RAM to the guest that needs it. This isn't really "bad" - VMs are stealing each other's RAM, so there's no disk swapping going on - but it could lead to mis-fired alerting and skewed metrics if these rely on analysing the VM's RAM usage, as the RAM won't be marked as "ballooned", just that it's "in use" by the OS.
The other feature that vSphere can use is Transparent Page Sharing (TPS) - which is essentially RAM de-duplication. vSphere will periodically scan all allocated RAM, looking for duplicated pages. When found, it will de-duplicate and free up the duplicated pages.
Take a look at vSphere's Memory Management whitepaper (PDF) - specifically "Memory Reclamation in ESXi" (page 8) - if you need a more in-depth explanation.
Assuming that the VMs can run in less RAM, is it fair to say that
there's overhead to configuring virtual machines with more RAM than
they need?
There's no visible overhead - you can allocate 100GB of RAM on a host with 16 GB (however, that doesn't mean you should, for the reasons above).
Total memory in use by all of your VMs is the "Active" curve shown in your graphs. Of course, you should never rely on just that figure when calculating how much you would like to overcommit, but if you have historical metrics as you have, you can analyse and work it out based on actual usage.
The difference between "Active" and "Consumed" RAM is discussed in this VMWare Community thread.
What is the counter-argument to: "if a VM has 16GB of RAM allocated,
but only uses 4GB, what's the problem??"? E.g. do customers need to be
educated?
The short answer to this is yes - customers should always be educated in best practices, regardless of the the tools at their disposal.
Customers should be educated to size their VMs according to what they use, rather than what they want. A lot of the time, people will over-specify their VMs just because they might need 16 GB of RAM, even if they're historically bumbling along on 2 GB day after day. As a vSphere administrator, you have the knowledge, metrics and power to challenge them and ask them if they actually need the RAM they've allocated.
That said, if you combine vSphere's memory management with carefully-controlled overcommit limits, you should rarely have an issue in practice, the likelihood of running out of RAM for an extended period of time is relatively remote.
In addition to this, automated vMotion (called Distributed Resource Scheduling by VMware) is essentially a load-balancer for your VMs - if a single VM is becoming a resource hog, DRS should migrate VMs around to make best use of the cluster's resources.
What specific metric should be used to meter RAM usage. Tracking the
peaks of "Active" versus time?
Mostly covered above - your main concern should be "Active" RAM usage, though you should carefully define your overcommit thresholds so that if you reach a certain ratio (this is a decent example, though it may be slightly outdated). Typically, I would certainly stay within 120% of total cluster RAM, but it's up to you to decide what ratio you're comfortable with.
A few good articles/discussions on memory over-commit:
Best Answer
What this means is that you'll never have more than 8 threads executing in parallel in your virtual machine. However, through the magic of ESX resource allocation you can give those threads quite a bit of horsepower. You're not limited to the max-rate of your actual CPUs, ESX's CPU load-balancing methods will permit running faster than that... so long as what you're doing with it is able to do such slightly out of order execution.
This is accomplished by leveraging a queue structure for CPU resources. Work is dispatched to multiple processors as needed. A single vCPU may execute on any of the physical CPUs on the system (or the CPUs in the local-node if in a NUMA system). Achieving vCPU performance in excess of physical CPU performance is done by dispatching work from a single vCPU to multiple physical CPUs in parallel or at least very close together.
When it comes time to reassemble work returned by multiple CPUs to emulate a single vCPU, it does look to the VM like a single CPU working very fast. ESX reassembles the multiple work-units into the correct order.
Not all workloads are well suited to this, of course. Jobs that submit a lot of iterative work that is loosely related to each other if at all is perhaps the best case for this. Jobs that involve lots of tight dependencies with earlier instructions, such as with the recursive calls of crypto algorithms, won't be able to scale nearly as far.