vSphere's memory management is pretty decent, though the terms used often cause a lot of confusion.
In general, memory over-commit should be avoided as it creates exactly this type of problem. However, there are times when it cannot be avoided, so forewarned is forearmed!
What are the downsides of overcommitting and over-configuring resources
(specifically RAM) in vSphere environments?
The major downside of over-committing resources is that should you have contention, your hosts would be forced to balloon, swap or intelligently schedule/de-duplicate behind the scenes in order to give each VM the RAM it needs.
For ballooning, vSphere will inflate a "balloon" of RAM within a chosen VM, then give that ballooned RAM to the guest that needs it. This isn't really "bad" - VMs are stealing each other's RAM, so there's no disk swapping going on - but it could lead to mis-fired alerting and skewed metrics if these rely on analysing the VM's RAM usage, as the RAM won't be marked as "ballooned", just that it's "in use" by the OS.
The other feature that vSphere can use is Transparent Page Sharing (TPS) - which is essentially RAM de-duplication. vSphere will periodically scan all allocated RAM, looking for duplicated pages. When found, it will de-duplicate and free up the duplicated pages.
Take a look at vSphere's Memory Management whitepaper (PDF) - specifically "Memory Reclamation in ESXi" (page 8) - if you need a more in-depth explanation.
Assuming that the VMs can run in less RAM, is it fair to say that
there's overhead to configuring virtual machines with more RAM than
they need?
There's no visible overhead - you can allocate 100GB of RAM on a host with 16 GB (however, that doesn't mean you should, for the reasons above).
Total memory in use by all of your VMs is the "Active" curve shown in your graphs. Of course, you should never rely on just that figure when calculating how much you would like to overcommit, but if you have historical metrics as you have, you can analyse and work it out based on actual usage.
The difference between "Active" and "Consumed" RAM is discussed in this VMWare Community thread.
What is the counter-argument to: "if a VM has 16GB of RAM allocated,
but only uses 4GB, what's the problem??"? E.g. do customers need to be
educated?
The short answer to this is yes - customers should always be educated in best practices, regardless of the the tools at their disposal.
Customers should be educated to size their VMs according to what they use, rather than what they want. A lot of the time, people will over-specify their VMs just because they might need 16 GB of RAM, even if they're historically bumbling along on 2 GB day after day. As a vSphere administrator, you have the knowledge, metrics and power to challenge them and ask them if they actually need the RAM they've allocated.
That said, if you combine vSphere's memory management with carefully-controlled overcommit limits, you should rarely have an issue in practice, the likelihood of running out of RAM for an extended period of time is relatively remote.
In addition to this, automated vMotion (called Distributed Resource Scheduling by VMware) is essentially a load-balancer for your VMs - if a single VM is becoming a resource hog, DRS should migrate VMs around to make best use of the cluster's resources.
What specific metric should be used to meter RAM usage. Tracking the
peaks of "Active" versus time?
Mostly covered above - your main concern should be "Active" RAM usage, though you should carefully define your overcommit thresholds so that if you reach a certain ratio (this is a decent example, though it may be slightly outdated). Typically, I would certainly stay within 120% of total cluster RAM, but it's up to you to decide what ratio you're comfortable with.
A few good articles/discussions on memory over-commit:
Best Answer
I turned off dynamic memory for this VM, and the "Hardware Reserved" went away.
Which makes me think that the "Hardware Reserved" was actually that part of dynamic memory that the VM was entitled to, but was not actually allocated to it by the hypervisor, as it was not needed. Not sure if that's what it actually was.