Choosing right ec2 instance type for Mongodb with sharded cluster configuration

There are two questions here, one for MS clustering, and another one for Mongo.

MS Clustering

The decision of where to put the public, heart-beat, inter-node communication, and quorum drive is significant. Also cluster architecture makes a difference; you pick different quroum options if the two nodes are in adjacent racks than if they were in completely different datacenters.

Put the heartbeat on the same interface/subnet as the public interface

This theory holds that if you lose your public interface, you want the heartbeat to fail because this node is effectively down to users.

Put the heartbeat on it's own private interface/subnet

This theory holds that something outside of the cluster is arbiting who is doing what role, and unnecessary node-death is to be avoided.

Put the WFS on the heartbeat network

If the two nodes are in the same overall network (the same set of switches is supporting the non-public networks for both nodes) then putting the WFS on the heartbeat network doesn't introduce any new vulnerabilities.

If the two nodes are in different network fault domains (such as different datacenters), this is a bad idea. The heartbeat network provides the 'node majority' quorum option, and the WFS provides the 'File Share Majority' quorum option. You really want both options to be in separate fault domains.

Your revised diagram makes sense if both nodes are in the same data-center, though I myself would but the heartbeat on the public side.

MongoDB

MongoDB is a bit simpler. With even numbers of nodes, you absolutely want a third node to act as tie-breaker. They're pretty clear about that. However, your diagram states:

Up to 12 replica members (7 can vote).

7 is an odd number. You don't require an Arbiter.

Unlike Microsoft clusters, Mongo's cluster voting doesn't care about multiple avenues of network to break voting deadlocks. Because of this, separate arbitration and cluster-internal networks do not provide any meaningful increase in robustness. The only reason you'd want a separate arbitration network is if replication traffic was expected to be so heavy that election-packets (the heartbeat, actually) would get pushed so far down the stack that it would miss the 10 second timeout.

First, if you take a snapshot, it will include the oplog - the oplog is just a capped collection living in the local database. Snapshots will get back to a point in time, and assuming you have journaling enabled (it is on by default), you do not need to do anything special for the snapshot to function as a backup.

The only absolute requirement is that the EBS snapshot has to be recent enough to fall within your oplog window - that is the last (most recent) operation recorded in the snapshot backup oplog must also still be in the oplog of the current primary so that they can find a common point. If that is the case it will work something like this:

1. You restore a secondary from an EBS snapshot backup
2. The mongod starts, looks for (and applies) any relevant journal files
3. Next, the secondary connects to the primary and finds a common point in the two oplogs
4. Any subsequent operations from the primary are applied on the RECOVERING secondary
5. Once the secondary catches up sufficiently, it moves to the SECONDARY state and the backup is complete

If the snapshot is not recent enough, then it can be discarded - without a common point in the oplog, the secondary will have to resync from scratch anyway.

To answer your specific questions:

Do I need to record oplogs and use those in conjunction to restore after a failure?

As explained above, if you snapshot, you already are backing up the oplog

Should I spin up another instance within the replica set specifically for backups and snapshot that vs. taking snapshots of primary and secondary? If so, we're back to the oplog issue aren't we?

There's no oplog issue beyond the common point/window one I mentioned above. Some people do choose to have a Secondary (usually hidden) for this purpose to avoid adding load to a normal node. Note: even a hidden member gets a vote, so if you added one for backup purposes you can remove the arbiter from your config, you would still have 3 voting members.

Should I snapshot each replica volume and rely on on the replica set completely to cover the time between failure and the last snapshot?

Every member of a replica set is intended to be identical - the data is the same, any secondary can become primary etc. - these are not slaves, every replica set member contains the full oplog and all the data.

So, taking multiple snapshots (assuming you trust the process) is going to be redundant (of course you may want that redundancy). And yes, the whole intention of the replica set functionality is to ensure that you don't need to take extraordinary measures to use a secondary in this way (with the caveats above in mind, of course).