Multiple Mongos on one server

If you are going to run the database remotely, run it at the same network segment than your app servers, or at least the same datacenter.

Even if you get low(ish) latency like < 10 ms, that can impact on your queries depending on the type of data load you have.
Also you end adding another point of failure to your system, communication between your application servers and database servers may fail. MongoDB doesn't use any encryption on it's protocol and your data will be running clear between god knows how many hops unless you tunnel it some way. And, as you pointed, there's the bandwidth use. Overall I suggest you to get another node to run your DB, but at the same place of your application server.

The reason Mongo/10Gen recommends against running multiple instances of Mongo on the same system has to do with resource availability assumptions. The mongod process assumes it's the only major tenant of the system, and running two such side-by-side will lead to overall worse performance than just running one mongod just with two databases.

Mongo is quite capable of running multiple databases under a single mongod process. These are discrete databases, just under a single master process. Each database handles its own authentication, which gives you separation. You'd run multiple mongod processes if you need different mongo versions for your products.

I'm not enough of an RDBMS DBA to know the best-practice arguments behind separating products by discrete database-binary processes, so I can't refute/assuage/talk-down each point. But as I understand it, accepted practice in Mongo environments is to have a separate database per product.