Is it a microservice if the services pull data from the same database/file storage

Does anyone know how the rainbow unicorns (Netflix, Amazon, Google, etc.) handle large files / data exchange between their services?

Unfortunately I do not know how they deal with such problems.

The problem is this - Does it make sense for all our microservices to be accepting this unique ID as part of their API for the purposes of interacting with documents or not?

It violates the Single Responsibility Principle, which should be inherently in your microservice's architecture. One microservice - logically one, physically many instances representing one - should be dealing with one topic.

In the case of your document store, you have one point, where all queries for documents go (of course you could split this logical unit up into multiple document stores for several kinds of documents).

• If your "application" needs to work on a document, it asks the respective microservice and processes its result(s).

• If another service needs an actual document or parts of it, it has to ask the document service.

One of the key contention points we are facing is how to communicate large quantities of data between our different services.

This is an architectural problem:

1. Decrease the need to transfer big amounts of data

   Ideally, each service has all of it's data and needs no transfer to simply serve requests. As an extension of this idea - if you have the need to transfer data, think of redundancy (*in a positive way_): Does it make sense to have the data redundant in many places (where they are needed)? Think of how possible inconsistencies might harm your processes. There is no transfer faster as actually none.

2. Decrease the size of the data itself

   Think of how you could compress your data: Starting with actual compression algortihms up to smart data structures. The less goes over the wire, the faster you are.

The fact that those different processes use separate connections is (mostly) irrelevant.

The usual concern is that opening and closing a connection is expensive in terms of time and CPU resources. With many databases, this concern is invalid because of connection pooling: there is an overhead the first time you connect to the database, but the next time you need to do a query within the same process, an existent connection is reused under the hood, even if in code, you explicitly closed the connection.

What will happen in your case is that you'll pay the cost of opening twenty connections the first time when twenty microservices start after, for instance, the server reboots. Once this is done, each service will reuse connections already opened before.

What would be a problem is if you have a lot of microservices accessing the database, or if the database, for a given reason, cannot accept too many connections. Since you are talking about twenty microservices, I won't be concerned; for instance, “SQL Server allows a maximum of 32,767 user connections.” (source) If you use a more exotic database or a database which has an exotic configuration or if each service needs, for some reason, to open many connections at once, this could indeed become an issue.