Architecture – A “Composite API” layer

We are giving services a complete overhaul at work: Swapping soap for rest, resculpting the domains to give better separation of concerns, etc.

These new services will be publicly available. During one of our on-going conversations to refine our architectural vision, the idea of a "Composite API" was introduced. I am hoping to get some community feedback regarding the validity of this approach. I will list the pros and cons as I understand them. Disclaimer: I am no SOA expert

Pros

1. Clients will received fully hydrated objects instead of having to make multiple service calls to compose something useful
2. A&A is isolated to one place and does not need to muddy service behavior

Cons (again, this is purely my understanding).

Composed resources obscure the security of individual resources
Requesting an individual child resource might return a 403, but requesting a parent resource returns a 200 with content and is missing parts of the data. While this may be acceptable client-side, I feel it's a bit or a risk server-side. Developers will have to maintain the security of the individual resources in one way, and the security of composed resources in another.

After reading Martin Fowler's article on microservices, I do think the domain service should govern the actual write, but this doesn't necessarily contradict the API gateway.

Complicated reads/writes based on user role

Basically, we may serve some info on a dto that some users should not see. Serving the id is not an issue, but serving the object that the id represents is. Of course, the composite api could simply not serve the object, but that's when the real problem comes. The back-end service no longer cares who the user is, only that the dto contains all necessary information to complete a write successfully. If we naively send null for a field because the client didn't have it, we'd blow away valid data. This means the composite layer must somehow fetch the existing object and merge with the pending request to make sure the appropriate writes happen. If we control what fields are updated within the respective service, it's as simple as not changing the field before writing to the database. If not, it seems like a lot more work.

Positing the API gateway leaves 2 options:

1. Let the composite api construct the aggregated response
2. Allow each domain service to construct a fully hydrated response, but only accept trimmed requests.

Option 1 does seem to be the least amount of code.

Consolidated A&A, not quite

For better or worse, we still have audit fields on a lot of tables. This means we need to include the username on an UpdatedBy field. Because the payload may have many entities on it and because we don't care about the audit fields client-side, we left them off the service DTOs. Instead, we are now passing custom headers into each service that contains the userId, username, and potentially the user roles. When the request hits the client, we convert those headers into something that will be available in the code block where we are creating/updating the domain models. I feel like there are plenty of plugins for various frameworks and platforms do this for us if we change our services to rely on an auth protocol, namely OAuth. Internally users/apps, and external users/apps should hit a common auth provider, get a token, and consume services as allowed. This makes each service responsible for controlling how a write should happen, which I kinda think is best place for that logic.

While passing the header works when there is a header, it doesn't work so well when there isn't. Say we have a backend service that needs to call a domain service. We could leverage the same authentication protocol a user would have, which isn't bad. Now say that backend service is doing work on a user's behalf. How would we distinguish the service acting on its own from the service acting on a behalf of a user? Perhaps it's best to keep the domain services behind the API gateway for external use and allow backend services to use custom headers to simulate who the user was.

Payload size

The client will have to fetch all of the data either way, but does it really need to write it all back? Why can't we just send ids?

It can send back the ids. The domain service will expect that, and the API gateway will act as a reverse proxy in that sense.

Domain ambiguity
If a client receives a composed object within one request, they get the impression they can put the resource back with modifications anywhere, even if some of those parts came from some other domain. Responding with ids only removes this ambiguity.

Concurrency
We'd have to expose data versions on the DTOs, and potentially versions on parts of the DTOs, hope for a good write, relay a failure back to the client, and determine which part of the transaction failed. Granted, this is true with any transactional write where "last in wins" is unacceptable. I think it just exacerbates the problem.

Service versioning requires more coupling
For mobile clients, service versioning is a big deal, so we'll already have service versioning. With a composite api, we now need to support versioning in a third place. Not the worst thing ever, but still, I'd rather not.

Strongly-typed clients may require more work to adapt
Consider the relationship between a lookup object and a hydrated dto that is composed with that lookup object. When the lookup object changes server-side, it will have to change in two places client-side, once on the lookup, and once on the hydrated object. If we choose not to compose them, the client only needs to send back ids to the server.

Why not just introduce an ESB

If we really need this, why not use an existing product instead of writing another service that is coupled to all the other services?

It looks like the API gateway is between the client and domain services, whereas an ESB might not be. In a lot of ways they are very similar. I'm still comparing the two.

I poured it on a little at the end, but I think these are all valid concerns. Anyway, I'd like to get a lot of feedback on this.

I finally found some documenation:

• Microservices.io: API Gateway Pattern
• Optimizing the Netflix API