I'm writing a small microservice-based application. One of the services is assigned the task of querying an external API and processing the JSON response (filtered list of houses). Since I'm using Protocol Buffers to serialize the message that will be sent to the message broker I need to convert the JSON output to the appropriate Protocol Buffer format.
This is a diagram that illustrates the overall architecture:
The problem is, that since the JSON response is long, paginated and also has many nested fields there is no easy way to manually create a corresponding ProtoBuff message struct.
I could potentially send the JSON response as a text string and deal with it on the database service side, but this then presents the problem of how to store each house object inside the database so that it can be queried later.
Unmarshalling the whole response into a Go struct would pose the same problems as trying to create a ProtoBuff message struct in the first place. Storing it inside a database as a raw JSON field is not an option since I need to be able to query some of the fields and actually extract each house description from the body of the response.
Additionally, creating a database schema that matches hundreds of different fields and nested objects, seems like a very tedious way of going about it.
What are the best practices when it comes to processing substantial JSON output across many web services?
Best Answer
There is no clear reason on why to convert the JSON format in the message broker. Probably, leaving it a JSON object through the message broker is the way to go.
In terms of how deep you should model it, it depends on several factors. Here are some hints:
The storing decision also depends on multiple factors (check the following articles and questions about Relational vs. non-relational 1, 2 and 3 - and there is much more available). You should have in mind that relationship databases (such as MySQL) are used to store relations.