Microservice architecture pattern for Batch based system

apache-kafkabatchevent-programminghadoopmicroservices

I have been exploring the microservice architecture for the batch-based system.

Here is our current setup:

Code:
We have 5 systems that are internally connected and they pass data from one system to another. Currently entire logic is sitting in Oracle as PL/SQL, Hadoop(Hive, Impala, Spark, etc) and Shell scripts.

Communications:
These systems share the data either through cross DB Table grants or they export the data in files and send it to each other.

Triggers:
These systems send trigger through custom workflow engines or processes look for some files in a repetitive mode.

Now coming to the main question:
Is it a good idea to convert these processes into microservices(Code) and use Kafka( Communication and Trigger) so that they can share data and we can have a more distributed well-choreographed process flow. Just to give an example, When one system finish process it can send data in Kafka is available(this act as trigger and producer) and all consumer system can start using that data in parallel instead of sending data in files or hitting databases individually.

Edit based on comments:
Looking for some insight on microservice based architecture for the Batch based system, Irrespective of the current setup or think we are building a brand new system.

Any suggestion through link/Blog, tools, and technologies would be greatly appreciated.

Best Answer

It is a decent idea. But what are your concerns(pain points) ? A micro service will mainly be:

a) more complex (it is a distributed system after all)

b) more maintainable

c) more flexible (evolution perspective, as I saw you are concerned with redesigning parts of the system in the future)

I have used the same design as you are asking for batch based system. It is not bad. And what I really like is how you can have multiple components listen to a single event(I suppose that is where the magic happens). Have a look at this book : Designing event driven systems Concepts and Patterns for Streaming Services with Apache Kafka . It gets a bit tiring in some chapters but most of it is really good. I think it misses some aspects too but it will give you plenty of insight and is rather simple.

Related Topic