Design – Java Design for Data enrichment based on logic defined in a database

designdesign-patterns

I have the following requirement.

Read data from a messaging Queue and process each message to enrich the message and then finally send the enriched message to a different system.
The enrichment logic is in DB and it is based on message type and each logic can be of different language. Each message will have some type like, bank, insurance, …

Sample enrichment logic data in Mongo DB,

{
  type : "Bank",
  logics : [
    {
      "key" : "location.phone",
      "type" : "javascript",
      "logic" : "function getPhone() { return '888-888-8888';}"
    },
    {
      "key" : "location.zip",
      "type" : "groovy",
      "logic" : "function getZip() { return '56781';}"
    },
    {
      "key" : "location.address",
      "type" : "groovy",
      "logic" : "function getAddress() { return '1234 new address';}"
    },
  ]
}

Now on receiving each message from the Queue, Check the type and enrich the data based on the logics for that type.

This is a spring boot based non-web Java Application.

My executing would start from the below run method. 

public class AppRunner implements ApplicationRunner {

  // Other Bean injection is here


    @Override
    public void run(final ApplicationArguments args) throws Exception {
       // Application starts here ...
    }
}

I defined EnrichmentLogicDAO to load the logic from the DB and have a EnrichmentDBService for the DAO.
Have an interface with a method exeucte( Command pattern ). There are 2 implementors. 1. Javascript, 2. Groovy

In Javascript executor,

ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");

In Groovy executor,

ScriptEngine engine = new ScriptEngineManager().getEngineByName("groovy");

In both,

engine.eval(logic1);
engine.eval(logic2);
....

Load all the logic from DB. Javascript based logic should be loaded in JavaScriptExecutor engine and groovy based logic should be loaded in GroovyExecutor engine.

Question 1 :-

How to load all the logics to the corresponding engine?

Currently, I have my code to load the logic inside the AppRunner's run method and call the corresponding Executor class static method based on the type from DB to load the logics into the engine.

Question 2 :-

The code to execute the enrichment logic remain same for both the engine. Only the engine will be different based on the logic type.

I need some help on Loading the logics to the corresponding engine and invoke the logic on each message.

Also, I thought of using the Chain of responsibility pattern, so that the input message would be passed to the series of engines and the message will be enriched based on the logics loaded in each engine. Is there any other better approach.

Best Answer

To start things off simple, you should create a simple loop taking each "logic" and performing each operation from start to finish. So have add a dependency for your EnrichmentDBService and directly request the logics for your input.

Then for each logics, you apply the change. To apply each change, you should use an enum or some sort of direct mapping between the type "javascript" or "groovy" and the appropiate ScriptEngine instance pertaining to that particular language.

Using this mapping, grab the correct ScriptEngine instance and execute its code. What gets returned you must set to an object using "key" value presumably.

Once you've finished all logics in this way you're done! Return the finished object.

If order is not crucial and there are many logics, you could consider using a thread pool to handle the tasks. I wouldn't suggest Chain of Responsibility because it seems like the actual work being done is the same here. However, I fear that what they want from you is that you interpret the values returned as a sort of mask for you to apply to the actual data.

If, for instance, '888-888-8888' is a mask for applying a number, then you'd need some sort of baseline rule for this. Does this mean whenever you see '8', you represent this as a digit? What about '1234 new address'? Is '1234' supposed to be replaced with the street number? Why wouldn't you interpret this as a literal '1234'?

These questions you cannot answer yourself. The one providing the specifications should tell you this, and you need to look for potential inconsistencies and bring them to the attention of the one performing the analysis. Do this until all cases are covered and there are no inconsistencies before you begin coding.

Good luck!

Related Topic