Java – Orchestration engines and frameworks

frameworksjavaorchestrationworkflow

I'm looking for an orchestration framework/engine/toolkit with which to replace/upgrade an existing software, mainly because of scalability limitations. By orchestration I mean asynchronous and distributed execution of generic tasks and workflows.

More specifically the requirements are pretty much these:

Wrapping and execution of generic tasks, in Java if language dependent
API for tasks and workflows on-demand triggering
Scheduling would be nice as well
Support for distributed architecture & scalability (mainly for big numbers of small tasks)
Persistency and resilience
Advanced workflow configuration capabilities (do this, then these 3 tasks in parallel, then this, having priorities, dependencies…)
Monitoring and administration UI (or at least API)

The existing system is an old fashion monolithic service (in Java) that has most of that, including the execution logic itself which should remain as untouched as possible.

Does anyone have experience with a similar problem? It seems to me it's supposed to be pretty common, would be strange if I have to implement it myself entirely. I found some questions here (like this and this) discussing the theory of orchestration and choreography systems, but not real examples of tools implementing it. Also I think we're not exactly talking about microservices – the tasks are not prolonged and heavy, they're just many, running in the background executing short jobs of many types. I wouldn't create a service for every job type.

I'm also not looking for cloud and container services at this point – to my understanding the deployment is a different issue.

The closest I got is the Netflix Conductor engine, which answers most of the requirements by running an orchestration server that manages tasks implemented in servlets (or any web services in any language – a plus). However it seems like it's built mainly for arranging heavy tasks in a workflow rather than running a huge number of small tasks, which makes me wonder what would be the overhead of invoking many small tasks in servlets for example.

Does anyone have experience or any input on the Conductor or other tools I could use? Or even my entire approach to the problem?

EDIT: I realize it's kind of a "research advice needed" so let's put it simply in 3 questions:

Am I right to look for an orchestration solution for the requirements above?
Does anyone have experience with the Netflix Conductor? Any feedback on it?
Does it have good competitors?

Best Answer

Perhaps you are looking for something like Airflow https://airflow.apache.org/ ?

Wrapping and execution of generic tasks, in Java if language dependent

https://github.com/apache/incubator-airflow/tree/master/airflow/hooks https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators

API for tasks and workflows on-demand triggering

https://airflow.apache.org/api.html (experimental)

Scheduling would be nice as well

think of cron on steroids - https://airflow.apache.org/scheduler.html

Support for distributed architecture & scalability (mainly for big numbers of small tasks)

scale with dask or celery nodes - Airflow + celery or dask. For what, when?

Persistency and resilience

uses a postgres db & rabbitMQ - if your deployment arch is stateless ( eg. repeatable containers & volumes with docker) you should be in good shape with WAL replication if you use Kubernetes or Consul there are other ways to implement more resilience on the other components

Advanced workflow configuration capabilities (do this, then these 3 tasks in parallel, then this, having priorities, dependencies...)

Airflow uses DAG's. The capabilities can be called fairly advanced. You also have parameter sharing using XCOMs if you really need that

Monitoring and administration UI (or at least API)

Has one, shows tasks & schedules & has a gantt view. also can see logs & run details easily & also manually schedule tasks directly from the UI

also look at oozie & azkaban

did this help?

Related Solutions

Java – reflection and why is it useful

The name reflection is used to describe code which is able to inspect other code in the same system (or itself).

For example, say you have an object of an unknown type in Java, and you would like to call a 'doSomething' method on it if one exists. Java's static typing system isn't really designed to support this unless the object conforms to a known interface, but using reflection, your code can look at the object and find out if it has a method called 'doSomething' and then call it if you want to.

So, to give you a code example of this in Java (imagine the object in question is foo) :

Method method = foo.getClass().getMethod("doSomething", null);
method.invoke(foo, null);

One very common use case in Java is the usage with annotations. JUnit 4, for example, will use reflection to look through your classes for methods tagged with the @Test annotation, and will then call them when running the unit test.

There are some good reflection examples to get you started at http://docs.oracle.com/javase/tutorial/reflect/index.html

And finally, yes, the concepts are pretty much similar in other statically typed languages which support reflection (like C#). In dynamically typed languages, the use case described above is less necessary (since the compiler will allow any method to be called on any object, failing at runtime if it does not exist), but the second case of looking for methods which are marked or work in a certain way is still common.

Update from a comment:

The ability to inspect the code in the system and see object types is not reflection, but rather Type Introspection. Reflection is then the ability to make modifications at runtime by making use of introspection. The distinction is necessary here as some languages support introspection, but do not support reflection. One such example is C++

Java – What are the differences between a HashMap and a Hashtable in Java

There are several differences between HashMap and Hashtable in Java:

Hashtable is synchronized, whereas HashMap is not. This makes HashMap better for non-threaded applications, as unsynchronized Objects typically perform better than synchronized ones.
Hashtable does not allow null keys or values. HashMap allows one null key and any number of null values.
One of HashMap's subclasses is LinkedHashMap, so in the event that you'd want predictable iteration order (which is insertion order by default), you could easily swap out the HashMap for a LinkedHashMap. This wouldn't be as easy if you were using Hashtable.

Since synchronization is not an issue for you, I'd recommend HashMap. If synchronization becomes an issue, you may also look at ConcurrentHashMap.

Best Answer

Related Solutions

Java – reflection and why is it useful

Java – What are the differences between a HashMap and a Hashtable in Java

Related Topic