Spring batch Multithreaded processing for Single file to Multiple FIle

multithreadingspringspring-batch

My problem statement. Read a csv file with 10 million data and store it in db. with as minimal time as possible.

I had implemented it using Simple multi threaded executor of java and the logic is almost similar to spring batch's chunk. Read preconfigured number of data from csv file and then create a thread, and passing the data to thread which validates data and then writes to file which runs in multi thread. once all the task is done I'm calling sql loader to load each file. Now I want to move this code to spring batch(I'm newbie to spring batch)

Here are my question
1. In task, is it possible to make ItemReader to Item writer multi threaded(as I read the file create a new thread to process the data before the thread writes to data)? if not I need to create two steps first step read the file which is single threaded and another step which is multi threaded writing to individual file, but how do I pass the list of data to another task from previous task.
2. In case if there are any failures in a single thread, how can I stop whole batch job processing.
3. How to retry the batch job in case of failure after certain interval. I know that there is retry option in case of failure but I could not find an option to retry the task after certain interval in case of failure. here I'm not talking about scheduler because I've batch job already runs under scheduler, but on failure it has to be re-run after 3 minutes are so.

Best Answer

Here is how I solved the problem.

  1. Read a file and chunk the file( split the file) using Buffered and File Channel reader and writer ( the fastest way of File read/write, even spring batch uses the same). I implemented such that this is executed before job is started( However it can be executed using job as step using method invoker)

  2. Start the Job with directory location as job parameter.

  3. Use multiResourcePartitioner which will get the directory location and for each file a slave step is created in separate thread
  4. In the Slave step get the file passed from Partitioner and use spring batchs itemreader to read the file
  5. Use the Database item writer( I'm using mybatis batch itemwriter) to push the data to Database.
  6. Its better to use the split count equal to commit-count of step.
Related Topic