Java – Multithreaded batch processing to write to and read from database

javamultithreadingspringspring-batch

I am supposed to design a component which should achieve the following tasks by using multiThreading in Java as the files are huge/multiple and the task has to happen in a very short window:

Read multiple csv/xml files and save all the data in database
Read the database and write the data in separate files csv & xmls as per the txn types. (Each file may contain different types of records life file header, batch header, batchfooter, file footer, different transactions, and checksum record)

I am very new to multithreading & doing some research on Spring Batch in order to use it for the above tasks.

Please let me know what you suggest to use traditional multithreading in Java or Spring Batch. The input sources are multiple here and output sources are also multiple.

Best Answer

I would recommend going with something from framework rather than writing whole threading part yourself. I've quite successfully used Sping's tasks and scheduling for scheduled tasks that involved reaching data from DB, doing some processing, and sending emails, writing data back to database).

Related Solutions

Java – In Java, what is the best way to determine the size of an object

You can use the java.lang.instrument package.

Compile and put this class in a JAR:

import java.lang.instrument.Instrumentation;

public class ObjectSizeFetcher {
    private static Instrumentation instrumentation;

    public static void premain(String args, Instrumentation inst) {
        instrumentation = inst;
    }

    public static long getObjectSize(Object o) {
        return instrumentation.getObjectSize(o);
    }
}

Add the following to your MANIFEST.MF:

Premain-Class: ObjectSizeFetcher

Use the getObjectSize() method:

public class C {
    private int x;
    private int y;

    public static void main(String [] args) {
        System.out.println(ObjectSizeFetcher.getObjectSize(new C()));
    }
}

Invoke with:

java -javaagent:ObjectSizeFetcherAgent.jar C

Java – How to create a file and write to it

Note that each of the code samples below may throw IOException. Try/catch/finally blocks have been omitted for brevity. See this tutorial for information about exception handling.

Note that each of the code samples below will overwrite the file if it already exists

Creating a text file:

PrintWriter writer = new PrintWriter("the-file-name.txt", "UTF-8");
writer.println("The first line");
writer.println("The second line");
writer.close();

Creating a binary file:

byte data[] = ...
FileOutputStream out = new FileOutputStream("the-file-name");
out.write(data);
out.close();

Java 7+ users can use the Files class to write to files: