Spring Batch read from database once, write to another database and write to CSV

springspring-batch

I'm new to Spring Batch and trying to implement a batch job where I

  1. Read from a MySQL database
  2. Write the results to a CSV file
  3. Do some processing of MySQL result set and write to another database.

I've looked through this question on StackOverflow, but the main accepted answer was essentially to implement two steps that read twice from the database:

<job id="myJob">
    <step id="step1" next="step2">
        <tasklet>
            <chunk reader="reader" writer="typeAwriter"/>
        </tasklet>
    </step>
    <step id="step2">
        <tasklet>
            <chunk reader="reader" processor="processor" writer="typeBwriter"/>
        </tasklet>
    </step>
</job>

Isn't there a more efficient way of doing this than reading twice from the MySQL database? For example, what if you query is quite large and drags system performance?

Best Answer

What you need is a chunk strategy instead of tasklet. The ItemReader will read chunks from your database, the processor will process you data and then you can for each item send them to the ItemWriter that can write to database and file. This is one of many possible strategies, I don't know the details on your business logic, but I think this is information enough to get you going on your own ideas.

<?xml version="1.0" encoding="UTF-8"?>
<job id="customerJob" xmlns="http://xmlns.jcp.org/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/jobXML_1_0.xsd"
version="1.0">

    <step id="step1">
        <chunk item-count="5">
            <reader ref="itemReader"/>
            <processor ref="itemProcessor"/>
            <writer ref="itemWriter"/>
        </chunk>
    </step>
</job>

This is the JSR-352 XML type, for Spring you have corresponding approach.

Related Topic