Java – API to write huge excel files using java

apache-poiexceljava

I am looking to write to an excel (.xls MS Excel 2003 format) file programatically using Java. The excel output files may contain ~200,000 rows which I plan to split over number of sheets (64k rows per sheet, due to the excel limit).

I have tried using the apache POI APIs but it seems to be a memory hog due to the API object model. I am forced to add cells/sheets to the workbook object in memory and only once all data is added, I can write the workbook to a file! Here is a sample of how the apache recommends i write excel files using their API:

Workbook wb = new HSSFWorkbook();
Sheet sheet = wb.createSheet("new sheet");

//Create a row and put some cells in it
Row row = sheet.createRow((short)0);

// Create a cell and put a value in it.
Cell cell = row.createCell(0);
cell.setCellValue(1);

// Write the output to a file
FileOutputStream fileOut = new FileOutputStream("workbook.xls");
wb.write(fileOut);
fileOut.close();

Clearly, writing ~20k rows(with some 10-20 columns in each row) gives me the dreaded "java.lang.OutOfMemoryError: Java heap space".

I have tried increasing JVM initial heapsize and max heap size using Xms and Xmx parameters as Xms512m and Xmx1024. Still cant write more than 150k rows to the file.

I am looking for a way to stream to an excel file instead of building the entire file in memory before writing it to disk which will hopefully save a lot of memory usage. Any alternative API or solutions would be appreciated, but I am restricted to usage of java. Thanks! 🙂

Best Answer

Try to use SXSSF workbook, thats great thing for huge xls documents, its build document and don't eat RAM at all, becase using nio

Related Solutions

Java – How to read and write excel file

Try the Apache POI HSSF. Here's an example on how to read an excel file:

try {
    POIFSFileSystem fs = new POIFSFileSystem(new FileInputStream(file));
    HSSFWorkbook wb = new HSSFWorkbook(fs);
    HSSFSheet sheet = wb.getSheetAt(0);
    HSSFRow row;
    HSSFCell cell;

    int rows; // No of rows
    rows = sheet.getPhysicalNumberOfRows();

    int cols = 0; // No of columns
    int tmp = 0;

    // This trick ensures that we get the data properly even if it doesn't start from first few rows
    for(int i = 0; i < 10 || i < rows; i++) {
        row = sheet.getRow(i);
        if(row != null) {
            tmp = sheet.getRow(i).getPhysicalNumberOfCells();
            if(tmp > cols) cols = tmp;
        }
    }

    for(int r = 0; r < rows; r++) {
        row = sheet.getRow(r);
        if(row != null) {
            for(int c = 0; c < cols; c++) {
                cell = row.getCell((short)c);
                if(cell != null) {
                    // Your code here
                }
            }
        }
    }
} catch(Exception ioe) {
    ioe.printStackTrace();
}

On the documentation page you also have examples of how to write to excel files.

Java – How to avoid Java code in JSP files, using JSP 2

The use of scriptlets (those <% %> things) in JSP is indeed highly discouraged since the birth of taglibs (like JSTL) and EL (Expression Language, those ${} things) way back in 2001.

The major disadvantages of scriptlets are:

Reusability: you can't reuse scriptlets.
Replaceability: you can't make scriptlets abstract.
OO-ability: you can't make use of inheritance/composition.
Debuggability: if scriptlet throws an exception halfway, all you get is a blank page.
Testability: scriptlets are not unit-testable.
Maintainability: per saldo more time is needed to maintain mingled/cluttered/duplicated code logic.

~~Sun~~ Oracle itself also recommends in the JSP coding conventions to avoid use of scriptlets whenever the same functionality is possible by (tag) classes. Here are several cites of relevance:

From JSP 1.2 Specification, it is highly recommended that the JSP Standard Tag Library (JSTL) be used in your web application to help reduce the need for JSP scriptlets in your pages. Pages that use JSTL are, in general, easier to read and maintain.

...

Where possible, avoid JSP scriptlets whenever tag libraries provide equivalent functionality. This makes pages easier to read and maintain, helps to separate business logic from presentation logic, and will make your pages easier to evolve into JSP 2.0-style pages (JSP 2.0 Specification supports but de-emphasizes the use of scriptlets).

...

In the spirit of adopting the model-view-controller (MVC) design pattern to reduce coupling between the presentation tier from the business logic, JSP scriptlets should not be used for writing business logic. Rather, JSP scriptlets are used if necessary to transform data (also called "value objects") returned from processing the client's requests into a proper client-ready format. Even then, this would be better done with a front controller servlet or a custom tag.

How to replace scriptlets entirely depends on the sole purpose of the code/logic. More than often this code is to be placed in a fullworthy Java class:

If you want to invoke the same Java code on every request, less-or-more regardless of the requested page, e.g. checking if a user is logged in, then implement a filter and write code accordingly in doFilter() method. E.g.:

  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws ServletException, IOException {
      if (((HttpServletRequest) request).getSession().getAttribute("user") == null) {
          ((HttpServletResponse) response).sendRedirect("login"); // Not logged in, redirect to login page.
      } else {
          chain.doFilter(request, response); // Logged in, just continue request.
      }
  }

When mapped on an appropriate <url-pattern> covering the JSP pages of interest, then you don't need to copypaste the same piece of code overall JSP pages.

If you want to invoke some Java code to process a GET request, e.g. preloading some list from a database to display in some table, if necessary based on some query parameters, then implement a servlet and write code accordingly in doGet() method. E.g.:
```
  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      try {
          List<Product> products = productService.list(); // Obtain all products.
          request.setAttribute("products", products); // Store products in request scope.
          request.getRequestDispatcher("/WEB-INF/products.jsp").forward(request, response); // Forward to JSP page to display them in a HTML table.
      } catch (SQLException e) {
          throw new ServletException("Retrieving products failed!", e);
      }
  }
```
This way dealing with exceptions is easier. The DB is not accessed in the midst of JSP rendering, but far before the JSP is been displayed. You still have the possibility to change the response whenever the DB access throws an exception. In the above example, the default error 500 page will be displayed which you can anyway customize by an <error-page> in web.xml.

If you want to invoke some Java code to process a POST request, such as gathering data from a submitted HTML form and doing some business stuff with it (conversion, validation, saving in DB, etcetera), then implement a servlet and write code accordingly in doPost() method. E.g.:

  protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      String username = request.getParameter("username");
      String password = request.getParameter("password");
      User user = userService.find(username, password);

      if (user != null) {
          request.getSession().setAttribute("user", user); // Login user.
          response.sendRedirect("home"); // Redirect to home page.
      } else {
          request.setAttribute("message", "Unknown username/password. Please retry."); // Store error message in request scope.
          request.getRequestDispatcher("/WEB-INF/login.jsp").forward(request, response); // Forward to JSP page to redisplay login form with error.
      }
  }

This way dealing with different result page destinations is easier: redisplaying the form with validation errors in case of an error (in this particular example you can redisplay it using ${message} in EL), or just taking to the desired target page in case of success.

If you want to invoke some Java code to control the execution plan and/or the destination of the request and the response, then implement a servlet according to the MVC's Front Controller Pattern. E.g.:

  protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      try {
          Action action = ActionFactory.getAction(request);
          String view = action.execute(request, response);

          if (view.equals(request.getPathInfo().substring(1)) {
              request.getRequestDispatcher("/WEB-INF/" + view + ".jsp").forward(request, response);
          } else {
              response.sendRedirect(view);
          }
      } catch (Exception e) {
          throw new ServletException("Executing action failed.", e);
      }
  }

Or just adopt an MVC framework like JSF, Spring MVC, Wicket, etc so that you end up with just a JSP/Facelets page and a JavaBean class without the need for a custom servlet.

If you want to invoke some Java code to control the flow inside a JSP page, then you need to grab an (existing) flow control taglib like JSTL core. E.g. displaying List<Product> in a table:
```
  <%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
  ...
  <table>
      <c:forEach items="${products}" var="product">
          <tr>
              <td>${product.name}</td>
              <td>${product.description}</td>
              <td>${product.price}</td>
          </tr>
      </c:forEach>
  </table>
```
With XML-style tags which fit nicely among all that HTML, the code is better readable (and thus better maintainable) than a bunch of scriptlets with various opening and closing braces ("Where the heck does this closing brace belong to?"). An easy aid is to configure your web application to throw an exception whenever scriptlets are still been used by adding the following piece to web.xml:
```
  <jsp-config>
      <jsp-property-group>
          <url-pattern>*.jsp</url-pattern>
          <scripting-invalid>true</scripting-invalid>
      </jsp-property-group>
  </jsp-config>
```
In Facelets, the successor of JSP, which is part of the Java EE provided MVC framework JSF, it is already not possible to use scriptlets. This way you're automatically forced to do things "the right way".
If you want to invoke some Java code to access and display "backend" data inside a JSP page, then you need to use EL (Expression Language), those ${} things. E.g. redisplaying submitted input values:
```
  <input type="text" name="foo" value="${param.foo}" />
```
The ${param.foo} displays the outcome of request.getParameter("foo").
If you want to invoke some utility Java code directly in the JSP page (typically public static methods), then you need to define them as EL functions. There's a standard functions taglib in JSTL, but you can also easily create functions yourself. Here's an example how JSTL fn:escapeXml is useful to prevent XSS attacks.
```
  <%@ taglib uri="http://java.sun.com/jsp/jstl/functions" prefix="fn" %>
  ...
  <input type="text" name="foo" value="${fn:escapeXml(param.foo)}" />
```
Note that the XSS sensitivity is in no way specifically related to Java/JSP/JSTL/EL/whatever, this problem needs to be taken into account in every web application you develop. The problem of scriptlets is that it provides no way of builtin preventions, at least not using the standard Java API. JSP's successor Facelets has already implicit HTML escaping, so you don't need to worry about XSS holes in Facelets.

Best Answer

Related Solutions

Java – How to read and write excel file

Java – How to avoid Java code in JSP files, using JSP 2

See also:

Related Topic