Java – Elegant Ways to Avoid Hard Coding CSV File Format

coding-stylejava

I know this is trivial issue, but I just feel this can be more elegant.

So I need to write/read data files for my program, lets say they are CSV for now. I can implement the format as I see fit, but I may have need to change that format later. The simply thing to do is something like

out.write(For.getValue()+","+bar.getMinValue()+","+fi.toString());

This is easy to write, but obviously is guilty of hard coding and the general 'magic number' issue. The format is hard-coded, requires parsing of the code to figure out the file format, and changing the format requires changing multiple methods.

I could instead have my constants specifying the location that I want each variable to be saved in the CSV file to remove some of the 'magic numbers'; then save/load into the an array at the location specified by the constants:

int FOO_LOCATION=0;
int BAR_MIN_VAL_LOCATION=1;
int FI_LOCATION=2
int NUM_ARGUMENTS=3;

String[] outputArguments=new String[NUM_ARGUMENTS];
outputArguments[FOO_LOCATION] = foo.getValue();
outputArgumetns[BAR_MIN_VAL_LOCATION] = bar.getMinValue();
outptArguments[FI_LOCATOIN==fi.toString();

writeAsCSV(outputArguments);

But this is…extremely verbose and still a bit ugly. It makes it easy to see the format of existing CSV and to swap the location of variables within the file easily. However, if I decide to add an extra value to the csv I need to not only add a new constant, but also modify the read and write methods to add the logic that actually saves/reads the argument from the array; I still have to hunt down every method using these variables and change them by hand!

If I use Java enums I can clean this up slightly, but the real issue is still present. Short of some sort of functional programming (and java's inner classes are too ugly to be considered functional) I still have no obvious way of clearly expressing what variable is associated with each constant short of writing (and maintaining) it in the read/write methods. For instance I still need to write somewhere that the FOO_LOCATION specifies the location of foo.getValue().

It seems as if there should be a prettier, easier to maintain, manner for approaching this?

Incidentally, I'm working in java at the moment, however, I am interested conceptually about the design approach regardless of language. Some library in java that does all the work for me is definitely welcome (though it may prove more hassle to get permission to add it to the codebase then to just write something by hand quickly), but what I'm really asking is more about how to write elegant code if you had to do this by hand.

Best Answer

Regardless of language, if you don't want to hard code the values, then one will need some sort of meta data that describes how the data will be mapped, formatted, and output.

In this example, it could be called a mapper or formatter.

Map/format files (XML, JSON, or simliar) would describe how the data would be formatted and written. Your application would read in the map/format file and use it to create the output.

Then your application could format the CSV anyway you see fit, without a programming change. One could also extend this to flat or fixed formats and XML as well. Then your code is generic as it uses the mapping meta data to create the CSV file.

For the CSV case, at a high level one would need to describe:

Name and Order of fields
Delimiter (Sometimes comma is not used)
Whether or not to include a header
Whether or not to include quotes around the data

Just as a side note, there is a time difference to develop the two applications. Hardcoded field values are much faster to develop, although as you have pointed out less elegant. But if you need to get something done quickly, the aproach is OK.

Developing something more generic would take more time up front, but if your producing a lot of CSV formatted files of different format, in the long run one would get ROI on it.

With this approach, one can also write some sort of nice "GUI" for business analysts to use to create the map files so developers will be less involved in the overall process.

Related Solutions

Should a String Constant Be Defined if Used Only Once?

Try this. The initial reflection is certainly expensive, but if you're going to use it many many times, which I think you will, this is most certainly a better solution what what you're proposing. I don't like using reflection, but I find myself using it when I don't like the alternative to reflection. I do think that this will save your team a lot of headache, but you must pass the name of the method (in lowercase).

In other words, rather than pass "name", you would pass "fullname" because the name of the get method is "getFullName()".

Map<String, Method> methodMapping = null;

public Object getNode(String name) {
    Map<String, Method> methods = getMethodMapping(contact.getClass());
    return methods.get(name).invoke(contact);
}

public Map<String, Method> getMethodMapping(Class<?> contact) {
    if(methodMapping == null) {
        Map<String, Method> mapping = new HashMap<String, Method>();
        Method[] methods = contact.getDeclaredMethods();
        for(Method method : methods) {
            if(method.getParameterTypes().length() == 0) {
                if(method.getName().startsWith("get")) {
                    mapping.put(method.getName().substring(3).toLower(), method);
                } else if (method.getName().startsWith("is"))) {
                    mapping.put(method.getName().substring(2).toLower(), method);
                }
            }
        }
        methodMapping = mapping;
    }
    return methodMapping;
}

If you need to access data contained within members of contact, you might consider building a wrapper class for contact which has all methods to access any information required. This would also be useful for guaranteeing that the names of the access fields will always remain the same (I.e. if wrapper class has getFullName() and you call with fullname, it will always work even if contact's getFullName() has been renamed -- it would cause compilation error before it would let you do that).

public class ContactWrapper {
    private Contact contact;

    public ContactWrapper(Contact contact) {
        this.contact = contact;
    }

    public String getFullName() {
        return contact.getFullName();
    }
    ...
}

This solution has saved me several times, namely when I wanted to have a single data representation to use in jsf datatables and when that data needed to be exported into a report using jasper (which doesn't handle complicated object accessors well in my experience).

Java REST API – How to Check for Duplicates Before Inserting Records

I cannot be sure that each CSV file has only been processed once...

You might want to attempt to solve your question by handling this first. If I am getting this right, the crux of your problem doesn't appear to be individual duplicate transactions (since you mentioned "I know for sure that there are no duplicate records in each CSV file"), but to prevent duplicate processing per file.

Hence, you can consider adding some kind of state logic in your Java application that knows whether a file has been processed by computing and storing its checksum, e.g. its MD5 hash. Once you have a matching checksum, you know that there's a good chance the file has been processed before. You can perform further verification such as by inspecting the number of lines, or other certain unique identifiers of each file.

Further extending this idea, if there are possibilities of the same transaction appear across different CSV files, then your only other option, besides updating the database schema to handle duplicate records properly, is to store all the processed transactions locally within your Java application. If there can be multiple instances of your application (either on the same computer, or across a network), then you'll either need yet another centralized database to handle this, or some distributed data grid... by then, the better option is still back to the drawing board to improve on your existing database schema.

edit

To flip things around, the other considerations to look into ~~, if changing the database schema to handle duplicates nicely is entirely don't-even-think-about-it nigh impossible,~~ is to evaluate how much data your Java application will need to process at any given time, and how fast the connection is between the database and your application.

On the lower end, say your application is processing only 10 records per file, averaging one file an hour. The network connection is very good, say almost as good as accessing a locally-hosted database. In this case, I don't think there's much of a performance impact from having to query all the records.

On the extreme end, your application is expected to read thousand-line-long transaction files every 10 seconds, and the network connection is extremely bad, say taking a minute to query all the records. In this case, you have more concern about processing the files in a speedy manner, and this is how you can probably suggest modifying the database schema. :)

So, assuming all is fine in the lower-end case, what would be an efficient way of comparing a relatively large data set with a smaller input set for duplicates? I'll suggest marshalling the XML payload you get into a HashSet. Also, I hope you have a Transaction domain class that has properly-implemented hashCode() and equals() methods. A Java 8 potential solution would then be:

// assuming your database records are marshalled into currentSet
inputSet.stream().filter(v -> !currentSet.contains(v))
                    .forEach( /* these are the new records to send to the database */);

Also, the elephant in the room: concurrent insertions. Will there be any? If so, how do you intend to handle it then?

Best Answer

Related Solutions

Should a String Constant Be Defined if Used Only Once?

Java REST API – How to Check for Duplicates Before Inserting Records

Related Topic