Java – Parsing and validation of a csv file

Architecturecouplingengineeringjavasingle-responsibility

I am trying to create a library where I need to validate and parse a file in a CSV-like format and then use this data to generate a Tree data structure.

At the moment I split the process into two steps:

Validate the file and store its lines in a list (in the parsing package)
Read the list plus a string array containing a subset of columns to generate the tree (in the hierarchy package)

I was thinking if the steps can be simplified as a single one, since I would only need to store data just once (and there would be better performance) and I would store less data because of the columns subset. But I am concerned that the single step would result in a merged package that has too many responsibilities: validation and tree preparation.

What would be the best option in terms of best practice of software architecture?

Best Answer

Do nothing, if you see no performance problems.

Otherwise make parser return Iterator instead of List, that way, you will only need enough memory to store a single line at cost of complicating parser's lifetime management.

Related Solutions

Java Strategy Pattern – Generic File Parser Design

I have a couple of concerns:

I would make sure you actually need a generic design before implementing one. Are you sure you're going to need file types other than XML? If not, why code for them? If you eventually need it, you can retrofit your code at that point. It won't take much longer, you will probably have other requirements that will make the code look different than what you're currently proposing, and you will probably never need to write it anyway. As they say, YAGNI (You Ain't Gonna Need It).
If you do actually need a generic design, and you're quite sure of this, then I'd say that Parser<T> is basically sound. I do see two potential problems: (1) it assumes file input -- what if you're trying to parse a JSON stream that you retrieved from an HTTP response, for example? and (2) it doesn't necessarily provide much value except as part of some larger generic framework where you have lots of different types of parsers for lots of different types of data. But I'm not convinced you need any such large generic framework. You just have a very simple, concrete use case right now, as far as I can tell: parse an XML file into a list of ProductDatas.
It's almost never a good idea to swallow exceptions as you are doing in ProductDataXmlParser. I would convert it to some sort of RuntimeException instead.

Java – Data Aggregation of CSV Files

Since the key-fileds are always the first colums i would sort the source rows (whithout the header) by the keys and then advance in both files line by line similar to Merge_algorithm used in Mergesort.

But instead of sorting the 2 csv-lists to one you compute the sum of elements that are in both list. Elements that are in one list only are simply copied.

The algorithm looks similar to this:

While (NOT EndOfFile(left-item) AND NOT EndOfFile(right-item))
  if (EndOfFile(right-item) OR left-item.key < right-item.key) store(left-item); advance left-item;
  if (EndOfFile(left-item) OR left-item.key > right-item.key) store(right-item); advance right-item;
  if (left-item.key = right-item.key) store(sum(right-item, left-item)); advance right-item; advance left-item

Best Answer

Related Solutions

Java Strategy Pattern – Generic File Parser Design

Java – Data Aggregation of CSV Files

Related Topic