Web-development – Should I use BLOB or Tables for storing large data

NoSQL isn't a very well defined term and all the solutions that run under this name have very different features, so a lot may be possible or not depending on what exactly you are planning to do with it.

Basically you could use some of the more general solutions like maybe MongoDB or Cassandra to simply replace your current relational database. In some cases this makes more sense in others less, but it will work once your team got used to it. Certain things will be easier then, others will be more difficult and you must weight those options against each other and decide (which often enough will mean that there are no advantages big enough and the simple fact that everybody in the team feels most comfortable with relationals and SQL will make the decision easy)

Other NoSQL solutions that are more specialised are not really good candidates to replace your relational DB, like graph databases or simple key value stores. So lets from here talk mainly about those databases that are at least to some degree similar to relational databases.

Scenario 1

Where I work we have exactly this scenario, though quite more complex with a lot of different attributes per article. Some of those attributes in hierarchies like Apple -> iPad -> Air.

The data is still stored in a relational database. But: searching this in real time became a pain. With SQL it was slow and code would have been terribly complex. Selects over many tables, with the additional option to exclude certain attributes like "not blue".

In this case Apache Solr or Elastic Search are a solution. Though of course data is duplicated from the relational database.

But from here our experience with this kind of document store showed that it can handle certain problems very well and we will consider to replace part of the existing relational structure with some other kind of storage. So not the whole database where we also store all the transactional data like orders etc, but for example take out all the attribute information which can be handled much better in the aggregate like data structures of NoSQL.

Scenario 2

Difficult to say, since what you describe is most likely only a very small part of your user handling. Having schemaless storage is an advantage with many NoSQL databases. But some relational databases allow to store such data too (as long as you don't need to query it via SQL in most cases).

Cassandra for example would allow you to define column families in such a case, where your first set of attributes would be one such family and the variable attributes another one.

As somebody said: NoSQL is less about storage and more about querying. So the question is what will be the typical use case for those queries.

A typical problem would be the transactional data here. If you want to store orders, one way would be a schema where users and their orders form an aggregate (kind of user document that contains the orders as subdocuments). This would make getting a user together with his orders very simple and fast, but would make it very difficult to retrieve all orders from last month for sales statistics.

Also strengths of NoSQL solutions are that it can be easier to run them on multiple clusters if you have to work with very large datasets.

Conclusion: Both your scenarios could be modelled with certain NoSQL solutions, but I don't think that (assuming they have to run in a larger environment) they really justify a large extra effort in learning, training and implementation and maybe some other additional disadvantages because both are not specific enough to really leverage the strengths of NoSQL. At least not in that simple form you describe it. Things may become very different once some aspects you describe would be very, very prominent in your usage scenario, like in scenario one the attribute data becomes very complex or in scenario two the variable fields become the largest part of data you store with every user.

I'd say 99 times out of 100 the answer is to grit your teeth and re-use the existing COBOL code. It sounds like it might have some fairly complex logic behind the scenes; unless you actually have an up-to-date and accurate spec you might have trouble re-producing the logic in all its nuances. This is bad mojo on a financial application as the bean counters will want bug-for-bug compatibility with the legacy code.

So, as a first stop, I would suggest that you seriously investigate what is needed to wrap your legacy process or integrate it with the front-end. You might take a look at the COBOL-Java integration options available from Micro Focus.

If you really can't re-cycle the existing code then you have a significant problem. While a sproc might well do what you want, you have the problem of making something mimic the exact behaviour of an apparently complex and probably poorly documented COBOL application.

Unless your replacement process is radically slower, wrapping 2000 DMLs in a transaction isn't going to be a major issue. It's possible to wrap entire ETL processes in a transaction and have them roll back on a failure. I have seen an application where it was necessary to do this, although it is unusual. Splitting a financial transaction is a no-no, especially on a double entry system. In almost all cases it should commit or roll back atomically. Stick with the single transaction.

The fact that the COBOL application does computations one DML at a time and uses the results in the next step of the computation does not bode well for conversion to set operations. Be prepared to have to do this row-by-row and find a way to make it run fast. A stored procedure may well be the way to do this. You don't say what DBMS you're using (DB/2 for Z/OS?).

Also, you might have to drop down to XA transactions (java.transactions API) and JDBC, especially if you need to work in the client layer. If you want good performance on 2000 row based operations you might have to go below the framework.