The software development department of my company is facing with the problem
that data migrations are considered as potentially dangerous, especially for my managers.
The background is that our customers are using a large amount of data with poor quality.
The reasons for this is only partially related to our software quality, but rather to the history of the data:
Most of them have been migrated from predecessor systems, some bugs caused (mostly business) inconsistencies in the data records or misentries by accident on the customer's side (which our software allowed by error).
The most important counter-arguments from my managers are that faulty data may turn into even worse data, the data troubles may awake some managers at the customer and some processes on the customer's side may not work anymore because their processes somewhat adapted to our system.
Personally, I consider data migrations as an integral part of the software development and that data migration can been seen to data what refactoring is to code.
I think that data migration is an essential for creating software that evolves.
Without it, we would have to create painful software which somewhat works around a bad data structure.
I am asking you:
- What are your thoughts to data migration, especially for the real life cases and not only from a developer's perspecticve?
- Do you have any arguments against my managers opinions?
- How does your company deal with data migrations and the difficulties caused by them?
- Any other interesting thoughts which belongs to this topics?
Best Answer
Data migrations are my bread and butter and data cleansing is indeed a hugely important matter. One strategy we use do migrate 100% of our customer's data is asymptotic data cleansing pre-migration tools.
This means developing tens of data-sanity checks (mostly sql queries).
Exchanging cleansing tools with the customer (since that's his data, we design the patching utilities, he validates them and executes them).
Refining the tool over iterations and reaching KPI-backed measurable quality asap.
Checking data consistency after the migration has happenned. This helps to make GO/NOGO decision on D-Day.
In the end a data migration is an immensely beneficial exercise that has to happen after 3 to 5 years.
It allows to boost the platform's ability to support business.
It allows to streamline the database.
It prepares the IT platform for next generation business tools (ESB/EAI, Portals, Self-Care platforms, reporting and data mining, you name it).
It reorganises DIY data flows between platforms that have accumulated over the years in a quick and dirty "temporary" way to fulfil "urgent requirements".
Above all it empowers the IT production team who come to know their platform better and foster 'can-do' attitudes. These kinds of benefits are difficult to measure but when you come to know many clients, this consideration becomes obvious. Companies shying away from migrations remain in the following tier, bold ones lead the pack.
It's a little bit like when the basement of your house becomes cluttered with lumber. One morning, you have to take everything out and put back only the things you need and throw the rest away. After that, you can use your basement again ;-)
Another fundamental consideration is that nowadays, customer expectations are always on the move, as in "customers are always more demanding". So that there will always be a significant proportion of a given company's competitors on the lookout for these new trends with the obvious intent to increase their market share. The way they will do so is by adapting their offering to stick to the trend or even drive the trends, and that entails constant business re-engineering. If your IT platform is too rigid, it will be a drag on your own aptitude to spouse or precede the market trends on your own side and, ultimately to maintain your own market share. In other words, in a moving market inertia is a recipe for irrelevance.
In contrast, a data migration to a newer system will roll out a more modern and more versatile productivity tool, making the best of newer technologies, more attractive to employees and this in turn, will contribute to support or even lead the company's internal innovation process, thereby securing or increasing its relative market share.
The considerations above actually answer only half of the question asked in the title "Data Migration - dangerous or essential". Yes Data Migrations are essential, but are they also dangerous ? On this account, many things in IT are dangerous then. By definition, anything where the stakes are high is dangerous; especially if you do not take the matter seriously. But this is actually the most common pattern in IT. Not taking data-centres or high availability or disaster tolerance seriously is dangerous.
Does that mean that today's companies should opt out of these pillars of today's Information Technology landscape ? Surely not !
To make your point jokingly, you could argue that "Flying is dangerous if you don't use a plane made by professionals". It's the same for Data Migrations. When executed and conducted by professionals, it is no more dangerous than flying in a well designed and well operated plane. And ROI is in the same proportion compared to terrestrial means of transport.
When entrusted to professionals, most migrations are well controlled successes and the failure+abandon rate is extremely low.
Your managers should be led to ask themselves "Whilst most companies go through Data Migration projects successfully what would make our company so different that it would instead experience a failure ? and can it fare well without one ?"