This post was originally published here.

Technical debt can take many forms as most of you probably know it. Data quality is just one of those forms.

Data quality is a major issue in large organisations, where multiple systems, applications and services interact between themselves, exchanging and altering data. Incoherences will always occur. Either because someone made a mistake, either because there’s an unidentified bug somewhere, either because the system’s architecture isn’t as robust as it should or simply because people prefer to ignore it (this last one happens way more than it should, trust me!). This will contribute for a consistent and sometimes quiet but steady increase of your technical debt.

Don’t let your selves be fooled. It’s easy to start getting reckless regarding data quality, furthermore when you’re working on a data migration project for several months, for example, and you simply start to reach a point when, even though you don’t want to, you start getting tired and making mistakes. Hell, sometimes you’re just having a bad day… It can happen to any of us!

There are several ways to address this issue. Here are some ways:

1. Let’s start by stating the obvious. Make sure your services are the least error-prone possible. This is where all technical debt should start being addressed: before it even has a chance to exist. I know I’m being a dreamer, unrealistic or whatever but in a perfect world, this is how it should work.
2. It’s nice when you have a team dedicated to performing data analysis and comparisons between different systems and enforcing data repairs in order to correct the data; this action should be performed as often as possible on a regular basis.
3. Understanding what’s causing the incoherence; it may be very difficult to achieve it but when you manage to do it, the probability of eradicating it will be extremely high. This all looks great and relatively simple to achieve but then enters another common problem in any organisation: people. We are our worst enemies most of the cases when we end up going in opposite directions when trying to solve some problem we have in common. Fortunately, this doesn’t happen always and as time goes by the tendency it’s becoming quite the opposite, although there’s still a large margin for improvement (there always is!).

The idea of writing this post was a direct result of identifying a data incoherence carried throughout 4 system upgrades and migrations and that finally haunted someone (me in this particular case) more than 16 years later the flawed data was first created. Nightmarish, don’t you think? What’s yours?

Photo credit: T a k (https://www.flickr.com/photos/takashi/18862634/) via VisualHunt / CC BY-NC