When Data Becomes the Constraint

Data quality has occupied an awkward position in many technology programmes for as long as I’ve worked with data. Everyone recognises that it matters, but it can be difficult to give it the same attention as a new platform, a reporting requirement or an application that promises an immediate improvement. Quality work often becomes something to revisit once the more visible delivery is complete.

Artificial intelligence is making that compromise harder to sustain. When systems use organisational data to generate recommendations or carry out tasks, weaknesses in the underlying information can influence outcomes much more directly than they might in a report that someone reviews periodically.

Quality depends on what we’re trying to do

I think it’s worth being careful with the phrase good quality data. Accuracy, completeness, timeliness and consistency are all important, but none tells us on its own whether information is suitable for a particular purpose.

A dataset might be entirely adequate for understanding broad trends across a service while being unsuitable for deciding what should happen to an individual case. An attribute might be recorded consistently but no longer reflect the way the business operates. A figure might be correct for the end of last month and misleading if treated as a description of the situation today.

These aren’t always problems that can be solved by cleaning the data. Sometimes they require a better understanding of what was captured, why it was captured and what conclusions it can reasonably support. That is why I see quality as inseparable from business meaning.

Knowing when something changes

As systems become more dependent on data, it also becomes important to understand how that data behaves over time. If an output changes unexpectedly, can we trace the records and transformations that influenced it? If a source stops updating, how quickly will anyone notice? If a business definition changes, can we identify the models and applications that rely on the previous version?

This is where observability becomes more than a technical concern about whether a pipeline completed successfully. A pipeline can run without errors while delivering information that is incomplete, stale or no longer appropriate for the decision it supports. We need ways of detecting those conditions as well as the more obvious failures.

There is a temptation to describe these challenges as new obstacles introduced by AI. In reality, many organisations have lived with them for years. The difference is that automated systems may act on information before a person has an opportunity to notice the problem.

For MycoFlow Systems, I see the practical work as connecting quality checks, lineage, ownership and business definitions to the places where information is actually used. The objective isn’t perfect data in the abstract. It’s enough understanding and visibility to know when information can be trusted, when it needs to be questioned and what to do when it falls short.