Where People Re Enter the System

Much of the conversation about artificial intelligence has focused on reducing the amount of human involvement required to complete a task. We talk about automation, autonomous agents and systems that can make decisions without waiting for someone to intervene. There are good reasons to pursue those capabilities, particularly where they remove repetitive work or make services more responsive.

What interests me is that, as these systems become more closely connected to real operations, questions about people begin to return. Not because automation has failed, but because decisions still exist within organisations that have responsibilities, relationships and competing priorities.

Responsibility doesn’t disappear

Consider a system that recommends an operational intervention based on a set of agreed rules. The recommendation may be produced automatically, but someone still decided which information mattered, how success should be defined and what kinds of intervention were acceptable. Those choices shape the outcome even if they are no longer visible to the person receiving it.

When an unexpected result occurs, the conversation quickly moves beyond the model. Who approved the rules? Who can change them? Who is responsible for investigating the result? Does the person affected have any way to challenge it? These are questions about organisational design as much as they are about technology.

It’s tempting to think of people as sitting outside an automated process, stepping in only when something goes wrong. In practice, people are present throughout the design, whether or not their involvement is made explicit. Reporting structures influence priorities, ownership determines whether problems are addressed and operational knowledge shapes which exceptions a system can recognise.

Designing for intervention

I don’t think the answer is to insist that every decision returns to a person for approval. That would miss much of the potential value of automation. The more useful question is where human judgement is necessary and how we make those points visible.

Some decisions can be automated within clearly understood limits. Others need review because the consequences are significant or the information is incomplete. In both cases, people need to know what the system has done, why it has done it and how its behaviour can be corrected when circumstances change.

That requires more than a statement that a human remains accountable. It requires practical arrangements for ownership, escalation, feedback and change. If those arrangements are absent, accountability becomes something we invoke after a problem rather than something that helps prevent it.

At MycoFlow Systems, I’m interested in systems that support human judgement rather than pretending that good judgement can be engineered out of the organisation. As AI becomes more capable, the role of people may change considerably. Their responsibility for defining purpose, challenging assumptions and responding to consequences does not simply disappear.