Before We Teach AI

I recently attended another data leadership networking event, with presentations on deploying AI and governing data in an increasingly AI driven world. Both were thoughtful and grounded in experience, but the part I kept returning to afterwards was a conversation during the break.

Someone suggested that organisations should begin thinking about AI agents in much the same way they think about employees. Agents would need training, governance, development and evaluation throughout their working lives. I understood the appeal of the analogy, and it seemed to resonate with others in the room. What surprised me was where it took my thinking. Rather than considering how we might train machines, I found myself reflecting on how organisations attempt to develop their people, and how often those attempts fail to produce lasting change.

Training isn’t the same as learning

Most organisations invest considerable time and money in learning and development. There are mandatory compliance courses, cyber security awareness sessions, leadership programmes, technical certifications and extensive libraries of online material. Some of this is genuinely valuable. But I suspect most of us have also completed training because we were required to, acknowledged policies we didn’t fully understand or returned from a course to find that our day to day behaviour remained much the same.

Providing information is not the same as developing judgement. People usually become more capable through a combination of experience, feedback, reflection and opportunities to apply what they’ve learned. They encounter situations that don’t quite match the textbook, make mistakes, discuss what happened and adjust their approach. Organisations can support that process, but they cannot replace it simply by recording that a training module has been completed.

Which makes the analogy with AI agents a little less reassuring than it first appears. If we genuinely trained agents the way some organisations train employees, we might give them a lengthy induction, ask them to confirm they’d read the guidance and review their performance against objectives that haven’t been revisited for a year. It would be a familiar process, but not necessarily an effective one.

The more I thought about it, the less convinced I became that the important question was how we teach AI. Perhaps we should first be asking how well our organisations learn.

The knowledge an organisation actually has

An AI system introduced into a business doesn’t encounter a neat, complete representation of how that business works. It encounters the information we’ve recorded: policies, documents, data, definitions and the processes we’ve chosen to formalise. It may also encounter contradictions that people have been quietly working around for years.

Imagine an organisation deploying an assistant to answer questions from internal policy documents. The intention is sensible. People spend time searching for guidance, so why not make it easier to find? But suppose the system retrieves three versions of the same policy, each slightly different, while the process staff actually follow has never been properly documented. The assistant may struggle to provide a reliable answer, not because it failed to retrieve information, but because the organisation hasn’t established which information is authoritative.

Experienced colleagues often bridge these gaps without drawing attention to them. They know which document is current, which rule has an exception and who to speak to when the written process doesn’t fit the circumstances. That informal knowledge allows organisations to function, but it also disguises weaknesses in the systems intended to support them.

AI can make those weaknesses more visible. It can help discover inconsistencies and identify where knowledge is missing, but it cannot independently decide what an organisation’s policies ought to be. Someone still has to establish the correct interpretation, take responsibility for it and make sure the system uses it consistently.

Who notices when the answer is wrong?

There is another part of the learning analogy that deserves attention. People can learn from experience, but a deployed AI application doesn’t automatically improve its underlying model because somebody was dissatisfied with an answer. Improvement usually depends on feedback being captured, investigated and translated into changes to the knowledge, instructions, tools or model supporting the application.

That raises a practical question: will we notice when the system gets something wrong?

Obvious mistakes are relatively easy to challenge. The more difficult cases are answers that sound reasonable, fit the expected format and arrive with the same confidence as the correct ones. As a system performs well over time, people may become less inclined to check it. Scrutiny starts to look like an unnecessary cost, particularly when the technology was introduced to reduce the time people spend doing the work manually.

The uncomfortable possibility is that the colleagues best equipped to recognise a subtle error are also the people whose expertise we become tempted to remove from the process. Someone who has spent years doing a particular job may notice that a recommendation doesn’t quite fit the circumstances before they can explain exactly why. That judgement is difficult to encode, but it can be essential to recognising when a system has reached the limits of what it knows.

None of this is an argument against AI or automation. It is an argument for designing feedback, review and accountability into the way these systems are used. We need to know which decisions require human oversight, how concerns are raised and who is responsible for improving the system when a problem is identified.

Before we teach the machines

I’ve become increasingly convinced that many of the difficulties organisations encounter with AI are not entirely new. Poor documentation, inconsistent processes, unclear ownership and fragmented knowledge existed long before large language models became widely available. Capable people have often compensated for those weaknesses through experience and informal relationships, which made the problems easier to live with and harder to see.

AI changes the consequences of leaving those gaps unresolved, particularly when systems are expected to make recommendations or take action at scale. But it also offers an opportunity to examine the way organisations develop, maintain and challenge their own knowledge.

Perhaps the most useful lesson from the employee analogy is not that AI agents need to be treated like people. It’s that organisations need to become better at learning from what happens when both people and systems are put to work. Until we take that responsibility seriously, improving the technology alone is unlikely to deliver the kind of intelligence we’re hoping for.