Judgement: The skill AI can't replace
Free masterclass page 2 of 3
Judgement traps, two types.
You might wonder why this masterclass doesn't focus on cognitive biases, the way most decision courses do.
There's a reason.
Biases are not defects. They are features of a brain that was built for speed. Fast, automatic, effortless thinking is what allowed our ancestors to spot a predator in the grass and tell a poisonous berry from a safe one. It still helps you today, navigating a crowded airport, reading a client's tone in the first thirty seconds of a call, deciding what to eat without paralysing yourself with analysis. The system works. The problem is that it now operates in a world of overwhelming complexity, information overload, and consequential choices it was never designed for. We trip not because the machinery is broken, but because the terrain has changed.
And this leads to the real issue: knowing about these traps is rarely enough to stop them.
Takeaway #2: Awareness of biases is not enough
I teach this for a living. I've spent fifteen years studying cognitive biases, training professionals to recognise them, building frameworks to address them. And not long ago, I caught myself deep inside a trap I warn others about.
I had become convinced that a particular type of client was the right strategic direction for Meta-decisions. The early signals were ambiguous, some conversations went well, others went nowhere, but I kept reading them as confirmation. A polite "let's stay in touch" became evidence of interest. A rejected proposal was bad timing. When a colleague pointed out that the clients actually signing up came from a completely different profile, I found reasons why that was temporary.
Weeks later, when I stepped back and looked at the pattern as a whole, only then did I recognise what had been happening. I had wanted that strategic direction to be right. And because I wanted it, my reasoning had bent itself around that conclusion, selecting the evidence that supported it, discounting what didn't, all below the level of conscious awareness. I wasn't lying to myself. I was reasoning carefully. The reasoning just happened to be in service of a conclusion I'd already reached.
This is motivated reasoning and it breeds a family of traps. Knowing its name did not protect me from it.
Now consider what motivated reasoning looks like when you have AI at your fingertips. You feel a hunch. You ask the AI. The AI, which has no judgement about your situation, only pattern-matching across its training data, surfaces three articles, two case studies, and a framework that all support your hunch. You feel vindicated. You were already wrong before you asked; now you're wrong with citations.
The question becomes: what would have caught me?
Not more knowledge. Not awareness. What I needed were changes to my process, my environment, and my habits, changes that would have made the trap harder, much harder, to fall into. That's what the rest of this masterclass is about.
Research in decision science draws a useful distinction between two types of traps: errors that happen because we use the wrong mental strategy (strategy-based errors), and errors that run deeper, rooted in fast thinking, pattern seeking or driven by our motivations (automatic associations and motivated reasoning).
Takeaway #3: Strategy-based errors can be overcome
These are structural errors in how we reason, caused by using the wrong mental model or the wrong logic. There are several such errors, each a place where thinking reliably goes wrong.
The good news: there are direct ways to fix them. Point out the mistake, give the right framework, and the fix holds. Judgement is repaired. In fact, with some adjustments in how we were taught to think from a young age, many could have been avoided entirely.
Below is a set of nine. Each has a specific cognitive or behavioural fix, often a single question you need to ask yourself. As you go through them, notice how quickly the fix becomes obvious once the trap is named.
These nine traps and their fixes are available as a set of digital cards, in printable format, that you can keep at your desk or share with your team. Email and I will send them to you.
Your judgement trap cards →
These errors don't disappear when AI enters the picture. They get a new delivery mechanism. An AI tool running a pilot analysis on a small dataset will present the results with the same false confidence that insensitivity to sample size produces in your own thinking. A detailed, polished AI output makes base rate neglect easier, not harder, because the specifics are so compelling. An AI-generated scenario triggers the conjunction fallacy just as effectively as a human pitch. The traps are the same. The packaging is better.
Unless you prompt your AI thinking partner to catch them. You can ask AI to check your reasoning for specific traps by name: "Am I considering sunk cost?" "What kind of base rate should I be looking at?" "Could this be a conjunction fallacy?" The quality of the answer depends on how specifically you ask. The trap is not asking at all, because the AI's output looks so polished you assume your reasoning has already been checked. It hasn't. Unless you ask.
Takeaway #4: Automatic associations and motivated reasoning require more than one-to-one fixes
The second type is more stubborn.
Knowing about the automatic associations our minds make, or how motivated reasoning affects us, doesn't stop us from falling into them. Motivated reasoning is essentially the psychological mechanism behind motivational biases: distortions in judgement caused by a desire for a particular outcome, wanting something to be true, or wanting to avoid something being true. It operates throughout the thinking process, shaping which information gets noticed, how it gets weighted, which conclusions feel satisfying. Because it's not always conscious, you cannot easily fix a process you don't know you're running.
AI works differently here. These traps don't come from a lack of information or processing power. They come from automatic thinking, motivation, emotion, and social dynamics. No amount of data or analysis fixes a problem you don't know you have. That said, AI is getting better at noticing patterns in your reasoning that you might miss, if you give it enough context about how you're thinking and ask it to look for specific traps. The limitation isn't the technology. It's that the person most affected by the trap is usually the last to think of asking.
My story above was motivated reasoning at work. I wanted the strategic direction to succeed. So the evidence that said otherwise got filtered, minimised, reinterpreted. Not because I chose to ignore it, but because the motivation was doing the filtering before my conscious reasoning even got involved.
There are more than 200 documented traps like this. You could learn every one of them and still fall into the next one that matters. That's the point: awareness is a starting point but it's not enough.
Here are nine such traps, organised by the level in which they first appear (individual: how you reason, group setting: how you reason with others, and setting: how your environment lets you reason).
As you read, notice something: recognising the pattern won't be enough to stop it. These operate below the level where awareness helps.
You'll notice this list doesn't come with direct fixes. That's not an oversight. These traps don't respond to single-tool solutions the way the strategy-based ones do. But they do respond to something.
What follows are three organisations that faced these traps at scale. Each story reveals a different layer of the same truth: first, what happens when individual judgement fails under pressure; then, what happens when a team's collective judgement fails; and finally, what happens when an entire organisation's environment is the thing suppressing good judgement.
Nineteen items
A surgeon is about to make the first incision. The patient is anaesthetised, the team is assembled, the lights are bright, the instruments laid out. Everyone in the room is highly trained, highly focused, and highly confident.
No one has confirmed which leg they're operating on.
Skilled professionals working in complex environments under time pressure reliably make errors of omission, not because they are incompetent but because they are human. Wrong-site surgery happens in real hospitals, performed by competent surgeons, more often than anyone in the profession would like to admit. The cause is never a lack of knowledge. It is a process that relies on individual memory when it should rely on structure.
In 2007, surgeon Atul Gawande and a team at the World Health Organisation set out to measure how often things actually went wrong. Across eight hospitals in eight countries, major complications occurred in 11% of surgical procedures. One in nine patients.
Most individual surgeons estimated their own complication rate as far below that number. This is overconfidence in its most dangerous form: not arrogance, but the quiet certainty that expertise produces. That certainty is precisely what allows the error to go unchecked.
The errors Gawande's team found were not failures of skill. They were failures of omission: antibiotics not given before incision, allergies not confirmed, the surgical team not introduced to each other by name. Each omission individually trivial. Collectively, they were producing harm in one of every nine surgeries. And nobody noticed the drift, because each time a shortcut produced no visible harm, it felt more acceptable the next time.
The surgeons' instinctive response was revealing. Most framed the question as: am I skilled enough to not need a checklist? Put that way, the answer was obviously yes. And the checklist was impossible.
Gawande reframed it. The issue was not whether any individual surgeon was good enough. It was whether the process surrounding the surgery was reliable enough to catch the errors that any individual, however skilled, would inevitably make under pressure. Not a question about competence. A question about system design.
With that reframe, the checklist was embedded as a process tool, the same kind aviation had used for decades. Not because pilots are incompetent. Because the consequences of omission are too high to leave to individual memory.
The checklist itself was almost absurdly simple. Nineteen items on a single sheet, completed in a few minutes at three points during every operation. Confirm identity. Confirm the site. Confirm the procedure. Has the anaesthesia equipment been checked? Are antibiotics on board? And one item that changed more than any other: does anyone in the room have concerns they haven't voiced?
That last question doesn't rely on a junior nurse finding the courage to challenge the senior surgeon. It builds the invitation into the process. The hierarchy is still there. But the checklist gives everyone in the room explicit permission to speak.
The results, published in the New England Journal of Medicine in 2009, were unambiguous. Across 7,688 patients in eight countries, complications fell from 11% to 7%. Deaths fell from 1.5% to 0.8%, a reduction of more than 40%. The results held in high-income and low-income settings alike.
Later research revealed that the checklist's effectiveness depends on how it's used. When teams treat it as bureaucratic box-ticking, it loses its power. Gawande acknowledged this: the checklist is not the solution. It's a catalyst for the behavioural and cultural changes that produce the solution.
The same hospitals that adopted Gawande's checklist are now integrating diagnostic algorithms that outperform individual doctors at pattern recognition in imaging, pathology, and differential diagnosis. The technology is impressive. And it is introducing a new form of the same old trap.
When the AI says the scan is clear, the radiologist's vigilance drops. When the algorithm recommends a course of treatment, the surgeon's independent judgement gets quieter. The feeling of reliability is where the next generation of errors live. And when the AI's recommendation appears on screen, who in the room will say "I disagree with the algorithm"? The cost of challenging a machine that is usually right is even higher than the cost of challenging a surgeon who is usually right.
Unless the behavioural and cultural requirements are there.
The checklist principle applies directly: before acting on an AI recommendation, build in a pause. Does anyone in the room see something the algorithm might have missed? What did the AI not have access to? The cultural requirement is the same one Gawande identified: the explicit, structural permission to exercise judgement, unapologetically.
The checklist changed what happened inside a single room: one team, one procedure, one set of judgements made under pressure. But what happens when the problem isn't one person's judgement? What happens when a whole team is too close to its own work to see what's wrong with it?
The screening room
In the middle of production, the senior creative team at Pixar sat in a darkened screening room and watched the latest cut of a film that was not working.
This was not unusual. What was unusual was the film: Toy Story 2, the sequel to the movie that had launched the studio. The cut was broken, the story incoherent, the emotional core missing, the characters flat. The entire production would have to be restarted. The team in the room knew it. The director knew it. But no one inside the project had been able to see it clearly until that moment.
Ed Catmull, Pixar's president, later made a statement that became famous inside the company: "Early on, all our movies suck." It wasn't false modesty. It was a description of how creative work actually develops, and an acknowledgment that the people closest to a project are always the last to see its problems.
The reason is simple and human. A director who has spent eighteen months building a world does not hear feedback the same way an outsider does. Every piece of criticism is filtered through everything they know about why the choices were made, and that knowledge makes it nearly impossible to see the work as the audience will see it. Pixar's extraordinary track record compounds the problem: every previous film went through a broken phase and emerged brilliantly, creating a quiet confidence that this one will too. And groupthink does the rest. The production team shares the same assumptions, the same language, the same references. Doubts about the director's vision feel disloyal to voice. The team converges on "it's coming together," an assessment that reflects social harmony more than honest evaluation.
Pixar's answer to all three is a practice called the Braintrust.
Every few months, a group of senior creative people, directors, writers, and story leads from other projects, gathers to watch the current state of a film in production and give the director candid feedback. Their job is to say what is not working and why. Not to be polite. Not to hedge. Not to suggest solutions (which is important, because solutions carry implicit authority). To diagnose.
The design choice that makes this work is counterintuitive: the Braintrust has no authority. The director doesn't have to follow a single suggestion. This sounds like it would make the feedback toothless. It does the opposite. When feedback comes from someone who can kill your project, you stop listening to the content and start managing the relationship. When feedback comes from a peer with no power over you, the defensiveness drops. You can actually hear what they're saying.
Toy Story 2 was restarted after such a session and became one of Pixar's most acclaimed films. Ratatouille, Inside Out, and numerous others were substantially reshaped by the same process.
What Pixar understood, long before the current moment, is that evaluative judgement, the capacity to look at something and say whether it works, is the faculty most easily dulled by proximity and most easily protected by structure. As AI tools generate more of the raw material in every industry, drafts, analyses, first cuts, the ability to evaluate what they produce, to say "this is wrong" or "this is missing the point," becomes the most critical skill of all. And it is the skill most at risk of atrophying if no one is structurally positioned to exercise it.
The Braintrust solved the problem within a creative team. One group, one project, one set of blind spots corrected by people who weren't too close. But what happens when the trap isn't inside a team? What happens when the entire organisation, its incentives, its culture, its definition of what good looks like, is the problem?
Know-it-all
In 2011, a cartoonist named Manu Cornet posted a series of satirical org charts of the major tech companies. Google's was a tangled but functional web. Apple's was a neat wheel with one man at the centre. Microsoft's was a set of boxes pointing guns at each other.
The cartoon went viral. Inside Microsoft, it stung. Because it was accurate.
Under CEO Steve Ballmer, Microsoft ran a performance review system called stack ranking. Every team, regardless of how well it performed collectively, was required to sort its members into a fixed distribution: a small number rated top performers, a mandatory bottom tier facing demotion or termination. The ratings were relative, not absolute. Your performance was measured against your teammates.
The intended signal was meritocracy. The actual signal was that your colleagues are your competitors. Engineers avoided strong teams because being ranked against excellent colleagues was dangerous. Helping a peer meant helping someone who might outrank you. Sharing knowledge across teams was irrational, because the team next door wasn't an ally but a competitor for the same resources and recognition. Information did not flow between groups because there was no incentive for it to flow and every incentive for it not to.
The system was designed to create a meritocracy. What it created was a war of all against all.
The result was a company with extraordinary technical talent that could not collaborate. Microsoft missed mobile. Windows Phone was a catastrophic failure. It was late to cloud, ceding the lead to Amazon. Between 2000 and 2014, while Apple and Google reshaped the technology industry, Microsoft's stock went essentially nowhere. The technical capabilities were never the problem. The system in which those capabilities operated was the problem.
Satya Nadella became CEO in February 2014. He later described what he inherited with unusual candour: "We'd lost our soul."
His diagnosis was not that Microsoft needed a better strategy. It needed a different decision environment:different conditions for how information was shared, how judgement was exercised, how people related to each other and to their work. Strategy would follow from culture. Culture would not follow from strategy.
The intervention started with a reframe so simple it almost sounded like a platitude. Microsoft needed to shift from a "know-it-all" culture to a "learn-it-all" culture. The phrase came from Carol Dweck's research on growth mindset.
In a know-it-all culture, the highest-status behaviour is demonstrating that you're right. Being right means you're smart. Being smart means you're safe. In this environment, admitting uncertainty is weakness, asking for help is failure, changing your mind is defeat. These are precisely the conditions that prevent an organisation from learning: from updating beliefs in response to evidence, hearing bad news early, catching errors before they compound. These are precisely the conditions under which every judgement trap in this masterclass thrives.
In a learn-it-all culture, the highest-status behaviour is curiosity. Admitting what you don't know is a starting point. Asking questions is valued. Changing your mind in response to evidence is strength, not inconsistency.
Nadella didn't just announce the shift. He modelled it. He listened more than he spoke in meetings. He shared personal stories about mistakes. He asked questions he didn't know the answer to in public, in front of senior leaders, and treated the answers as genuinely informative rather than as challenges to his authority. This was was behavioural modelling from the most visible person in the organisation, repeated consistently over time.
Then he changed the structures. Stack ranking was replaced it with a performance framework that rewarded collaboration, learning, and team impact. The question was no longer "who should be ranked lowest?" but "how did this team create impact together?" The incentive system, which had been actively rewarding the behaviours destroying the company, was realigned with the behaviours the company needed. Microsoft shifted from a Windows-centric structure, where every team defended the franchise, to a functional structure organised around capabilities: cloud, AI, productivity. Teams that had been walled off from each other were reconnected.
The results took time, but when they came, they were extraordinary. Azure became the world's second-largest cloud platform. Microsoft embraced former rivals, making Office available on iOS and Android, partnering with Linux (which Ballmer had once called "a cancer"), becoming one of the largest contributors to open source. The company's market value grew from just over $300 billion to more than $3 trillion within a decade.
The temptation is to tell this as a leadership story about one brilliant CEO. It's more accurate, and more useful, to tell it as a decision environment story. The people inside Microsoft in 2013 were substantially the same people inside Microsoft in 2016. Their individual abilities didn't change. What changed were the conditions under which those abilities were exercised: what was rewarded, what was safe to say, what counted as strength. When the incentives shifted, the behaviour followed. When the behaviour shifted, the decisions improved.
The people didn't change. The conditions around their judgement did.
Microsoft's transformation was, in essence, a shift from a culture that optimised for having answers to one that optimised for asking questions. A know-it-all culture treats certainty as the highest virtue. A learn-it-all culture treats curiosity as the highest virtue. That distinction mattered when the competition was Google and Apple. It matters far more now, when the most powerful tool in your organisation can produce confident, polished, data-supported answers to virtually any question in seconds. An organisation that rewards having answers will defer to AI. An organisation that rewards asking good questions will use it, and then exercise judgement over what it produces. The first outsources the very faculty that makes human leadership valuable. The second strengthens it.
Takeaway #5: Judgement improves with the environment
Notice how, in these three cases, their people didn't get smarter. They didn't change. Their knowledge stayed the same. Their cognitive abilities stayed the same. It was the conditions around their judgement that changed.
This is an important insight because we tend to treat poor judgement as a personal failing. We assume that better decisions require better people. In reality, judgement is shaped as much by the conditions surrounding the mind as by what happens inside it. Time pressure, social dynamics, incentives, information flow, accountability, all of these distort judgement in predictable ways. Change them, and the same people judge differently. A manager who rushes to conclusions makes far better assessments when the process forces a pause. A team prone to groupthink reaches better conclusions when dissent is structurally encouraged.
The goal is not to create smarter people. It is to create conditions in which people can think more clearly.