Most AI oversight training asks directors whether the model is biased. That is the wrong first question. The first question is whether the number in front of you was measured at all. This is a working session built on real cases where an instrument reported a confidently wrong answer and every automated check stayed green. The complete facilitator kit is published below, free, because most boards can run it themselves.
AI-derived figures now reach the board through vendor reports, dashboards and management commentary. Most arrive with no stated sample, no variance and no named instrument. The failure mode is rarely a dramatic one: a system produces a number, a process validates it, the validation passes, and the number is wrong anyway, because the check tested something adjacent to the thing that broke.
This session is deliberately not a survey of AI risk frameworks, and it is not a list of questions to ask. The published board AI material is almost entirely question lists, which give a board no way to discover it is bad at this. Here each case hands the room a figure in the form it would actually arrive, lets them decide it is fine, and then shows them it was not. An exercise with a failure state is what makes the material work without a facilitator in the room.
Sort any figure in a board pack into measured, captured or relayed, and say which one governs a decision and which does not.
Ask the two questions that make an AI-generated metric meaningful: how many runs behind it, and what the spread was.
Recognise a validation that tested something adjacent to the break, and a gate that has never failed because it has never worked.
Leave with a short question set that works on any AI-derived figure, from any vendor, without technical knowledge.
Every case is a real measurement from a live production system, anonymized. Participants get the reported figure first, in the form a board would receive it, and work out what question breaks it before the cause is revealed. Three examples, without the reveals:
The reveals are in the kit, along with five more cases, including two where nothing malfunctioned at all and no report could have shown the defect.
A half day with a reception, or a two-hour cut for a committee slot.
12 to 30 people. Board education session, audit or risk committee deep dive, or paired with a keynote.
Laptops optional. The exercises work on paper, because the discipline has to survive outside the room.
Participants may bring one page from a real board pack, redacted however they wish. Nothing is required.
Two variables change which parts you run: whether AI is already making decisions inside the business or mostly producing information about it, and how many of the figures reaching your board originate outside the company. The kit says which blocks to pick for each.
The full facilitator material is a public page: eight cases with their reveals, four tabletop simulations, the provenance exercise, block timings, an eleven-question card to print for each director, and notes on running each block. A corporate secretary, committee chair or general counsel can run it cold. Nothing to license, no attribution required, no form to fill in.
Most boards will not need it, which is why the kit is free rather than gated. If you would rather have the session facilitated, or want the simulations rebuilt around decisions your board is actually facing, that can be arranged.