Someone puts the number on a slide: we rolled out Copilot to 500 developers. The room nods. Nobody asks what those 500 people now do differently, because the question sounds ungrateful, and the number looks like a result.
If the only thing that has moved is how fast the code gets written, you’ve just made the organization faster at building the wrong thing.
What I would measure
Start with three questions, asked out loud in the next leadership meeting. Which decision do we make differently than three months ago? Which meeting has disappeared? What have we stopped building?
All three are easy to talk around if there are no numbers next to them. So find two.
If you run Granola or Lyt, the quarter’s meeting transcripts are already sitting there. Give them to Claude and ask it to find how long it took from the question being raised to the decision being made. That’s your real baseline.
Pull the time from “code ready” to “in production” out of Linear or Jira. Measure at the review end, not the build end. If only build time has dropped, you’ve moved the bottleneck, not removed it.
Put the decision time and the review time on the same slide as the license count. Side by side, no commentary. The slide runs the meeting for you.
And no, you don’t need to buy a new measurement tool for this. Linear and Jira already have the numbers.
Tools are the last of five layers, the least interesting one, and the one most people start with. It’s also the easiest layer to put on a purchase order. If culture, structure, skills and processes haven’t moved, all you get is faster bad output.
Your tip this week
Find the latest AI adoption number someone has shown off at your company. Write one sentence underneath it: “The decision we make differently because of this is ___.”
If you can’t fill it in, you’ve found your next project.
PS
The playbook with the five layers is freely available at martinibsen.dk/en/playbook. Layer 5, the tools, is the shortest chapter. That’s not a mistake.
Best, Martin