The leadership variable in AI productivity

We’re seeing something strange in the data on AI adoption.

The industry data on GenAI looks... underwhelming. Single-digit improvements in code quality, documentation, complexity. If you just look at averages, you'd wonder what all the fuss is about.

I've been digging into the company-level data, and it tells a different story.

The averages are hiding extreme variance. Some organizations are seeing 20% gains in velocity and change confidence. Others, running the same tools, are seeing a 20% drop in productivity and shipping more defects than before.

Per-company Change Confidence impact with AI adoption. Each bar represents a company. — Source: DX AI Strategy Playbook, 2025

The tools are the same. The models are the same. The difference is the system they're deployed into, and that's a leadership problem.

In Context is All You Need, I wrote about how tools fail without the right information architecture. The context that matters goes beyond the technical. It's cultural. How teams share knowledge, whether people feel safe to experiment, what leaders actually reward.

You can't purchase productivity. You build the environment where it becomes possible.

The Illusion of Adoption

Most organizations measure AI success with a single, flawed metric: Acceptance Rate. I've seen this play out: leadership mandates "X% AI adoption by Q3," assumes productivity will follow, and moves on to the next initiative. Classic case of confusing a proxy metric with the actual goal.

As Will Larson notes in An Elegant Puzzle, when you measure engineering by volume or activity, you incentivize behavior that often degrades system health. When you grade engineers on acceptance rate, you invoke Goodhart's Law. Rational engineers will optimize for the metric. They hit tab. They accept the boilerplate. They commit the diff. The dashboard turns green, but the codebase bloats.

The danger is that AI creates a "productivity sugar high." It drastically reduces the friction of generating code, inducing a flow state that feels like speed. But in a software system, writing code is rarely the constraint. Understanding and maintaining it is. By optimizing for generation speed, you risk flooding the system with low-quality volume, shifting the bottleneck to your scarcest resource: senior engineering attention during code review.

Fixing the Wrong Bottleneck

Eli Goldratt’s The Goal taught us a fundamental truth decades ago: An hour saved on something that isn't the bottleneck is worthless.

In software development, typing code is rarely the bottleneck.

If your engineers spend 2 hours coding and 6 hours waiting for CI, arguing in meetings, or deciphering ambiguous requirements, then making them type 50% faster achieves nothing. Actually, it might hurt. You're just piling up more unreviewed PRs in front of the bottleneck. This is just queuing theory. Little's Law:

$L = \lambda \times W$

Crank up the arrival rate, keep capacity fixed, and watch your WIP explode.

The Bottleneck Pipe: AI code generation floods a wide pipe, but code review is a narrow constraint where inventory piles up. — An hour saved on something that isn't the bottleneck is worthless.

Morgan Stanley built "DevGen.AI" to analyze legacy code and generate specs. 280,000 hours saved annually.
Spotify used agents to handle 90% of incident alerts and suggest runbooks. MTTR dropped significantly.
Faire deployed "Fairey," an always-on code review agent. 3,000 reviews a week, freeing seniors for architecture.
Zapier built bots for internal admin and onboarding. Ramp-up time went from months to weeks.

Psychological Safety is the Foundation

You can tune your system prompts all day, but the most important "prompt" is the one you give your organization. You can't just mandate tools and ignore the silence.

When engineers go quiet about AI, it's usually because they're smart. They know that if they become 2x as productive, the spreadsheet logic suggests you need half as many of them.

Leaders have to address this directly. Zapier didn't cut headcount when their AI bots reduced onboarding time from 90 days to 2 weeks. They increased hiring. The ROI on a single engineer went up, so they wanted more engineers, not fewer. The goal is throughput, not headcount reduction.

Google's Project Aristotle found that psychological safety was the number one predictor of high-performing teams. This applies doubly to AI adoption. AI won't take their job. An engineer who masters AI will. Your job as a leader is to make sure your team ends up on the winning side of that equation.

The Fear Feedback Loop: Fear leads to hidden usage, which prevents shared learning, causing stagnation, which reinforces fear. — The culture determines the output, not the tool.

When engineers feel safe to experiment, the best use cases emerge organically. A junior engineer figures out a better way to use agents for unit tests, shares it, and the pattern spreads.

Measuring What Actually Matters

W. Edwards Deming famously observed that "90-95% of productivity is determined by the system, not the worker."

If we want to measure AI success, we have to measure system health, not tool usage. No single data source tells the full story. You need to triangulate across three types:

Metric Type	Good For	Not Good For	Challenges
Telemetry (API data)	Measuring impact on developer output	Quantifying ROI; understanding how tools are being used	Limited, possibly inaccurate insight; incomplete story
Experience Sampling (in-context prompts)	Quantifying ROI; identifying best use cases	Collecting large amounts of data at once	Difficult to set up; must run over time
Self-Reported (surveys)	Measuring adoption, satisfaction, perceived productivity	Quantifying ROI directly	Can only run periodically; participation rates and bias matter

Telemetry is seductive but misleading. Accept-vs-suggest was the rage until we realized engineers need to click accept for the API to even know about it. Even then, they might rewrite every line afterward. Experience sampling closes that gap. Add a field to your PR form: "Did you use AI? How much time did it save?" Self-reported surveys track sentiment over time. But they only work with 90%+ participation rates and questions that treat developer experience as a systems problem.

The leading organizations blend all three:

Microsoft (Engineering Thrive^[1]): Tracks adoption^[2] alongside "Bad Developer Days,"^[3] a telemetry-based measure of toil and disruption. Also measures system velocity, developer satisfaction, and change failure rate.
Dropbox (DX Core 4 + DX AI Framework): Combines DAU/WAU with developer CSAT, PR throughput, and % AI code.
Booking.com: Correlates time saved per developer with change failure rate. Tracks % of PRs that are AI-assisted alongside PR throughput.

Everyone tracks adoption. But mature orgs immediately correlate it with velocity and quality metrics. Nobody trusts adoption alone.

The Measurement Framework

Foundational DevEx and DevProd metrics still matter the most. AI metrics measure activity. Core engineering metrics measure outcomes. If your AI adoption is at 80% but your change failure rate is climbing, you have a problem.

DX organizes this into three dimensions, which also represent a maturity curve:

AI Measurement Maturity: Utilization → Impact → Cost — Most orgs are stuck at stage one.

Utilization	Impact	Cost
How much are devs adopting AI?	How is AI impacting productivity?	Is our ROI optimal?
AI tool usage (DAU/WAU)	AI-driven time savings (hours/week)	AI spend (total and per dev)
% of PRs that are AI-assisted	Developer satisfaction	Net time gain per dev
% of committed code that is AI-generated	DX Core 4: PR throughput, code maintainability, change confidence, change fail %	Agent hourly rate (HEH / AI spend)
Tasks assigned to agents	Human-equivalent hours (HEH) by agents

Most organizations start at Utilization. Just figuring out who's using the tech. But you need to mature quickly to Impact by correlating usage with your engineering strategy metrics. Cost optimization comes last. We're 15 years past the cloud hype cycle and still learning to optimize cloud spend. The same learning curve is coming for AI. Horror stories of teams burning $2,000/day on runaway token loops are already circulating.

The Leadership Job

This technology is weird. It's probabilistic, not deterministic. It hallucinates. It's brilliant one minute and dumb the next.

Managing it requires a different kind of engineering leadership. Curating the environment matters more than mandating the tools.

Traditional value distribution vs AI-augmented value distribution — AI doesn't eliminate engineering work. It shifts where the value lives.

AI commoditizes the bottom of the stack. Boilerplate, syntax, typing - the low-cognition work that used to consume 70% of engineering time shrinks to 10%. But value doesn't disappear. It moves up-stack.

The new distribution: 40% high-cognition work (system thinking, complex problem solving), 50% judgment and risk management (reviewing AI output, security, architecture decisions). Leaders who understand this shift invest in the top of the pyramid, not the bottom.

Buying Copilot seats is administrative. Your real job is to:

Hunt the bottleneck: Don't pave the cow path. Find where the work actually sticks.
Create safety: Make it safe to experiment and safe to fail.
Measure the system: Ignore the hype, watch your DORA metrics.
Demand human-in-the-loop: The AI is the junior engineer. Your team is the senior engineer reviewing the work.

The variance we see in the data, that +20% to -20% swing, is a leadership report card. The tech is neutral. The outcome depends on us.