AI Productivity Is Hard to Measure When the Work Itself Changes

AI productivity becomes harder to measure when AI changes more than the speed of a familiar task. It can make an old activity faster, encourage people to attempt work they previously skipped, or shift time into checking, coordination and higher-value decisions. A single “hours saved” number can miss all three changes.

The practical thesis is simple: measure stable work, the new task mix and reviewed value separately. Otherwise a team may celebrate a fast draft that created extra review work, or overlook a valuable new activity because it has no clean pre-AI baseline.

Why one AI productivity number can tell three different stories

METR’s May 2026 note distinguishes uplift on old tasks, uplift on new tasks and uplift in value. The distinction matters because people reallocate time towards tasks for which AI is especially useful. Measuring only familiar work may miss that substitution; measuring only the new work may overstate value because some tasks were attempted precisely because AI made them cheap.

Measurement also changes the behaviour being measured. METR’s February 2026 developer study update reported that wider AI adoption made its task-level experiment harder to interpret: some developers did not want to work without AI, some withheld tasks they expected AI to help most, and parallel agents complicated time tracking. The lesson is not that productivity cannot be measured. It is that task selection and comparison conditions must be recorded.

Use the MIX check before claiming productivity

MIX is a management check, not a universal economic model. It helps a team describe its own change without pretending that one local workflow proves economy-wide productivity. Keep the measures separate when they answer different decisions.

Worked example: the weekly customer brief

Imagine a customer-success team that used to spend three hours producing one weekly brief. With AI, the first draft takes forty minutes. The team also starts adding competitor news, three audience versions and a risk summary that did not exist before. Review grows from twenty minutes to ninety because every new claim needs checking.

The result may show that the original brief genuinely became faster, the risk summary added value, and two audience variants were unnecessary. That is more useful than one blended percentage because it tells the team what to keep.

Protect the baseline before the workflow moves

The UK Government’s updated guidance on evaluating AI interventions recommends defining the comparison and collecting baseline evidence early. It also notes that AI may enable new activities and change complex work, which makes the original baseline harder to apply. Record business-as-usual while it still exists, then document why a changed task mix needs a different measure.

Connect measurement to workflow design

Use the workflow friction test to decide which checks must remain. Map how AI changes the task mix, then decide what saved time becomes. These are different decisions, and a useful measurement design keeps them visible.

AI productivity is not unknowable. It is conditional on the task, the comparison and the outcome that matters. Measure a stable reference, make task substitution visible and judge the reviewed result.