Amazon admits AI metrics are flawed but says they still count

“This effect is driven both by increasing productivity and delivery on teams that have embraced AI the most (we call them frontier teams), and that teams and individuals who are not adopting AI or increasing output metrics are falling behind,” the memo read.

It concluded that using these metrics with care and judgement was appropriate when engineers think about their careers and their organisation’s performance. Managers were advised to focus on the 10th percentile of AI adoption, to understand why those teams are not using AI effectively, and the 90th percentile, to check whether heavy usage is producing matching gains in delivery.

The memo does not say whether the metrics are used in formal performance reviews or promotion decisions.

It also told engineers that if current AI tools do not benefit their work, this should be treated as a finding rather than a failing, and that they should document the reasons and raise them with their manager.

“An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for,” the memo said.