Are We Measuring AI-Assisted Coding Correctly?

In this post, Greg Wilson lays out twelve traps teams can fall into when they try to judge whether AI coding assistants are actually helping.

Many of the usual productivity metrics - that teams usually lean on - don't actually tell you much. Writing more lines of code, making more commits, creating pull requests, closing more tickets, seeing higher adoption numbers, or simply asking developers if they feel more productive can all increase without real productivity improving. In his post, Greg Wilson walks through why each of these measures can fail.

The other half of his argument is about what these simple metrics miss: extra time spent reviewing AI-generated code, security problems, technical debt, and the additional review burden placed on other developers. He also points to familiar research pitfalls, such as having no control group, comparing self-selected early adopters with everyone else, measuring only during the initial novelty period, and comparing AI assistance to doing nothing rather than to the tools and resources developers would actually use. For RSEs and software teams trying to understand whether AI coding tools are worth the cost, this article gives readers a useful checklist for asking better questions and measuring value rather than simply measuring activity.