Have you ever wondered if robots are actually making coders faster? Everyone is talking about Artificial Intelligence, but how do we know if it is truly helping or just creating a mess? Let’s dive into the world of software metrics and learn why numbers can sometimes tell big lies.
In our lesson today, we must first understand that not all measurements are created equal. In the world of software engineering, we categorize metrics into three levels. First, we have Activity Metrics, which measure behaviors. These are very easy to track but have low value. For example, counting how many hours someone sits at their desk doesn’t tell you if they built anything useful. Second, we have Output Metrics. These measure deliverables, like how many lines of code were written. These are slightly more useful but still don’t show the full picture. Finally, we have Outcome Metrics. These measure system changes and actual value. They are the hardest to measure but provide the highest information value because they tell us if the software actually made the world better or the business more successful.
Lately, there has been a massive “hype train” regarding Generative AI (Gen AI). Many companies are acting like we are in a stock market bubble, rushing to prove that AI is making them more productive. This has led to a return of an old, flawed idea called Taylorism. This is a management theory from the early 1900s that treats people like parts of a machine, measuring every tiny movement. In software, this looks like managers trying to measure every line of code or every “Pull Request” (PR). However, as a teacher, I want you to remember a very important rule called Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”
Imagine a nail factory. If the manager tells the workers they will be paid based on the number of nails they make, the workers will produce thousands of tiny, useless nails. If the manager changes the rule and pays them based on the weight of the nails, the workers will make a few giant, heavy nails that are also useless. In both cases, the workers changed their behavior to meet the target, but the factory didn’t actually produce better products. This is exactly what happens when we try to measure AI productivity using the wrong numbers.
Let’s look at a recent report from a vendor called DX, which studied over 135,000 developers. They shared four major headlines that might sound impressive at first, but we need to look closer. Their first claim is that AI adoption is over 90%. This is an Activity Metric. Just because someone uses a tool doesn’t mean it’s a “core part of the engineering strategy.” The second claim is that developers save 3.6 hours per week. This is based on “self-reported” data, which is subjective. People often say what they think their bosses want to hear. The third claim is that 22% of code is AI-authored. But what counts as “AI-authored”? If an AI writes a method and a human changes three words, is it still AI code? It is very hard to measure this accurately.
The fourth headline is the most dangerous: “Daily AI users ship 60% more Pull Requests.” This is an Output Metric. Shipping more PRs does not mean the team is delivering value faster. If a developer sends 100 PRs in a day but the software still only gets updated once a month because the testing team is slow, then those 60% more PRs are just “noise” in the system. In fact, it might even make things worse by creating more bugs for others to fix.
So, how should we actually measure success? We should look at the DORA (DevOps Research and Assessment) metrics, which were popularized in the book Accelerate. There are four key metrics that actually predict business success:
- Deployment Frequency: How often does the team successfully release code to the real world?
- Lead Time for Changes: How long does it take for a line of code to go from an idea to a finished product used by a customer?
- Change Failure Rate: What percentage of releases lead to a failure in the system?
- Time to Restore Service: How long does it take to fix a problem when the system goes down?
If your team uses AI and these four numbers improve together, then the AI is actually helping. If you are shipping more code but your failure rate is going up, your AI is just helping you make mistakes faster. Always remember that a team is the true unit of delivery, not an individual person. Critical thinking is your best tool. Don’t let flashy statistics fool you; always look for the outcome, not just the activity.
To wrap up, do not let flashy AI statistics fool you. While tools like Gen AI are exciting, real success is measured by how quickly and reliably a team delivers value to users. Focus on the DORA metrics—Deployment Frequency and Lead Time—rather than counting lines of code or pull requests. I recommend reading the Accelerate book to understand the science behind high-performing teams. Always remember that a team is the true unit of delivery, not an individual coder with an AI assistant. Keep questioning the data, stay curious, and keep building great things that actually solve problems.