Survey: Need for Speed Tops All Other AI Coding Metrics

A survey of 428 engineering and technology leaders published today by Uplevel, a provider of a software engineering intelligence platform, finds nearly all (96%) nearly all believe artificial intelligence (AI) is improving software development but only 4% have adopted new metrics to measure value in the AI era.

In fact, only 29% primarily evaluate the value of AI coding based on business outcomes and only 40% use controlled comparisons to isolate AI’s impact from other factors, the survey finds.

Instead, speed (39%) is the most significant primary factor they consider when determining whether AI provides value to software engineering teams.

At the same time, however, 29% of respondents noted that increased time reviewing AI-generated work is now their biggest new source of friction over the past quarter. Governance, security and compliance concerns topped the list of challenges (24%), followed by gaps in engineering skills or AI literacy (21%), missing context, such as documentation, data or institutional knowledge (21%) and workflow friction (17% and infrastructure limitations (17%).

Amy Carrillo Cotten, director of client transformation for Uplevel, said the survey results suggest that software engineering teams are still in the early stages of AI adoption. For example, only 38% are at more advanced stages where they develop, maintain and share AI agents and capabilities across teams.

In contrast, 23% are able to configure AI agents with organizational context, such as codebases, conventions and runbooks, while another 38% consistently use AI tools for tasks such as code completion or have begun using AI agents with limited organizational context, the survey finds.

The average enterprise has yet to embrace agentic engineering as fully as a relatively small number of “Big Tech” companies that have built AI software factories, noted Carrillo Cotten. Many of those organizations are also not necessarily going to be able to absorb the increased rate of change that AI coding brings on, she added.

Regardless of sophistication, code reviews have become the new bottleneck in the AI era. There’s little doubt that AI coding tools are making individual developers more productive, but the rate at which software is being deployed has not commensurately increased. More software is rolled out faster, but gains are not being made at orders of magnitude largely because the legacy workflows have generally not yet been updated to accommodate massive increases in code volume. However, as AI is infused deeper into DevOps workflows, the pace at which software is being deployed should further accelerate.

In the meantime, more code that has not been vetted as fully as it should is finding its way into production environments. Much of that code tends to be overly verbose and often includes vulnerabilities that, once discovered, can now be more easily exploited. In many cases, exploits are now being created faster than patches are made available because cybercriminals have become more adept at using AI to reverse engineer vulnerabilities.

Eventually, DevOps workflows will be re-engineered for the AI era. The challenge, in the short term, is to make sure the AI cure for software development doesn’t wind up becoming worse than the proverbial disease.