In the last year and a half, we’ve seen a glut of new generative AI-related tools reach the market. Various AI pair programmers powered by large language models (LLMs) are becoming integrated into software development environments. The number of new generative AI startups is staggering, and generative AI seems to be the new “catch-all” to accelerate software development and operations.
In this new paradigm, AI tooling accumulation seems inevitable. Yet, the explosion of new AI APIs, libraries and frameworks has the very real potential to increase technical debt and, in the process, overwhelm DevOps teams. “These new applications and capabilities will require many new MLOps processes and tools that could overwhelm any existing, already overloaded DevOps team,” said Bob Quillin, vFunction chief ecosystem officer.
Below, we’ll consider if and how new generative AI tooling has the potential to exacerbate technical debt. Notably, the rush to adopt AI, shifts in the AI tooling landscape and accelerated code output could all be producers of technical debt. Choosing your tools correctly, iterating thoughtfully and upskilling teams will be important guidelines to mitigate negative outcomes. That, and creating unique solutions to ‘flip the coin’ — using AI to consolidate code and reduce existing technical debt.
How AI Will Contribute to Technical Debt
The rise of generative AI-related tools will likely increase technical debt, both due to the rush to hastily adopt new capabilities and the need to mold AI models to suit specific requirements. “New LLMs and generative AI applications will undoubtedly increase technical debt in the future, or at a minimum, greatly increase the need to manage that debt proactively,” said Quillin. “It starts with new requirements to continually manage, maintain, and nurture these models from a broad range of new KPIs from bias, concept drift, and shifting business, consumer, and environmental inputs and goals,” he said.
Incorporating AI may require a significant upfront commitment, leading to additional technical debt. “It won’t be just a build-and-maintain scenario, but rather, the first of many steps on a long road ahead,” said Prince Kohli, CTO of Automation Anywhere. Product companies with a generative AI focus must invest in creating a data and model strategy, a data architecture to work with AI, controls for the AI and more. “Technology disruptions and pivots such as this always lead to this kind of technical debt that must be continually paid down, but it’s the price of admittance,” he said.
AI is a tumultuous space, and some degree of technical debt will likely be produced by future organizational shifts. “The pace of change and innovation in the AI field itself is exceedingly rapid,” said Juan Orlandini, CTO of Insight North America, citing the staggering emergence of new AI startups and tools. “There will be consolidation, and also, unfortunately, a number of the startups will cease to exist.” When this occurs, teams must grapple with unsupported packages or tools.
Another type of debt is the nature of AI and how it accelerates development across the board. “One of the great values of these tools is the acceleration they bring to development,” said Orlandini. “Simply put, we can put out more features faster.” The other side of this double-edged sword is that AI accelerates the creation of all programming endeavors, even those that become technical debt.
Generative AI-related tools are most likely to accelerate the pace of change and deliver new code development outcomes more quickly. “In the process of consuming their output, there is a possibility of accumulating technical debt if the usage is broad, rapid, and pursued indiscriminately,” said Ajay Sabhlok, CIO and CDO, Rubrik. “Developers, in this case, may be too focused on program execution and forget to pay attention to code complexity, repetition or use of non-standard logic.”
Ways to Mitigate AI Technical Debt
Thankfully, there are specific methods organizations can adopt to manage new technical debt. For starters, you should focus on creating smaller proof of concepts and then iterating as you learn, said Kohli. “When leaders like the head of technology or IT are deploying these processes, they should first create small agile teams that learn fast and disseminate those learnings efficiently,” he said.
Also, technical debt is more likely to accrue when the wrong platform or tool is chosen. Therefore, “organizations should be very deliberate in choosing what products to leverage,” said Orlandini. He encourages IT leaders to avoid relying on companies or products with flashy presentations and poor tech or great tech with poor business acumen. “Our industry is rife with stories of great products with terrible business plans — which ultimately leads to their demise,” he said.
Other experts similarly foresee upcoming shifts that may affect internal operations. “Companies should keep their options open when considering AI model vendors or tool providers,” said Kohli. “They must keep their ‘hands on the wheel’ and be prepared for turns.”
Just like any new technological adoption, generative AI requires forethought. Therefore, putting the same amount of planning and preparation into deploying generative AI as we do for developing coding standards around manual coding could help minimize technical debt, said Sabhlok. “It will help to ease into areas of usage by adopting a tiered approach to expansion and, at each step of the way, review the results against standards, compare notes across the development organization, and course correct where necessary. This can avoid the accumulation of technical debt in the long run.”
“IT leaders should first ensure that additional resourcing and training are in place to both upskill existing teams and hire new talent as required,” said Quillin. “This includes adding observability tools that can identify new and existing forms of architectural and LLM drift. Finally, to always be ready for any new, rapidly emerging future requirements and opportunities, IT leaders should invest in best practices and tools that will proactively keep their app architectures clean and ready for the ‘next big thing.'”
How AI Could Decrease Technical Debt
On the other hand, some believe that if used properly, the contrary is true — that new AI technologies will actually reduce technical debt. “The rise of LLMs, frameworks, and APIs, when applied properly together, will actually reduce technical debt,” said Ashwin Rajeeva, CTO and co-founder of Acceldata. “Code discovery and reuse is more easily accomplished with LLMs.” He points to the ability of this technology to identify duplicative code and consolidate it into a more streamlined interface.
Automated documentation generation is another area where LLMs could help improve code discovery and reduce duplicative efforts. In addition, LLMs could also help reduce the prevalence of code that doesn’t follow company standards, said Rajeeva. “It is much easier for LLMs to understand the company standards by learning from good coding practices and applying it to newly written code.”
Others also see the power of using generative AI to enhance automation across the board. “AI tooling is probably going to help the most in automation, velocity to transform old codebases, and empowering non-coders to accelerate the adoption of new business processes into their workflows,” said Orlandini.
AI is a multifaceted subject, and its use could go one of two ways. On one hand, if not used correctly, generative AI could only add to the surmounting technical debt pile. To avoid this outcome, organizations will likely need guardrails on when and how to use LLMs, said Rajeeva. He also advocates “making it a best practice within the corporation with sufficient training for newer developers.”
Then, on the other hand, generative AI could be utilized to respond to technical debt. It has the awesome potential to unlock automation gains and enhance employee satisfaction and retention, especially within older industries that may be behind the technological curve, said Kohli. “With AI and now generative AI, intelligent automation can deliver on all those promises such that it is fast becoming a new pillar of enterprise architecture.”
So, take the new wave of AI with a grain of salt. It will open up immense solutions but will also come with a significant degree of technical debt. Thankfully, there are ways to manage debt caused by AI in a way that can benefit operations and push development objectives forward.