Blogs

Do You Need a Cloud AI Engineer?

For all the technological advancements we are witnessing in AI and machine learning, the practical benefits remain elusive in the non-tech business world, frustrating decision-makers as AI investment climbs. For example, over the past several years, many Fortune 500 companies have invested in strong data science teams and AI labs, but still struggle when it comes to producing and scaling their models. Moreover, a recent report by Algorithmia found that, while enterprise budgets for AI and ML are rapidly increasing for 83% of organizations studied, deployment challenges persist.

These deployment issues often begin when leadership fails to exert enough upfront engineering focus to ensure their proof of concepts (POCs) are “real-world ready.” The challenges are compounded when businesses put these models into action using insufficient, outdated or cobbled-together IT infrastructures. This combination results in POCs that either fall apart under real-world conditions or are rife with inefficiencies. Fortunately, cloud tech today provides the opportunity to overcome these challenges.

Businesses can enable development, experimentation and scaling of data science and AI solutions with more flexibility and speed than ever through cloud platforms. A design-led AI project that uses the tools available in the cloud can help organizations create business impact and realize ROI much faster.  So while the role of data scientist may have been the ‘sexiest job’ in recent years, we may be seeing a new ‘sexiest job’ title emerging: the cloud AI engineer.

Why do we need a Cloud AI Engineer?

Traditionally, a successful AI project in the cloud requires a team with different skills and titles, including data scientists, big data engineers, cloud engineers and full stack developers. This has led businesses to look at ways to merge these specialties into one specific title. Enter the cloud AI engineer who, theoretically, would be able to cover all these duties — including knowledge of and experience with the tools and accelerators that cloud platforms now offer, as well as demonstrated aptitude for fast-paced development in AutoML. These are skills and tasks previously spread out across other areas and roles within the organization.

What is the Job Description?

So what would this job actually look like? The cloud AI engineers of the future will focus on the deployment of AI and ML models at scale in the cloud, and on integrating them with existing products and IT systems. Additionally, cloud AI engineers would need knowledge of the different types of machine-learning algorithms; familiarity with frameworks like Tensorflow and PyTorch; hands-on experience with data processing; analysis and feature selection using analytic tools and ETL/ELT frameworks (Apache Spark, Hive, Amazon, Athena, etc.); and model deployment experience on cloud platforms like AWS, Azure or Google Cloud.

Obviously, tackling all of these would be a huge undertaking for one individual; given the vast workloads on both the cloud and engineering fronts, it isn’t possible to replace these workers and their skills with just a handful of cloud AI engineers. Instead, cloud AI engineers will sit between the two disciplines and the workforces and will serve as a key bridge to easily deploy models and integrate them with enterprise systems and tools for consumption.

Where are We Going to Find Cloud AI Engineers?

Given that cloud AI engineers will have a robust level of responsibility, they will quickly become some of the most sought-after team members — but that also means talent for the role will likely be in short supply.

Additionally, individuals with this broad skill set will not appear overnight — at least not en masse. Therefore, to build the cloud architecture capable of delivering the real-world AI results that businesses need today, tech companies need to find ways to break down the internal divides between data scientists and cloud engineers so that they can cross-train, learn and develop skills in each area. If this is done correctly, not only will tech companies have a more well-rounded workforce, but they will be able to deliver the practical AI success that non-tech businesses are looking for today.

Despite excitement about AI and its benefits, operational challenges remain. By focusing on the cloud and building up specialists in this space, AI can begin to drive the practical results the business world needs far more quickly.

Sandeep Dutta

Sandeep Dutta is chief practice officer at Fractal Analytics.

Recent Posts

Survey Surfaces Substantial Platform Engineering Gains

While most app developers work for organizations that have platform teams, there isn't much consistency regarding where that team reports.

14 hours ago

EP 43: DevOps Building Blocks Part 6 – Day 2 DevOps, Operations and SRE

Day Two DevOps is a phase in the SDLC that focuses on enhancing, optimizing and continuously improving the software development…

15 hours ago

Survey Surfaces Lack of Significant Observability Progress

A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to…

16 hours ago

EP 42: DevOps Building Blocks Part 5: Flow, Bottlenecks and Continuous Improvement

In part five of this series, hosts Alan Shimel and Mitch Ashley are joined by Bryan Cole (Tricentis), Ixchel Ruiz…

16 hours ago

Is Backstage the Right Internal Developer Portal for You?

There has never been a better time to be a software developer. There is a language and framework to solve…

21 hours ago

Our Infrastructure is Still Expanding

Infrastructure is expanding in almost every possible way, and this creates more of a burden on every aspect of IT,…

22 hours ago