The AI talent shortage — can companies close the skills gap?

Your deep generative AI (genAI) large language model (LLM) knowledge and experience could set you up for a $1 million pay day.

The Wall Street Journal reported recently that software engineers who are experienced in training LLMs and who can rectify troublesome genAI problems, such as “AI hallucinations,” are in extremely high demand. According to the publication, the industry is willing to pay over $1 million in salary, bonus, and accelerated stock options to the most experienced individuals. 

“There is a secular shift in what talents we’re going after,” Naveen Rao, head of generative AI at Databricks, told the Journal. “We have a glut of people on one side and a shortage on the other.” Rao says there might be only a couple of hundred people out there who are qualified.

Meta CEO Mark Zuckerberg has sent emails directly to top people at Google’s DeepMind in an attempt to persuade them to accept Meta’s AI-related job offers. Google’s Sergey Brin personally called a Google employee who was leaving for OpenAI and — by offering a pay increase and other perks — persuaded the employee to remain at Google.

It’s not just Meta and Google that are after the top minds in genAI; start-ups, large corporate entities and even whole countries are after the best AI talent. There are reports of companies trying to hire away whole genAI teams from their competitors. 

The competition isn’t just about employees. Meta is also apparently seeking to corner the market on Nvidia H100 AI GPUs, which cost $30,000 each. The company placed an order with Nvidia for 350,000 units for 2024 (amid estimates that put Nvidia’s entire 2023 run of H100 AI GPUs at about 550,000.)

GenAI job seekers: Beware

Be wary of career fads built on knee-jerk assumptions about how AI will take over the business world. Companies need top expertise now, and are willing to pay for it; but what happens when companies reach their genAI goals? Are they going to keep paying you a pretty penny in perpetuity? Or will they look for a way out when the urgent need is no longer so urgent?

AI is on a fast track, but hype and immaturity could derail it. It’s human nature to amp up the outlook of emerging technologies and fast-moving tech trends. A lot of things are being predicted right now about where AI will take us. Hint: some of them won’t be true. 

“Historically, academia was at the heart of breakthroughs in machine learning models, with universities and research institutions leading the charge,” Neil C. Hughes writes in Techopedia. In recent years, the tech industry has taken over the AI innovation lead. One reason for that: academic institutions can’t afford the price of admission for hardware. This discrepancy results in a significant skills gap, in which competencies taught through standard educational methodologies fall short of the industry’s current requirements for AI technology, Hughes adds. 

The upshot: many of our teaching institutions can’t deliver the pertinent in-depth training needed by software engineering students and those looking to upskill with genAI.

For now, the nuances of building and managing LLMs are known to only a small fraction of the workforce; ultimately, companies need to rethink and reconsider how to get much better at upskilling and training their employees for the roles that need filling in a genAI world. 

Closing the AI skills gap

To some extent, chasing the small number of experienced genAI experts is a bit like rearranging deck chairs on the Titanic. AI is a huge wave of disruption that will transform many aspects of business globally. According to research by IBM, executives estimate that 40% of their workforces will need to reskill over the next three years as a result of implementing AI. This is the chief challenge businesses need to focus on. 

Although many companies have not yet come to terms with how to address AI upskilling and reskilling, it’s dawning on them that the knowledge that needs to be imparted can only partially be handled in traditional ways. 

According to Boston Consulting Group, the average half-life of skills is under five years, and in some tech fields it’s as short as two and a half years. Skills will overtake degrees as the key signposts on resumés leading to employment. And the focus of upskilling and reskilling should be on genAI skills needed by your company, not generic AI training. Some even foresee a new skills-based economy, where skills become equivalent to currency.

A few forward-thinking companies such as Amazon, Ericsson, and Vodaphone are operating internal AI upskilling programs, but a lot more needs to be done. By and large, companies aren’t yet meeting the needs of workers, who would very much like AI upskilling. Worldwide, almost 80% say their AI training is insufficient, according to an OliverWyman Forum report.

It’s time for companies to invest in genAI and machine learning/deep learning and put their money where their mouths are to build internal training programs for employees. Given where the tech industry is headed with genAI, it’s the smart bet, both for companies and the people who’ll lead them to success.