Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: New ideas bring low-power ML inference, and more big-tech jobs are going.

1. FP8 Brings Fast AI

First up this week: New proposals to standardize 8-bit floating point storage. The idea is to speed up real-time inference, where speed and power efficiency are more important than accuracy..

Analysis: FP8 > INT8

Restricting data to byte widths has obvious speed and power benefits. 8-bit integers have been used in deep learning for a while, but suffer from overflow problems. Carving out an exponent permits a far greater dynamic range.

Karen Heyman: Will Floating Point 8 Solve AI/ML Overhead?

“FP32 or FP16 is just unnecessary overhead”
While the media buzzes about … ChatGPT, engineers are focused on the hardware challenges of running large language models and other deep learning networks. High on the ML punch list is how to run models more efficiently using less power, especially in critical applications like self-driving vehicles where latency becomes a matter of life or death.
…
IEEE 754, which defines FP32 bit and FP64, was designed for scientific computing, in which precision was the ultimate consideration. … Machine learning often needs less precision. … Arm, Intel, and Nvidia published a white paper proposing … two different flavors of FP8: E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa) … to save energy and performance overhead.
…
Training requires a wide dynamic range for the continual adjustment of coefficients that is the hallmark of backpropagation. … For inference, continuing operations in FP32 or FP16 is just unnecessary overhead, so there’s a quantization step to shift the network down to FP8 or Integer 8 (Int8). … As the quest for lower power continues, there’s debate about whether there might even be a FP4 standard.

Fascinating discussion. And ardit33 gets it:

This is a great article. I have had to deal with this lately, and FP8 / INT8 is very important especially for mobile/on-device inference.

AI will start to get embedded in all kind of devices … and often backend inference is not desired or feasible. It is actually pretty awesome if the industry moves towards INT8 / FP8 inference as a standard.

Is there another way? Wulfsta:

Personally, I’m hopeful we see further adoption of posits. Mostly just because their behavior is a little more sensible than floating point, but also there are benefits related to using them for AI.

2. Amazon and Microsoft Shed 28,000 Jobs

The two titans of Washington state both announced big job cuts this week. It’s almost as if they coordinated their news.

Analysis: Welcome to the not-a-recession recession

By its official definition, we’re not in recession. But, given tech firms’ behavior, it sure seems like it.

Matt Day and Spencer Soper: Amazon Is Set for New Round of Job Cuts

“Among several large tech companies”
The company announced earlier this month that it was laying off more than 18,000 employees. … They amount to about 6% of Amazon’s 350,000 corporate employees.
…
Amazon is among several large tech companies that are trimming their ranks, including Cisco Systems Inc., Intel Corp., Meta Platforms Inc., Qualcomm Inc. and Salesforce Inc.

But it’s not necessarily DevOps jobs affected. However, theodp doesn’t like the optics:

The hiring party is over at Amazon. The number of open jobs in the Software Development category has declined to 299 in January 2023 from 32,692 in May 2022.
…
[Last month] Amazon … delayed start dates for some university graduates who had been set to the join the company in May 2023, blaming the “macroeconomic environment” and telling students they would now not be able to begin until the end of 2023.

And then the other shoe dropped. Dina Bass joined Matt and Spencer for Round II: Microsoft Plans to Cut 10,000 Jobs

The software giant began notifying some of the 10,000 workers that will lose their jobs this quarter. … Both companies said the painful measures were necessary to offset slowing sales and a possible recession that has made customers more cautious.

Speaking before the cuts were announced, Chief Executive Officer Satya Nadella noted the tech industry is going through a period of slowing growth and will need to adjust: … “We will have to do more with less — we will have to show our own productivity gains with our own technology.”

Unusually, this time, we learned about the Microsoft layoffs from the press—not the CEO. verst feels strange about it:

We learned about the Microsoft layoffs in the news yesterday and right now nobody knows who will be impacted. I’m sure they will handle it as well as they can, but learning about it from the news first instead of the CEO is … odd. We did get an email from Satya this morning but … most of the company isn’t hiring now so there won’t be many internal jobs that those being terminated could try to apply for.
…
A leak is [not] unusual, but … learning about it this way feels strange. … At Microsoft the CEO does a great job usually announcing major news via email to employees and releasing it for investors publicly at the same time. Not so this time.

Wait. Pause. Are we in a recession or aren’t we? nightflameauto mourns the dignity of labor:

I believe the current definition of recession, as it’s being utilized by business and government alike recently is, “A nebulous concept of a future where profits are down, that we must fight against by lowering employee bargaining power, employee stake in their careers, and employee morale—until they get back in line and stop thinking their work has any value to the companies they work for.” Big business is essentially sick and damned tired of employees that have the temerity to believe they should be fairly compensated for their work. Solution: constant chatter of a recession while continually pulling moves that are historically terrific at destroying employee morale.
…
Man, capitalism is great. Smells like death and decay, don’t it?

What’s that? You want to unionize? mabbo is not a socialist:

I think Union isn’t the right word for what I want. I want a co-op where the employee organization owns the company. Profits are bonuses for all.
…
I don’t want protection from company’s goal of making the most money possible. I want the goal of the company to be the success of the employees.

The Moral of the Story:
You never really learn much from hearing yourself speak

—George Clooney

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: Jonny Gios (via Unsplash; leveled and cropped)