DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • Calendar View
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • Calendar View
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Cloud Native Now
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • CI/CD
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Sustainability
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Chronosphere Adds Professional Services to Jumpstart Observability
  • Friend or Foe? ChatGPT's Impact on Open Source Software
  • VMware Streamlines IT Management via Cloud Foundation Update
  • Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
  • No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs

Home » News » 8-Bit Floating Point for AI/ML? | Amazon and Microsoft Shed Tech Jobs

8-Bit Floating Point for AI/ML? | Amazon and Microsoft Shed Tech Jobs

By: Richi Jennings on January 19, 2023 Leave a Comment

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: New ideas bring low-power ML inference, and more big-tech jobs are going.

1. FP8 Brings Fast AI

First up this week: New proposals to standardize 8-bit floating point storage. The idea is to speed up real-time inference, where speed and power efficiency are more important than accuracy..

Analysis: FP8 > INT8

Restricting data to byte widths has obvious speed and power benefits. 8-bit integers have been used in deep learning for a while, but suffer from overflow problems. Carving out an exponent permits a far greater dynamic range.

Karen Heyman: Will Floating Point 8 Solve AI/ML Overhead?

“FP32 or FP16 is just unnecessary overhead”
While the media buzzes about … ChatGPT, engineers are focused on the hardware challenges of running large language models and other deep learning networks. High on the ML punch list is how to run models more efficiently using less power, especially in critical applications like self-driving vehicles where latency becomes a matter of life or death.
…
IEEE 754, which defines FP32 bit and FP64, was designed for scientific computing, in which precision was the ultimate consideration. … Machine learning often needs less precision. … Arm, Intel, and Nvidia published a white paper proposing … two different flavors of FP8: E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa) … to save energy and performance overhead.
…
Training requires a wide dynamic range for the continual adjustment of coefficients that is the hallmark of backpropagation. … For inference, continuing operations in FP32 or FP16 is just unnecessary overhead, so there’s a quantization step to shift the network down to FP8 or Integer 8 (Int8). … As the quest for lower power continues, there’s debate about whether there might even be a FP4 standard.


Fascinating discussion. And ardit33 gets it:

This is a great article. I have had to deal with this lately, and FP8 / INT8 is very important especially for mobile/on-device inference.

AI will start to get embedded in all kind of devices … and often backend inference is not desired or feasible. It is actually pretty awesome if the industry moves towards INT8 / FP8 inference as a standard.


Is there another way? Wulfsta:

Personally, I’m hopeful we see further adoption of posits. Mostly just because their behavior is a little more sensible than floating point, but also there are benefits related to using them for AI.


2. Amazon and Microsoft Shed 28,000 Jobs

The two titans of Washington state both announced big job cuts this week. It’s almost as if they coordinated their news.

Analysis: Welcome to the not-a-recession recession

By its official definition, we’re not in recession. But, given tech firms’ behavior, it sure seems like it.

Matt Day and Spencer Soper: Amazon Is Set for New Round of Job Cuts

“Among several large tech companies”
The company announced earlier this month that it was laying off more than 18,000 employees. … They amount to about 6% of Amazon’s 350,000 corporate employees.
…
Amazon is among several large tech companies that are trimming their ranks, including Cisco Systems Inc., Intel Corp., Meta Platforms Inc., Qualcomm Inc. and Salesforce Inc.


But it’s not necessarily DevOps jobs affected. However, theodp doesn’t like the optics:

The hiring party is over at Amazon. The number of open jobs in the Software Development category has declined to 299 in January 2023 from 32,692 in May 2022.
…
[Last month] Amazon … delayed start dates for some university graduates who had been set to the join the company in May 2023, blaming the “macroeconomic environment” and telling students they would now not be able to begin until the end of 2023.


And then the other shoe dropped. Dina Bass joined Matt and Spencer for Round II: Microsoft Plans to Cut 10,000 Jobs

The software giant began notifying some of the 10,000 workers that will lose their jobs this quarter. … Both companies said the painful measures were necessary to offset slowing sales and a possible recession that has made customers more cautious.

Speaking before the cuts were announced, Chief Executive Officer Satya Nadella noted the tech industry is going through a period of slowing growth and will need to adjust: … “We will have to do more with less — we will have to show our own productivity gains with our own technology.”


Unusually, this time, we learned about the Microsoft layoffs from the press—not the CEO. verst feels strange about it:

We learned about the Microsoft layoffs in the news yesterday and right now nobody knows who will be impacted. I’m sure they will handle it as well as they can, but learning about it from the news first instead of the CEO is … odd. We did get an email from Satya this morning but … most of the company isn’t hiring now so there won’t be many internal jobs that those being terminated could try to apply for.
…
A leak is [not] unusual, but … learning about it this way feels strange. … At Microsoft the CEO does a great job usually announcing major news via email to employees and releasing it for investors publicly at the same time. Not so this time.


Wait. Pause. Are we in a recession or aren’t we? nightflameauto mourns the dignity of labor:

I believe the current definition of recession, as it’s being utilized by business and government alike recently is, “A nebulous concept of a future where profits are down, that we must fight against by lowering employee bargaining power, employee stake in their careers, and employee morale—until they get back in line and stop thinking their work has any value to the companies they work for.” Big business is essentially sick and damned tired of employees that have the temerity to believe they should be fairly compensated for their work. Solution: constant chatter of a recession while continually pulling moves that are historically terrific at destroying employee morale.
…
Man, capitalism is great. Smells like death and decay, don’t it?


What’s that? You want to unionize? mabbo is not a socialist:

I think Union isn’t the right word for what I want. I want a co-op where the employee organization owns the company. Profits are bonuses for all.
…
I don’t want protection from company’s goal of making the most money possible. I want the goal of the company to be the success of the employees.


The Moral of the Story:
You never really learn much from hearing yourself speak

—George Clooney

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or [email protected].

Image: Jonny Gios (via Unsplash; leveled and cropped)

Recent Posts By Richi Jennings
  • No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
  • US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead
  • OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’
More from Richi Jennings
Related Posts
  • 8-Bit Floating Point for AI/ML? | Amazon and Microsoft Shed Tech Jobs
  • How Artificial Intelligence, Machine Learning Can Help DevOps
  • Machine Learning, AI Driving DevOps Evolution
    Related Categories
  • AI
  • Blogs
  • Business of DevOps
  • DevOps and Open Technologies
  • DevOps Culture
  • DevOps Practice
  • DevOps Toolbox
  • Doin' DevOps
  • Features
  • Leadership Suite
  • Most Read
  • News
    Related Topics
  • ai
  • Amazon
  • deep learning
  • floating point
  • FP8
  • inference
  • layoffs
  • microsoft
  • ml
  • recession
  • The Long View
Show more
Show less

Filed Under: AI, Blogs, Business of DevOps, DevOps and Open Technologies, DevOps Culture, DevOps Practice, DevOps Toolbox, Doin' DevOps, Features, Leadership Suite, Most Read, News Tagged With: ai, Amazon, deep learning, floating point, FP8, inference, layoffs, microsoft, ml, recession, The Long View

« ClearML Releases New Reports Feature to Share Real-Time Results of Machine Learning Projects and Ignite ML Collaboration Across the Enterprise
6 Ways To Empower Developers and Increase Productivity »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Securing Your Software Supply Chain with JFrog and AWS
Tuesday, June 6, 2023 - 1:00 pm EDT
Maximize IT Operations Observability with IBM i Within Splunk
Wednesday, June 7, 2023 - 1:00 pm EDT
Secure Your Container Workloads in Build-Time with Snyk and AWS
Wednesday, June 7, 2023 - 3:00 pm EDT

GET THE TOP STORIES OF THE WEEK

Sponsored Content

PlatformCon 2023: This Year’s Hottest Platform Engineering Event

May 30, 2023 | Karolina Junčytė

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Latest from DevOps.com

Chronosphere Adds Professional Services to Jumpstart Observability
June 2, 2023 | Mike Vizard
Friend or Foe? ChatGPT’s Impact on Open Source Software
June 2, 2023 | Javier Perez
VMware Streamlines IT Management via Cloud Foundation Update
June 2, 2023 | Mike Vizard
Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
June 2, 2023 | Marc Hornbeek
No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
June 1, 2023 | Richi Jennings

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Most Read on DevOps.com

What Is a Cloud Operations Engineer?
May 30, 2023 | Gilad David Maayan
Forget Change, Embrace Stability
May 31, 2023 | Don Macvittie
No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
June 1, 2023 | Richi Jennings
Five Great DevOps Job Opportunities
May 30, 2023 | Mike Vizard
Checkmarx Brings Generative AI to SAST and IaC Security Tools
May 31, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.