DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • DevOps Chats
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Rocket on DevOps.com
    • Traceable on DevOps.com
    • Quali on DevOps.com
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Digital Anarchist
  • Media Kit
  • About
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • Leadership Suite
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More Topics
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » DevOps Practice » Developing Simple and Stable Machine Learning Models

Developing Simple and Stable Machine Learning Models

Developing Simple and Stable Machine Learning Models

By: Meir Maor on April 12, 2019 2 Comments

A current challenge and debate in artificial intelligence is building simple and stable machine learning models capable of identifying patterns and even objects. Is it possible to know which models will work best or to simply see the data? It is possible, and the question is figuring out how to get there.

Related Posts
  • Developing Simple and Stable Machine Learning Models
  • Cloudera Calls for MLOps Standards Initiative
  • Are Self-Service Machine Learning Models the Future of AI Integration?
    Related Categories
  • AI
  • Blogs
  • DevOps Practice
  • Leadership Suite
    Related Topics
  • artificial intelligence
  • L1 regularization
  • L2 regularization
  • machine learning
Show more
Show less

Simple models are popular because they are easy to understand, trustworthy and tend to be more resilient to change than complex models. Simple models also have the benefit of overcoming transfer learning pains. While everyone wants simple models, they also want accurate models. In practice, there are always transfer learning issues when applying learning from the past, into future practice.

DevOps/Cloud-Native Live! Boston

An issue arises when trying to fit a model from a known family to the available data and hoping it will generalize effectively. It could be any proposed family: linear models, trees or trained deep neural networks.

There remains a fundamental problem with any overly complicated model—linear, tree or neural network—in the risk of “overfitting” the data resulting in poor generalization and results. At the same time, an overly simple model risks missing the nuances of the data and sacrificing accuracy. There are techniques to handle this problem.

First is to reduce the size of the set of models. This works well with the PAC model. For example, reduce the number of parameters, reduce the depth of the tree and reduce the nodes in the network. Reducing the size of the set means it’s easier to understand, and with fewer possibilities built into the model, it’s less likely to overfit the data.

Next, add L1 or L2 regularization. Models with small weight, or many zero weights, are preferable and will help the model generalize. The question is, why? Why do we use trees or neural networks rather than another hypothesis family? Why do we want models with small weights? Why do some network topologies work well?

These questions bring in the actual world we live in and the types of problems that likely need solving. There is no learning when any function is allowed, and the model family isn’t limited. It is well-known that when the family of allowed hypotheses is too large, the model overfits the data ad-absurdum and becomes useless. Thus, machine learning almost always optimizes along a well-understood family of hypotheses while human feature engineering and representation construction captures our prior knowledge about the world. The question is how to exploit such knowledge when automating machine learning. Arbitrary limiting is better than not limiting at all, but we can do better.

The world has patterns repeating themselves across many domains. Different applications will use similar building blocks to analyze rocket movement or medical sensors, or to solve a predictive maintenance task. But we are all experts in the world we live in. Even studying a domain where we don’t have specific expertise, we still recognize some patterns and rules.

For example, take a look at this ring-tailed lemur. Even if you’ve never seen one before, you are now likely to identify this type of primate if you come across one. My 5-year-old son was shown this picture and he could visit the zoo and identify a ring-tailed lemur. It only took one photo.

Compare that result to modern neural network research which typically requires many examples—often into the millions—when training to identify objects, and still they suffer from embarrassing mistakes when new data differs from the data the network was trained on.

How did my 5-year-old son learn to identify ring-tailed lemurs from a single picture? He’s an expert. Not a lemur or primate expert, but an expert in the world we live in. He understands identifying an object from its background. What a tail, eyes, ears, legs and fur look like. He knows how to imagine a 3D object from different directions. At 5, he already brought a ton of knowledge with him when learning to identify lemurs.

This is the essence of learning. We understand the notion of simplicity is not in number of nodes in the network or number of lines of code, but that it relates to simplicity when explaining human language.

This process can be formalized by codifying human knowledge to solve tough problems. We can formalize the many patterns seen before in various domains and in the various tools data scientists use when constructing features. Then we can look for patterns that make sense and are similar to what we’ve seen before.

We can also look for more novel patterns represented within the complex combinations of things we’ve seen before. As an industry, we need to do exactly this: Codify human knowledge in curated code libraries, and in structured knowledge and facts, use them to search for patterns in data and build highly accurate resilient models. This will bring simplicity and stability to machine learning models without sacrificing accuracy by leveraging humanity’s existing knowledge.

— Meir Maor

Filed Under: AI, Blogs, DevOps Practice, Leadership Suite Tagged With: artificial intelligence, L1 regularization, L2 regularization, machine learning

Sponsored Content
Featured eBook
The State of Open Source Vulnerabilities 2020

The State of Open Source Vulnerabilities 2020

Open source components have become an integral part of today’s software applications — it’s impossible to keep up with the hectic pace of release cycles without them. As open source usage continues to grow, so does the number of eyes focused on open source security research, resulting in a record-breaking ... Read More
« Connections
Tax Day IT Lessons, or How to Avoid Wiping Out on Your Biggest Day of the Year »

TechStrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Accelerating Continuous Security With Value Stream Management
Monday, May 23, 2022 - 11:00 am EDT
The Complete Guide to Open Source Licenses 2022
Monday, May 23, 2022 - 3:00 pm EDT
Building a Successful Open Source Program Office
Tuesday, May 24, 2022 - 11:00 am EDT

Latest from DevOps.com

DevOps Institute Releases Upskilling IT 2022 Report 
May 18, 2022 | Natan Solomon
Creating Automated GitHub Bots in Go
May 18, 2022 | Sebastian Spaink
Is Your Future in SaaS? Yes, Except …
May 18, 2022 | Don Macvittie
Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?
May 17, 2022 | Richi Jennings
Making DevOps Smoother
May 17, 2022 | Gaurav Belani

Get The Top Stories of the Week

  • View DevOps.com Privacy Policy
  • This field is for validation purposes and should be left unchanged.

Download Free eBook

The 101 of Continuous Software Delivery
New call-to-action

Most Read on DevOps.com

Top 3 Requirements for Next-Gen ML Tools
May 13, 2022 | Jervis Hui
15 Ways Software Becomes a Cyberthreat
May 13, 2022 | Anas Baig
Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil
May 16, 2022 | Vladi Sandler
Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Micro...
May 17, 2022 | Richi Jennings
Making DevOps Smoother
May 17, 2022 | Gaurav Belani

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2022 ·Techstrong Group, Inc.All rights reserved.