top of page

Overview of Chapter 3

 

​Performance: Establish the Evaluation Metrics​

​

Once you’ve established what machine learning will predict, the next question is how well it predicts. Fortunately, evaluating its performance doesn’t require becoming a technical expert, since you can benchmark a model without regard to its inner workings. Here, we only judge how well it predicts, not how it predicts. It’s only a matter of arithmetic, not “rocket science.”

​

Often, you will hear of accuracy, a simple tally of how often a model predicts correctly. But accuracy is not only the wrong measure for most ML projects; it also feeds a common fallacy that tremendously mismanages expectations.

​

If not accuracy, then what metric? One is lift, a simple multiplier that tells you how many times better than guessing your model predicts. Another is cost—the price of each false positive and the (usually very different) price of each false negative.

​

Once established, the metrics serve to evaluate both model training (step 5) and deployment (step 6). This chapter gets to the heart of the matter. Exactly how valuable is imperfect prediction? In what way do all ML deployments serve to triage and prioritize? And how do you translate from raw predictive performance to true business metrics like profit?

ai_playbook_chapters_7.jpg

Articles excerpted from this chapter:

​

Built In: Predictive AI Streamlines Operations In This Surprisingly Simple Way

​

Scientific American: Accuracy Fallacy: The Media's Coverage of AI Is Bogus (pre-dates the book, but included therein)

​

MIT Sloan Management Review: What Leaders Should Know About Measuring AI Project Value â€‹(read it in Polish)

​

​

See also the author's article series in Forbes, which focuses largely on this chapter's topic of metrics and ML valuation.

Book overview

The AI Playbook - front cover.png

The AI Playbook 

Mastering the Rare Art of

Machine Learning Deployment

​

Eric Siegel delivers a robust primer on machine learning, the key mechanism in AI. A forward-looking, practical book and a must-read for anyone in the information economy.”​

—Scott Galloway, NYU Stern Professor of Marketing and bestselling author of The Four

​

“An antidote to today's relentless AI hype – why some AI initiatives thrive while others fail and what it takes for companies and people to succeed.” 

—Charles Duhigg, author of bestsellers The Power of Habit and Smarter Faster Better

​

“Set aside the hype and focus on getting things to work in practice. This is a crisp, necessary, and deeply helpful guide to getting things done with AI. Essential reading.”

—Mustafa Suleyman, Co-founder & CEO, Inflection AI and author of The Coming Wave

​

“In this book, Eric brings machine learning to life and provides a roadmap for how to operationalize it in the real world.”

—Will Lansing, CEO, FICO

In his bestselling first book, Eric Siegel explained how machine learning works. Now, in The AI Playbook, he shows how to capitalize on it.

The greatest tool is the hardest to use. Machine learning is the world’s most important general-purpose technology – but it’s notoriously difficult to launch. Outside Big Tech and a handful of other leading companies, machine learning initiatives routinely fail to deploy, never realizing value. What’s missing? A specialized business practice suitable for wide adoption. In The AI Playbook, bestselling author Eric Siegel presents the gold-standard, six-step practice for ushering machine learning projects from conception to deployment – bizML. He illustrates the practice with stories of success and of failure, including revealing case studies from UPS, FICO, and prominent dot-coms. This disciplined approach serves both sides: It empowers business professionals and it establishes a sorely needed strategic framework for data professionals.

​

Beyond detailing the practice, this book also upskills business professionals – painlessly. It delivers a vital yet friendly dose of semi-technical background knowledge that all stakeholders need in order to lead or participate in machine learning projects, end to end. This puts business and data professionals on the same page so that they can collaborate deeply, jointly establishing precisely what machine learning is called upon to predict, how well it predicts, and how its predictions are acted upon to improve operations. These essentials make or break each initiative – getting them right paves the way for machine learning’s value-driven deployment.

​

​

Author: Eric Siegel, Ph.D., is a leading consultant and former Columbia University and UVA Darden professor. He is the founder of the long-running Machine Learning Week conference series, a frequent keynote speaker, and author of the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or DieMore about Eric

QUICK LINKS FOR READERS:

​

Press: radio, reviews, excerpts, podcasts

Book FAQ (what it's about and who it’s for)

BizML Cheat Sheet

Audiobook PDF (includes all figures)
Chapter notes  |  Glossary

Hardcover, ebook, or audiobook

Amazon  //  Amazon UK  //  Amazon Canada

Audible // B&N  //  Apple  //  Google Play

Target // Bookshop.org  //  Books-A-Million

Indigo // Powell's // Porchlight

 

“There's a big difference between theory and practice. This is the guide that tells you how to really make data come alive through machine learning.

—DJ Patil, General Partner, Great Point Ventures and Former U.S. Chief Data Scientist

 

This should be requisite reading for any professional serious about driving true value through the power of machine learning. Eric presents a pragmatic approach and it’s not just about having the best algorithms – it’s about ensuring you have a path to true productionalization at scale.”
—Jon Francis, Chief Data and Analytics Officer, GM

​

The AI Playbook grabs you from the first pages. It's an indispensable guide for anyone, both technical and non-technical, interested in discovering how AI is really put into practice.”
—Barbara Oakley, author of A Mind for Numbers and co-instructor of Coursera’s Learning How to Learn

​

The AI Playbook clearly explains what you need to know and what to avoid to boost your return on AI investments.”
—Terry Sejnowski, President of NeurIPS, Professor at Salk Institute and UCSD, and co-instructor of Coursera's Learning How To Learn

​

This book blows the lid off, showing precisely what it takes to boost enterprise efficiencies with AI.”
—Chris Pouliot, VP Data Science & Analytics, Snowflake

​

This book is the driver's manual for machine learning – every business and analytics professional should read it.”
—Morgan Vawter, Global VP Data & Analytics, Unilever

​

"We have all heard about how AI changes everything. But to translate AI into actual value for your organization, you need to read this book!”
—Viktor Mayer-Schönberger, Professor of Internet Governance, Oxford and co-author of Big Data

​

See 18 more endorsements from Thomas Davenport, Alex Pentland, Sarah Kalicin, Gerd Gigerenzer, John Elder, and more.

This book is part of the “Management on the Cutting Edge” series from MIT Sloan Management Review and MIT Press.

bottom of page