Blog

Historically speaking, why haven’t companies used statistics to manage their innovation portfolios? I’ve had this question a lot lately.

It’s a great question. Innovation is one of the most risky aspects of business, and statistics are the tools of risk management. Yet for some reason, despite 70% – 90% failure rates and trillions of dollars spent by companies each year in new product launches, acquisitions and venture capital, innovation has somehow remained a statistics-free-zone.

Why can’t we calculate the risks of innovation investments?  Why can’t probabilities help us find innovation opportunities, decide what to fund, pivot projects, shape portfolios and apportion resources?

I’m no historian, but as someone who uses statistics to manage innovation portfolios, it would have been a lot harder to do twenty years ago.

Not long ago, if your job used statistics you were probably an engineer, an economist or an actuary.  Actuaries and economists have been sitting quietly in society’s backrooms for thousands of years, crunching the numbers all of us assume “someone” is crunching. Most people associate actuaries with insurance companies, where they figure out your risk of getting sick or having a car crash before deciding what to charge you. Other actuaries sit in banks and investment firms, trying to figure out the risks of stocks, bonds, currencies and other things to help investors make money. 

The sad truth about statistics; they aren’t much fun.  No kid grows up wanting to do statistics for an insurance company.  Learning about probabilities and risk management is tedious and hard. It takes special training. Unlike movie stars, number crunchers also aren’t society’s most beloved members because it’s often their job to assign cold dollar figures to priceless, emotional human events (ex. death, sickness, injury). However – like plumbers, trash collectors and funeral directors – they’ve been accepted as necessary members of society who do a dirty job that, regrettably, someone has to do.

Then, twenty years ago this began to change.  In one generation, actuarial science and econometrics crept up from obscurity to reshape almost every major field of human endeavor. From roots in universities, insurance and finance, statistics now inform how doctors diagnose patients, how judges issue jail sentences, how athletes train and coaches recruit, how elections are run, how travel is booked, how romantic dates are set up, how children are taught, how college students are admitted, how lost autistic kids are found, how drugs are tested, how movies are made, how crime is deterred and how wars are fought.

The explosion in statistics marched in lockstep with the rise of modern computing. There are a few things you need before you can spread statistics into a new domain. First, you need data. You also need something to “crunch” those data with – it used to be paper, then abacus, then slide rules, then calculators and now computers.

For obvious reasons, the rise of modern computing has made every ingredient of statistics more abundant, cheap and accessible than ever before. Data is everywhere. Computing power is everywhere. Free software has never been more powerful. As a result, statistical analyses that used to be the exclusive domain of highly trained “quants” with elite resources can now be learned by teenagers with free open source software, a laptop and a little instruction from YouTube. This merger of statistics and technology gave rise to what’s now called “data science”. Math is becoming democratized.

As hard math moved from exotic to mainstream, people have become refreshingly more comfortable with the idea of using statistics to shape innovation efforts. When I first started doing this kind of work nearly a decade ago, the very premise of bringing statistics into the “creative” world of innovation struck lots of people as either offensively naïve at one extreme, or offensively arrogant at the other extreme. It was an alien concept. There was no word for “big data” or “data science.” There was no iPhone. Blockbuster was bigger than Netflix. It wasn’t that long ago, but in tech years it was ages ago.

Now words like data science and even “analytics” have entered the modern lexicon, sparing me at least half an hour of explanation in every conversation. People have stopped saying it’s impossible to use statistics in innovation. Instead they’ve started asking; “why haven’t people always used statistics to manage innovation portfolios?”. What was once offensive has now become obvious, and the change has been refreshing. Hard math is becoming easier and more accessible, allowing it to enter new domains faster than ever. This is leading to new questions, new answers and world where knowledge-seeking has become noticeably more democratized.