Should banks stop using averages?

Banking continues to grow more complex, creating more uncertainties and risks. Yet banks and credit unions tend to oversimplify when gauging probabilities because they lack enough standardized data about key risk metrics to build more effective predictive models.

Sam Savage, executive director of the nonprofit Probability Management, says financial services organizations must develop a more sophisticated approach to probabilities and should change the way they think about and deal with uncertainties.

They should stop using averages because they are not robust enough, given banking complexities. Banks should also bring top decision-makers, data scientists, statisticians and the IT department together to shape a more accurate and valuable predictive method.

Savage is the author of the book, “The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty.” He has a Ph.D. in computational complexity from Yale University and is an adjunct professor of civil and environmental engineering at Stanford University

“The Flaw of Averages” is the title of one of your books. What’s a compact definition of the flaw you’re talking about?

Imagine an organization building out a new website, and the site has 10 separate pages. And you’ve got 10 teams working in parallel to get this website live. Imagine that each team takes an average of six weeks to complete their page. Of course, you’re not finished until the last page is done, and every team is a little uncertain. So, the boss says, “When do we go live?” and you say, “Well, I don’t know. I don’t know how long team one will take, or team two.” The boss says, “Give me a number.” You’d be amazed how many people will say, “Well, on average, each team takes six weeks, so I would expect us to be done in six weeks.”

Guess what? There’s one chance in a thousand of being done in six weeks. Imagine that each team flips a coin to see if they come in over or under six weeks. You’re not done till the last team is done, so to finish in six weeks is a little bit like flipping 10 heads in a row.

The name of your organization is Probability Management, and it so happens that probability management is also a discipline. Tell us more about this discipline in the context of the “flaw of averages.”

The discipline of probability management represents uncertainties as data. And we don’t mean, here’s the mean of a distribution. We mean it represents the entire range of uncertainties. And it does it in a way that you can do calculations with the uncertainties. I don’t want to bore you with mathematical details, but it has taken a while to do.

It’s an open standard. It has involved a number of pretty smart people. So, there’s a whole technology stack built around this, and it allows us to cure the flaw of averages because we don’t use averages, we use uncertainties as represented by the discipline.

Banking is in the midst of turmoil as the string of sizable banks failing continues to grow, most recently with First Republic joining Silicon Valley Bank, Signature Bank and several others. What connections do you see between what we’re talking about here and what happened to those banks?

All these things involved uncertainties. The banks undoubtedly were running some internal stress tests. These were not standardized in the same way that the discount rate is standardized coming from the CFO. So, in many places, they’re still using averages. And of course, that masks risks and opportunities.

Did it surprise you that in Silicon Valley Bank’s case, the downfall couldn’t be traced to anything exotic? Instead, it seemed to be simply mismanaging assets and liabilities on the balance sheet, which strikes me as a banking 101 thing.

Ultimately, the solution to these things involves running a simulation. To clarify, this is analogous to shaking a ladder before you climb on it to paint the side of your house. The discipline of probability management in effect stores ladder shakes as data.

Now, let’s go to Silicon Valley Bank. They actually had two ladders. One was their assets that got badly dinged with inflation. The other was their deposits, which were coming from people who were venture-funded. As inflation went up and venture funding went down, suddenly these people had to take their money out.

So, let’s talk now about shaking two ladders. If you just shake them independently, there may be 1 chance in 10 of the first ladder falling down, and 1 chance in 10 of the second ladder falling down. The chance that both fall down is 1 in 100. But these two ladders were connected by a board. Both ladders were operating in the same interest rate environment. Silicon Valley Bank may well have been dutifully shaking each one of their ladders and saying, “No problem here,” without coordinating their results. If both ladders are tied together and either one falls over, they’re both going to fall over.

We’ve been talking about the issues that can arise when bankers seek to boil complex uncertainties down to a single number, and, in the case of the recently failed banks, some real-life ramifications of this kind of oversimplification. I want to ask you now about how to fix this. Aside from the board between the ladders, what other types of high-level advice would you give to banking institutions about how to avoid these kinds of modeling problems?

The real beauty here is that, if they avail themselves of the latest open technology, they’ve got all the people they need. They’ve got analysts, they’ve got bosses. The analysts often go to the bosses and try to tell them something statistical, and they trigger what I call post-traumatic statistical disorder, or PTSD, and the boss says, “Give me a number.” You’ve got the bosses, who’ve got to be trained to stop asking for a number. And you don’t need anyone else. In other words, you don’t have to go out and hire new people to start taking advantage of this new data-centric approach.

This interview was conducted by the BAI Editorial Team.

We offer actionable insights on other data and analytics topics that can benefit financial institutions in the BAI Executive Report, “The insights of data and analytics.”