Exploring the “flaw of averages” at banks

Banking is getting more complex, and that greater complexity is creating more uncertainties and more risks.

Sam Savage from Probability Management joins us to make a case for why banking institutions should change the way they think about and deal with uncertainties.

A few takeaways from the conversation:

• Banks tend to oversimplify when gauging probabilities because they lack enough standardized data about key risk metrics to build more effective predictive models.
• To develop a more sophisticated approach to probabilities, his first bit of advice is to stop using averages because they is not robust enough given bank complexities.
• Banks should also bring top decision-makers, data scientists, statisticians and the IT department together to shape a more accurate and valuable predictive method.

‎

INTERVIEW TRANSCRIPT

Sam Savage, executive director at Probability Management, welcome to the BAI Banking Strategies Podcast.

Nice to be here, Terry.

Sam, “The Flaw of Averages” is the title of one of your books, so no doubt you can go deeper into that than our conversational format allows. But give us, if you could, a compact definition of the flaw that you’re talking about.

Imagine an organization that is building out a new website, and the site has 10 separate pages. There’s the registration page and the products page and the shopping cart page and the legal page. And you’ve got 10 teams working in parallel to get this website live because, of course, time is money. Imagine that each team takes on average six weeks to complete their page. But of course, you’re not finished until the last page is done, and every team is a little uncertain. It’s true they average six weeks. So the boss comes in and says, “When do we go live?” And you say, “Well, I don’t know. I don’t know how long team one will take, or team two.” And then the boss says, “Give me a number.” And you’d be amazed how many people will say, “Well, on average, each team takes six weeks, so I would expect us to be done in six weeks.” Well, guess what? There’s one chance in a thousand of being done in six weeks. The way to picture this is imagine each team flips a coin to see if they come in over or under six weeks. You’re not done till the last team is done, so to finish in six weeks is a little bit like flipping 10 heads in a row. This is not a small effect, and it impacts every business of which I am aware.

In what sort of ways does it affect them?

Well, it explains why everything is behind schedule, beyond budget and below projection. I just explained why everything is behind schedule. Here’s why everything is below projection. So you’re deciding how much of some new product to manufacture, and you know the average demand for this thing is going to be 100,000 units, so you produce 100,000 units. And if you sell 100,000 units, you’ll make $10 million. So should you expect $10 million? No. If the demand is below 100,000 units, you’re not going to make your money. Hey, but what if it’s greater than 100,000? You’re stuck. You produced 100,000 units, you’re capped. Why are things beyond budget? You plan for the average demand for some product, but you have to ship the stuff in air freight if you don’t have enough, or it spoils if you have too much, like with pharmaceuticals. So if the demand is exactly the average that you stored, then you have no operating costs. But if it deviates to the left or the right, if the demand is either greater or less than that average, you get dinged with the cost in either direction.

The name of your organization is Probability Management, and it so happens that probability management is also a discipline. So can you tell us more about this discipline in the context of the “flaw of averages”?

The discipline of probability management represents uncertainties as data. And we don’t mean, well, here’s the mean of a distribution. We mean it represents the entire range of uncertainties. And it does it in a way that you can do calculations with the uncertainties. And I don’t want to bore you with mathematical details, but it has taken a while to do. It’s an open standard. It has involved a number of pretty smart people. So there’s a whole technology stack built around this, and it allows us to cure the flaw of averages because we don’t use averages, we use uncertainties as represented by the discipline.

Banks are sophisticated enterprises. They’re complex in their operational structure. They deal with complex products and services. And the people who run these institutions tend to be pretty smart and savvy. I say this because what you’re talking about here doesn’t seem all that sophisticated on the face of it, so what do you think accounts for why statistical oversimplification, why it persists?

That is really a great question. And I think there are two primary reasons, one you’d be quite surprised at. But most people, like virtually all managers, have had a statistics course. And according to my informal studies, about 95% of them have what I call post-traumatic statistics disorder, or PTSD. They don’t want to know about this stuff, and it makes them feel dumb. The second part of this, though, is more profound. Let me give you an analogous problem here. You know what NPV is, net present value. Can you imagine that every project manager in a bank picked their own discount rate for their NPV? That would be ridiculous. I mean, someone who was losing money would pick a 1,000% interest rate, so it wouldn’t matter. And someone who’s making a small amount of money would pick a zero. There’s someone in the bank called the CFO who gets together, I’m sure with a committee, and they say, “Look, this is the discount rate we are going to use for all our NPV calculations.” We do not have an analogous thing for uncertainties yet. We have a name for such a person. This is a chief probability officer, and we have the open standard now for distributing that information. And so if you talk about the uncertainty of how many sales each salesman is going to make, that has to be stored somewhere for all the bank to use in a uniform manner, the way they use the discount rate. The sheer terror of statistics for most people, and then there’s the lack of standardized data.

So the industry, of course, is in the midst of turmoil as the string of sizable banks failing continues to grow, most recently with First Republic joining Silicon Valley Bank, Signature Bank and several others in that category. What connections do you see between what we’re talking about here and what happened to those banks?

All these things involved uncertainties. And the banks undoubtedly were running some internal stress tests and this and that. These were not standardized in the same way that the discount rate is standardized coming from the CFO. And so, in many places, they’re still using averages. And of course, that masks risks and opportunities. The flip side of a risk is an option. And if you don’t recognize the uncertainty, then you don’t see where options appear, you don’t see where risks appear. And I think it was really the lack of standardization.

Did it surprise you at all that in Silicon Valley Bank’s case, the downfall can’t be traced to anything exotic? Instead, it seemed to be simply mismanaging assets and liabilities on the balance sheet, which strikes me as kind of a Banking 101 thing.

Here’s what I think was missing. Ultimately, the solution to these things involves running a simulation. And just to clarify this for the users, this is analogous to shaking a ladder before you climb on it to paint the side of your house. Everybody does it. And by the way, the discipline of probability management in effect stores ladder shakes as data. Now let’s go to Silicon Valley Bank. They actually had two ladders. One of the ladders was their assets that got badly dinged with inflation. Another ladder had to do with their deposits, and the deposits were coming from people who were venture-funded, which meant that as inflation went up and venture funding went down, suddenly these people had to take their money out. So let’s talk now about shaking two ladders. The two ladders I just told you about, if you just shake them independently, imagine I had one chance in 10 of the first ladder falling down and one chance in 10 of the second ladder falling down. Well, the chance that both fall down is 1 in 100. But these two ladders were connected by a board. Both ladders were operating in the same interest rate environment. And so Silicon Valley Bank may well have been dutifully shaking each one of their ladders and saying, “No problem here,” without coordinating their results. So if both ladders are tied together and either one falls over, they’re both going to fall over, so it makes a much greater risk.

So, Sam, we’ve been talking about the issues that can arise when bankers seek to boil complex uncertainties down to a single number, and in the case of the recently failed banks, some real life ramifications of this kind of oversimplification. I want to ask you now about how to fix this. Aside from the board, which you talked about in your last answer, between the ladders, what other types of high-level advice would you give to banking institutions about how to avoid these kinds of modeling problems?

The real beauty here is that, if they avail themselves of the latest open technology, that they’ve got all the people they need. They’ve got analysts, they’ve got bosses. The analysts often go to the bosses and try to tell them something statistical and they trigger PTSD and the boss says, “Give me a number.” You’ve got the bosses, who’ve got to be trained to stop asking for a number. And you don’t need anyone else. In other words, you don’t have to go out and hire new people to start taking advantage of this new data-centric approach. There is this analogy with electrification. So in electrification, you’ve got light bulbs. We all know what light bulbs are. And you’ve got power generating plants, and you’ve got transmission lines. What makes electrification work is that everybody has one of these little three-prong plugs in their office. And if we did not have those standardized plugs, you couldn’t connect all these diverse groups together. So it turns out that the latest technologies do something analogous for probability, which I call chancification. And chancification does not involve hiring new people, but you end up connecting your statisticians and data scientists and analysts to the decision makers, through the IT systems. So all you need to do is start speaking the same language.

Tell us more about chancification. We’ve got the analogy of the plugs in the office. Tie it back to what you were talking about before in terms of the risks and the other issues that banks are facing here. How does chancification really stand to change their outcomes?

Sure. I think I’ll use a really simple mathematical example. I’m going to roll a pair of dice, and the boss says, “What number’s going to come up? Give me a number.” And I say, “Well, how big do you want it to be, boss?” Boss says, “Well, I want it to be at least eight.” “Great, boss.” So here’s what chancification does. It delivers to all the parties, it delivers 10,000 die rolls stored as data. And for two different dice, this could be in a column and a spreadsheet, two columns. So what you can do is add those two columns together, and now you’ve got the sum of two dice. And you can simply count the number of times it’s eight or greater or whatever the boss wanted. Then you’re supposed to divide by 10,000. So the chancification is instead of moving electricity around, we’re moving these parallel universes around, these ladder shakes. I mean, shaking a ladder, if you shook a ladder 10,000 times, 10,000 motions, you roll a die 10,000 times. A mathematically similar concept.

So are there banks out there that have embraced chancification? And if there are, do you have any indicators of how it’s working for them? And for banks that aren’t using it, which I’d imagine are probably most of them, what’s your elevator pitch to them on why they should?

Yeah, so let’s start with, are there banks out there who embrace chancification? For example, internally, in their derivative portfolio, outfits like Goldman Sachs have probably been doing this for 30 or 40 years. The point is that is completely internal and non-open and they’re not going to tell you what’s going on. But the idea of storing the die rolls, absolutely, that goes back to financial engineering and the insurance industry. I don’t know of banks currently using the open standard. The fact is it’s very new. I think the final version of our standard and software tools around it are just maybe a little over a year old. But for example, Kaiser Permanente apparently recently saved 20 million bucks by modeling their technology risks using this approach. And there’s a big defense contractor that has also been using it for a while. With the defense contractor, they’re worried about how long can you expect an airplane to fly before you have to replace a part. You can’t do that with average times to failure. It’s being used in a bunch of different organizations. And what’s required, if we go back to what do the banks have to do, they don’t need to hire new people. They have to be aware that the standard exists and that it’s easy to use in R, in Python, in Excel, on an abacus. But you do have to get, I think, three groups in a room together. You’ve got to get the decision makers, you got to get the statisticians and data scientists. I’m lumping them into one group. And then the IT personnel who’ve got to move the data around, and is it feasible to do this? The decision maker has got to be admonished for asking for a single number. They don’t need to anymore. And then the data scientists, of course, have got to think about how to structure it so you move the data around securely, and make sure that it has provenance, people know where it came from, that we don’t have everybody in the organization picking their own discount rate. No, we don’t want them picking their own versions of the uncertainties either. They have to go to the chief probability officer to get that.

So, Sam Savage, executive director at Probability Management, many thanks again for joining us on the BAI Banking Strategies Podcast.

Thank you, Terry.