The curious art of stress testing

So I’m down in Miami, lying on the beach, reading Snow White and trying to forget about New York’s brutal cold and the 90-point monthly drop in the S&P500. I’ve always loved Miami, provided I stay exactly two days and a half. I’ll leave the details of the trip for a lighter post but, as a prelude, I strongly recommend a visit, if you’re at all interested in the true, spectacular meaning of “ponzi scheme”, “real estate bubble”, “subprime”, “jingle mail” and “foreclosures of the week”… oh, and “silicone.”

Anyway.. the topic du jour is actually Geithner’s stress tests. I wasn’t planning on touching this potato, but, while at the beach, something surreal happened… I was approached by a woman who, a few minutes into our conversation, asked me whether I was from New York and… would I like a free stress test?!!

I should say I’m far more skeptical about Geithner’s stress tests than those free ones I was offered (which I did not take). And it looks like I’m not the only one, although many of the criticisms so far have centered on what may be a “secondary” problem—namely, that the “dark” scenario is not dark enough.

[As a refresher, the stress tests are supposed to test the resilience of banks’ capital under two macroeconomic scenarios. More details here.]

Undeniably, the dark scenario reveals some lack of imagination by those who conceived it. But there are many more problems with the stress tests before we even get there.

First, unless I’m missing something, we still don’t know how Geithner & co. are planning to treat the toxic assets in banks’ books. Now that’s an omission with a Big O! Stress tests are supposed to estimate the probability distribution of an institution’s losses under a “stressed” scenario. So a “stress-tester” would need to have a starting value of a bank’s assets; and a model of how the prices of these assets will behave under the scenarios considered.

We certainly lack the former. And I wonder… Is it because the tests will only cover the banks’ loan books (and not the trading books)? Is it because the Treasury plans to sideline the toxic assets for now, under the (ambitious) assumption they will be dealt with by Geithner’s one-trillion-dollar Public-Private Investment Fund (the “PPIF”)? Or is it because they plan to use banks’ own model-driven valuations to price these assets?

Whatever it is, the whole thing smells of “obscurity” and/or “partial coverage”… neither of which is good for instilling credibility in the government’s verdict about the health of financial institutions, post stress-tests.

Secondly, stress tests on individual institutions fail to capture the systemic implications of financial distress, e.g. due to contagion or feedback effects. We’ve already experienced “contagion par excellence” after the collapse of Lehmans. We’ve also witnessed (and continue to witness) the feedback effects of financial distress: Declines in asset prices have undermined banks’ capital position, forcing them to deleverage. This has meant further asset sales and, in turn, further asset price declines and so on. Alternatively, as banks’ capital is eroded, they tighten lending standards, prompting the distress in firms that need financing—which could of course lead to further losses for banks.

The lack of a(n adequate) systemic perspective in financial supervision was arguably one of the big supervisory failures that contributed to the crisis today. And yet, Geithner’s stress tests, to my knowledge, do not try to address it.

Third, it is unclear whether the tests will assess banks’ vulnerability to liquidity risks beyond the risks to their capital position. Once again, that would be a huge omission. Banks face tremendous liquidity risks today both on their assets and their liabilities. In the case of the former, they hold a whole lot of illiquid stuff that is currently impossible to sell without major haircuts. In the case of liabilities, banks’ increasing reliance on the wholesale market for their funding, and the dysfunctions of that market ever since the credit crisis hit, have made them vulnerable to the vagaries of investor sentiment.

True, Ben and his crew at the Fed have stepped in to neutralize this risk. But if the stress tests are meant to encourage investors to lend to those institutions that “pass” them (with or without recapitalization), leaving liquidity risks aside will not do the job.

Finally, a fairly pedantic, yet inevitable point: I mentioned earlier that a “stress-tester” needs, among other things, a model that relates the macroeconomic variables of the stressed scenario (e.g. GDP growth, unemployment, house prices etc) and the variables one needs to assess banks’ potential losses—e.g. probabilities of default, interest margins or the prices of, say, mortgage-backed securities.

Well, good luck with that! Not because such models (for what they’re worth) do not exist; they do, but they necessarily rely on historic data. Yet, the macroeconomic and market dislocations we have recently experienced may have caused a change in the models’ parameters. This means that whatever loss distributions are estimated may be off the mark!

I don’t want to throw the idea of stress tests out of the window. On the contrary. Stress tests are becoming an increasingly useful tool in the financial supervision process and there is a whole body of literature trying to devise ways to make them better.

But it’s important to understand their tremendous limitations: I encourage you to (re-) read the Bank of England’s July 2006 Financial Stability Report. I must say it’s more surreal than Barthelme’s Snow White! The report identifies on a qualitative basis almost each and every extreme risk that ended up materializing about a year later.

Yet, despite its state-of-the-art stress-testing methodologies, the BoE could only go as far as to acknowledge the difficulties in quantifying many of those risks! The risks it did quantify with the help of the stress tests were just a fraction of the problems that eventually emerged last Fall. Of course, the limitations of stress-testing are even larger when the scope of the tests is fairly narrow (as per the Geithner plan).

Ultimately what investors need is clarity over the hole in banks’ books. The downside risks from a severe macroeconomic scenario are (an important) part of that assessment. But there are many other burning components—notably the toxic asset issue—which, unfortunately, remain unaddressed.

I therefore fear that the stress tests, particularly in their “Geithnerian” form, will fail to rekindle investor confidence in the health of the financial sector. Worse… they won’t be for free!

Originally published at the Models & Agents blog and reproduced here with the author’s permission.

One Response to "The curious art of stress testing"

  1. Richard4691   March 2, 2009 at 10:31 am

    You could read it a little differently, starting from the question: What does Geithner seek to learn that he does not already know from having headed up the New York Fed? I think the anwswer is about the banks’ exposure to defaults on commercial and industrial loans and on commercial real estate loans, under weak and weaker macro-economic scenarios. Given two iterations, even prepared by the perpetrators, it will not be hard for an analyst to alter the assumptions for more stress. I think we can assume that Geithner knows what there is to know about all the stuff you mention that is being left out of the stress test. It’s been the focus of a lot of attention. But I haven’t seen any figures that indicate what kind of additional capital banks would need if say revenues fall 10% at firms whose C&I loans were predicated on 1.1 x EBITDA to debt service coverage. Nobody has yet talked about the extent to which banking in the last 10 years has migrated completely away from an asset coverage underwriting model to cash flow lending in which leverage is measured by multiples of EBITDA. Discussion of the reasons for this shift in underwriting technology are for another time. I want to make another point from a different angle: We have seen enough of the style of the Obama Administration to know that Geither’s pronouncements have to have been carefully scripted and I think we can also observe that they say exactly what they mean, but in a way that, as regards the banking crisis, is intended to forestall any kind of panic. Meanwhile, Wall Street is impatient to know how much more the government is going to pay or if they are going to take down the banks. Responsibly, what else can Geither do but try to get a handle on what ownership would really mean?