Thursday, February 25, 2010

A Tale of Model Risk

I saw the following dialog published years ago on the Internet. The exchange remains instructive through today regarding how naïve model risk creeps into enterprise analytics:
Clueless in Chicago (7/28/00)

Dear Dr Risk – We've got a pricing model that we use on a daily basis, hundreds -- sometimes thousands -- of times. Our regulator is pressing us for details of the model. Our problem is that we don't have any documentation to give. We know the model's good, 'cause we're still in business, but we can't document the fact. Can you help us figure out which model we're using.

Dear Chicago – Your bank is certainly not the only one using an undocumented "black box" to price options with millions of dollars of notional value. Once, Dr. Risk was faced with the problem of documenting a pricing application for which the bank had no write-up. They didn't even have a name for the model, such as Black-Scholes.

"No problem," said Dr Risk. "Let me see the source code. I don't care what language."

A few days later, the liaison had some bad news: "The source code isn't on the network. They were running out of disk space one day, a few years ago, and nobody had accessed the file for a few years, so they moved it onto magnetic tape and stuck the tape into an archive. We'll have to get it out of the archive."

A week later, the liaison had some [more] bad news: "The archive was getting full, and nobody had asked for that tape after five years, so somebody chucked it."

"No problem," said Dr Risk. "Let me talk to the developer."

A few days later, the liaison had some [really] bad news: "The developer left the bank a few years ago for a new job. Then he left that job. We can't find him. A guy who worked with the developer is still with the bank. He's not answering his phone. Maybe he's on vacation. We'll ask him and get back to you."

A few days later, the liaison had some [really, really] bad news: "The assistant didn't answer his phone, because he's recovering from a heart attack. He's supposed to rest and may be taking early retirement. I don't think he's going to be available. Got any ideas?"

Dr Risk said, "Let's see if I can match the output with something in my library. We don't have a whole lot of right ways to go, here."

Turns out the model wasn't difficult to match. It would be nice to see the guts of the application, to make sure it doesn't contain a Trojan horse, but that's not going to happen.

In your case, if you like, Dr Risk can try to reverse engineer your application. With a little bit of luck we'll find a perfect match for your app in our extensive library of models. Let us know if you want us to pursue this as a consulting project.
You can visit Dr Risk's blog through the link below.

Source: The Derivatives 'Zine

No comments:

Post a Comment