Skip Navigation

Rethinking Intelligence: Interview with John Mueller and Mark G. Stewart

John Mueller is a political scientist at Ohio State University and at the Cato Institute. He is the author, co-author, or editor of 18 books and hundreds of scholarly and popular articles, including two books with co-author Mark G. Stewart: : Terror, Security, and Money: Balancing the Risks, Benefits, and Costs of Homeland Security; and Chasing Ghosts: The Policing of Terrorism.

Published: March 31, 2016

Inter­view Tran­script

John Mueller is a polit­ical scient­ist at Ohio State Univer­sity and at the Cato Insti­tute. He is the author, co-author, or editor of 18 books and hundreds of schol­arly and popu­lar articles. Mark G. Stew­art is Professor of Civil Engin­eer­ing at The Univer­sity of Newcastle, Australia. He has 30 years of exper­i­ence in prob­ab­il­istic risk and vulner­ab­il­ity assess­ment of secur­ity systems, has published more than 400 articles, papers, and reports. Together they have writ­ten two books, Terror, Secur­ity, and Money: Balan­cing the Risks, Bene­fits, and Costs of Home­land Secur­ity; and Chas­ing Ghosts: The Poli­cing of Terror­ism.

Mike German, a fellow at the Bren­nan Center for Justice at NYU Law School, inter­viewed Mueller and Stew­art on March 31, 2016. The follow­ing is an edited tran­script of that inter­view.

Q:  My name is Mike German. I’m a fellow with the Bren­nan Center for Justice’s Liberty and National Secur­ity Program. Today we’re very pleased to have Mark Stew­art, who is a professor of civil engin­eer­ing at the Univer­sity of New Castle, Australia. Not often in our discus­sions do we have some­body from the engin­eer­ing field. But that’s why it’s so import­ant that you’re here. And John Mueller, who is an old friend of mine, is a polit­ical scient­ist from Ohio State Univer­sity and also a senior fellow with the Cato Insti­tute in D.C. Both have published numer­ous works and this is their second effort together, called “Chas­ing Ghosts: The Poli­cing of Terror­ism.”

It’s a fascin­at­ing read, writ­ten with a lot of good humor, but seri­ous data analysis as well. It’s an attempt, long over­due, to bring reason and scientific rigor to an assess­ment of our law enforce­ment, home­land secur­ity and intel­li­gence efforts to counter the terror­ist. So maybe that’s where we should start, talk­ing about what is the threat. You start the book with a quote from George Bush, where he remin­isces about a meet­ing with the FBI a few weeks after 9/11 where they said there are 331 al-Qaeda oper­at­ives within the United States. What happened to those 331 oper­at­ives?

MUELLER: No one knows. They never showed up. What’s inter­est­ing is that Bush, writ­ing this book in 2010 or 2011, never goes back and tries to say what happened to those people. By 2005, the FBI, in fact, was saying that there are no al-Qaeda cells in the United States at all. And that contin­ues to be true. Actu­ally in 2002, this 331 number got bigger when intel­li­gence people were telling report­ers who have good in’s to the intel­li­gence community that there were between two and four thou­sand al-Qaeda oper­at­ives in the United States, ready to go.

It was a complete fantasy. Of couse, they might still be here, and they’re just being very quiet. Maybe they’re wait­ing for the apoca­lypse or they’re too busy watch­ing porno­graphy and doing drugs or some­thing. But certainly, obvi­ously, if they’d done anything, we’d know it. And/or the FBI or some­body would root them out. So some­how they were seeing ghosts. They were seeing all these people who were not there. And that contin­ues.                          

Q: And it’s not that there isn’t a threat. You actu­ally docu­ment in the book 62 cases.

MUELLER: Right. There’s more since the book came out. ISIS and everything.

Q: Tell us about that. You actu­ally have that on a separ­ate web-book.

MUELLER: Oh, yeah, one of the things that I’ve done, so you can’t blame Mark for this, is put together a book of all the cases of terror­ism in which Islamic terror­ists, since 9/11, have seem­ingly targeted or actu­ally targeted the United States whether they are based in the United States or abroad. So it does­n’t deal with cases of people going abroad to fight either with al-Qaeda, or Taliban, or ISIS, but only those focused on the inside. It’s up to well over 60 case now. Ohio State has very good honor students and they wrote these cases up, and I would edit them. Anyway, it’s published on the web as a web-book of up to about 830 pages.

Q: Let’s just briefly talk about the response. You talk about some of the data that the Wash­ing­ton Post brought out about ”Top Secret Amer­ica.” And if you break it down with the terror­ist plots that you were able to find, I think you say in the book that “the U.S. has created or re-organ­ized two counter terror­ism organ­iz­a­tions for every person arres­ted for plot­ting a terror­ist act in the U.S.”

MUELLER: The wonder­ful book called “Top Secret Amer­ica” by Dana Priest and Bill Arkin from the Wash­ing­ton Post that came out about 2010. Just tons and tons of inform­a­tion about how many people are work­ing counter terror­ism there are. One calcu­la­tion is that the appar­atus has launched far more covert oper­a­tions in the after­math of 9/11 than it had during the whole Cold War. There’s just this fant­astic amount of effort. So the issue in our book in many respects is, are they find­ing enough terror­ists to justify these fant­astic costs?

I’ll just give you one example. In fact, we have to update the book now, it turns out. A book called “The Threat Matrix” by Garrett Graff, a reporter, says that in any one day the United States is chas­ing down about 5,000 terror­ism lead­s—every single day. If each lead, on aver­age, takes two days to clear, they have chased over 10 million.

But Peter Bergen just came out with a book and he talked to another coun­terter­ror­ism agent who said that there’s actu­ally 10 thou­sand leads that they’re follow­ing up every single day. Every single day your tax money is paying for this. We certainly know how many they’ve found, because they brought to trial or found on lesser charges or whatever. And the total number of people that this appar­atus has come up with, terror­ists, is certainly less than a thou­sand. Prob­ably well less. And that would include ones also trying to go over­seas to fight. So that leaves a fant­astic number of false posit­ives.

Q:  When we talk about these tips, there’s also a policy choice that the FBI in partic­u­lar made that you call the 9/11 Commis­sion Syndrome. Why don’t you tell us about that?

MUELLER: After 9/11, my name­sake, Robert Mueller, the Director of the FBI mandated that every terror­ism lead has to be followed up. Before, what they would do is they would triage. Some of this stuff is real garbage. In fact, I can give you a case of one lead.

It’s a “threat” from the Phil­ip­pines to attack the United States unless black­mail money was paid. So they checked it out, and if you ask what was that threat, well, it’s an email they got saying, “Dear Amer­ica, I will attack you if you don’t pay the 999999999 dollars. Mwah ha ha ha.” That’s the threat. But they had to follow it up. They got the Phil­ip­pine author­it­ies to go and visit his parents. So far, he hasn’t been heard from again.

I also talked to an FBI guy, a top guy, at a confer­ence less than a year ago. I wanted to make sure that was still the case. So I said do you still follow up every single lead, every single tip includ­ing things like this “mwah ha ha” one? And he said – I didn’t put this word in his mouth, it came out of him, twice in fact. He said, “Unfor­tu­nately, yes.” So even in the FBI there’s some real­iz­a­tion that they’re expend­ing terrific amount of man power, $3 billion a year, chas­ing a vast number of false posit­ives. You can spend your whole career in the FBI and never find a single thing chas­ing false posit­ives constantly.

Q: And we actu­ally have, I mean part of the appar­atus that we built, this counter terror­ism appar­atus, includes this warn­ing system that collects these poten­tial tips. The mass surveil­lance programs the National Secur­ity Agency engages in, but if you’ve ridden the subway up, you’ll see all over the place, “See Some­thing and Say Some­thing.” What does that do?

MUELLER: Well, it gener­ates a huge number of tips. It’s “If you see some­thing, say some­thing ®” — you can’t use that phrase without getting the permis­sion of the New York Police Depart­ment. However, if you say “See some­thing, say some­thing,” you don’t have to get their permis­sion.

A reporter from the New York Times asked them, how many terror­ist arrests has this led to? The answer a few years ago was zero. They’ve gone through tens of thou­sands of these tips since the program star­ted. The federal govern­ment is spend­ing hundreds of thou­sands of dollars every year simply to publi­cize the “If You See Some­thing, Say Some­thing” banner. And no one is saying that it does­n’t do any good, noth­ing’s come from it. And one of the things that we’re trying to do in the book is try to eval­u­ate how success­ful these programs have been and then compare them to what their costs are to see if they’re cost effect­ive over­all. Mostly the programs don’t come out look­ing too good.

Q: When we talk about costs, obvi­ously here at the Bren­nan Center, at it’s Liberty and National Secur­ity program, we focus a lot on the cost to our indi­vidual privacy, the cost to civil liber­ties, the cost of govern­ment over­reach in prosec­u­tion, invest­ig­a­tion. But you suggest in the book that that’s not enough. That that won’t convince people who have a fear of terror­ism that it’s greater than the threat actu­ally poses. Talk a little bit about the public fear of terror­ism and whether it’s based on real­ity.

MUELLER: Well, I guess the main message from the book for this group is that you can’t only say that these meas­ures have privacy concerns. You also have to look at how much good have they done. You can’t let Dick Cheney say that coun­terter­ror­ism and torture, and the other coun­terter­ror­ism things have saved thou­sands, now he’s saying hundreds of thou­sands, of lives. Then you say, yeah, but they’ve intruded on people’s privacy. It’s not a very effect­ive argu­ment.

So what you have to do, basic­ally, is look at what is the threat? And how much are these secur­ity meas­ures basic­ally redu­cing it? That’s the main thing we’re trying to do using certain stand­ard meth­ods of cost/bene­fit analysis.

It is also related to litig­a­tion, judi­cial decisions. For example, in Decem­ber 2013, there are two decisions about Snowden, the metadata program, and whether it’s consti­tu­tional or not, or legal or not. One decision said it was not legal and mentioned in the judge­ment that the metadata program hasn’t caught anybody. It was supposed to be just a legal judge­ment. But it uses the program’s lack of success as ammuni­tion to say it isn’t legal.

The other judg­ment starts out saying we live in a deeply danger­ous world and NSA tells me about how zillions of people have been caught and so forth, and then concludes that the program is legal. So the legal judge­ment is being inflec­ted by whether the program is effect­ive or not. This issue is one that has to be dealt with directly.

I looked at all the Front­line programs about the Snowden thing, and nowhere in any of those programs did they talk about whether the metadata program or any of the NSA programs actu­ally are success­ful. You can’t just talk about the cost, you have to talk about the bene­fits. If there are no bene­fits and some costs, then obvi­ously it’s a no-brainer. But you can’t deal without that. Also, the movie, the Snowden movie, basic­ally never talks about whether these surveil­lance meas­ures have been effect­ive.

You can’t simply say it’s an incur­sion on privacy, that it’s unreas­on­able search and seizure. People then say “well, it saved a hundred thou­sand lives, that proves it wasn’t unreas­on­able, right? They caught a lot of people, so that sounds damn reas­on­able to me.” Unless you’re able to deal with that argu­ment, which this book tries to do, you don’t have as much to stand on it seems.

Stew­art: And there’s a lot of worst case think­ing in this area. So if you can imagine it, then the voters believe, it can happen. That there­fore leads to very risk adverse decision making, because they’re worried about the next attack can be cata­strophic, thou­sands of lives could be lost. I’ve heard senior offi­cials in Australia say that the next attack in Australia would be cata­strophic to the coun­try. Given we’ve only had two, maybe three in the last fifteen years, and in each one, one person was killed. So instead of look­ing back and saying …

MUELLER: And one was killed by the police, too.

STEW­ART: Yes, that’s four. But no one actu­ally looks back and says, what is the data? I mean, there’ve been plots against the New York subway and other transit systems, and the author­it­ies say, “well we stopped this plot and saved tons of lives.” Well, the only time there’s another attack on the subway before was in London, and that cost about 40 lives, not hundreds of lives. So they’re always exag­ger­at­ing the threat all the time. If you think you’re in a state of war, then that justi­fies the loss of a lot of civil liber­ties.

Q: Let’s talk about how we do risk assess­ment every­where and obvi­ously…

MUELLER: That’s Mark’s thing.

Q: …your contri­bu­tion to this effort. It’s hard to imagine. How the govern­ment would like to have this argu­ment is: “we can’t afford to have even one terror­ist slip through the cracks.” And that justi­fies the expendit­ure of billions if not tril­lions of dollars. How does some­body who works in risk analysis look at that?

STEW­ART: Well, that would make sense if you’re think­ing that one terror­ist could kill ten thou­sand people. There is data that most terror­ist attacks in the west kill only one or two people. There are excep­tions. But they’re not going to threaten the exist­ence of a nation state. They are not going to be exist­en­tial. So when you do risk assess­ment, and that’s why I’m involved in this because I’ve done a lot of risk assess­ment in natural hazards: how high should we build a levy bank for floods, or how strong should you design a build­ing against cyclones. You never aim for zero risk, because that’s just impossible. There’s always that balance between how much you spend and what safety to get. You try to find that balance, recog­niz­ing that perfect safety just can’t be achieved.

So the first step, you would ask your­self, what risks do you think are accept­able, or you’d toler­ate? For example, when you’re look­ing at nuclear safety, envir­on­mental legis­la­tion, things which are pretty emotive issues, the general consensus is that soci­ety tends to accept the risk of death of no more than one in a million per year, which is a bit lower than the risk of being struck by light­ning, much, much lower than the risk of being killed in car wrecks, or some­thing like that. So that one in a million has been a bench­mark across a whole range of risks.


Q:  And there are a whole range of risks. Not just natural disasters or crim­inal acts.


STEW­ART: Yeah, and that’s not where we might wind up, but that’s a good start­ing point. If the risk of death is like one in a thou­sand per year, then every­one would agree that that’s far too high and you must do some­thing. So there­fore, the first step would be, we’re not as inter­ested in terror­ism, what is the risk? What is the risk of being killed by a terror­ist?


MUELLER: That’s the basic ques­tion, and it’s never asked.


STEW­ART:  And it’s not a diffi­cult ques­tion to answer. I could do it in about half a day. It wasn’t really diffi­cult because there aren’t that many attacks so it’s fairly easy to do the calcu­la­tions. You don’t need to write a large computer program and get a PhD on this, it’s just basic stuff. Maybe don’t put that in the film. [laughter]


Yeah, you want to keep a job.


MUELLER: He slaves for the cause.


STEW­ART: So if you look at the last forty years in the states, includ­ing what happened on 9/11, the annual risk of being killed by a terror­ist in the U.S. in the last forty years is about one in four million.


MUELLER: Per year.


STEW­ART: Per year. If you look at the risk of death from terror­ism post 9/11, in the U.S. it’s one in 90 million per year in the US. If we look at Australia, it’s about one in eight million per year, and that includes the attacks on Bali in 2002, because that was like on our door­step, so. If you look at Australia post 9/11, the annual risk of death is about one in 15 million. They’re very small numbers. John and I present at lots of confer­ences and because of my back­ground in risk, I present at risk confer­ences, partic­u­larly secur­ity risk confer­ences. There’s two or three days, every­one talk­ing about risk. They’re waving their arms about, you know threats are evolving, cyber and everything else.. I am the only speaker to quantify, to put out a single number, just the numbers I just discussed. And I’ve had people amazed. They’re quite surprised.


MUELLER: They’ve never heard it before.


STEW­ART:  The whole idea of accept­able risk, they haven’t really thought about that notion before. To me, those numbers are a start­ing point for a discus­sion. I would prop­erly inter­pret those numbers to say the risks are very low. So it shows the police and counter terror­ism meas­ures in the West, is work­ing pretty well, other­wise the risk would be much higher, there’d be a lot more success­ful attacks. On the other side I’d argue it prob­ably shows maybe the threat isn’t as high as what we think it is because there aren’t that many people really plan to commit acts of terror­ism. And the true answer is prob­ably a bit of both. And there could be other reas­ons as well. But that’s the point of having a dialog and a discus­sion about what’s work­ing, what isn’t work­ing. You can look at the data in more detail to see what years are worse than others, maybe linked to differ­ent changes in govern­ment policy. You can delve into it in much more detail and get a lot more insights from it. We just don’t see that happen­ing.


Q: The govern­ment might come back and say all you have to do is turn on the news and you’ll see that terror­ism is actu­ally an increas­ing phenomenon. So if you go back 40 years, maybe it wasn’t as bad and now it’s much worse so there­fore justi­fies these expendit­ures.


MUELLER: Yeah, that’s not partic­u­larly true. If you look at the United States, there was a lot of terror­ism in the seven­ties and eighties, so the actual numbers… The same with Europe. There’s terror­ism because of the ETA in Spain and IRA in England.


STEW­ART: It begs the ques­tion that should be asked. If govern­ment came back and said please explain this. That’s great, then we can discuss it and say differ­ent time peri­ods went up and down and these are some trends, what we’re seeing, what we’re not seeing. You don’t want to rely too much on just a single metric, but it helps inform what poten­tial solu­tions should be. It’s really just part of a discus­sion. If you’re going to spend hundred and twenty billions dollars a year in the U.S. on domestic counter terror­ism, you’d think you’d have that sort of discus­sion.


Q: Right.


MUELLER: And in fact that’s a tril­lion dollars, more than a tril­lion dollars since 9/11. I’ll give you just one example, it’s actu­ally two examples. They did the same thing twice. We were at a confer­ence on aviation. So they sent these TSA types there. I simply asked, “the only way to make airplanes completely safe is to ground the airlines. I assume you don’t want to do that. So there­fore, you have to take a certain amount of risk.”


Then I said basic­ally, “what’s an accept­able risk?” Suppose if you get on an airplane, your chance of being killed by a terror­ist—in­clud­ing 9/11 in the calcu­la­tion by the way, which is of course a special outlier­—is one in a million, is that accept­able? How about one in five million? How about one in 40 million? How about one in 80 million, which is basic­ally what it is, includ­ing 9/11. If you get on an airplane, your chance of being killed by a terror­ist is about one in 80 million per flight. Is that accept­able? They look at me like—they have no answer to it because they’ve never even considered the ques­tion.


That’s where they should really start. I mean the prob­lem is you can say is that safe enough? How safe do you want it to be? Maybe it’s not safe enough. Maybe you want it to be one in 130 million. If you want it to be zero, however, there’s only one solu­tion.


STEW­ART: And if you want to start, and that discus­sion ends with how does the system work? In engin­eer­ing, they like to have models about how build­ings behave. The first thing is, let’s try to repres­ent how the system works, as best we can. To see which factors influ­ence the outcomes. We’re not seeing any of that really happen­ing at all. For example, in the aviation area, the focus is really on how to protect planes and airports partic­u­larly. They’re not really think­ing about what are some of the oppor­tun­ity costs, right? And as John alluded to, driv­ing is a thou­sand times more risky than flying. Flying is extremely safe. Driv­ing is one of the most danger­ous activ­it­ies we can do.


So directly after 9/11 when people were scared to fly and there was so much emphasis on secur­ity at airports, and big queues, and it could take you an hour to get through the check­point, people dropped off. “Bugger this, I’ll drive.” So after 9/11 there has been a clear spike in traffic deaths, between 200 and 400 per year. This isn’t our work, this is the work of others. That put down to more people decided for shorter destin­a­tions, it’s more conveni­ent to drive.


So here you have a public policy that encour­ages people to drive rather than fly that’s cost­ing lives, not saving lives.


Q: So that needs to be part of the cost element in the risk analysis?


MUELLER: The irony is that it’s there in the rest of the govern­ment. For example, you have a safety meas­ure, you say “let’s require there be seat­belts in the back seats of cars.” Well, maybe that’s a good idea, maybe not. We have to talk about it. The first thing you’d want to do is find out how many people are in back­seats and get killed in auto­mobile crashes. We have tons of data on that. If it turns out like three people a year are killed in crashes in the back seat, you prob­ably don’t want to spend a whole lot of money deal­ing with it. You can spend the money on some­thing else. They did decide, finally, to require them in cars. Because there must have been enough deaths to justify the meas­ure.


However, they decided that new cars had to have seat belts in the back, but not old cars. Now, one possib­il­ity is they made a moral calcu­la­tion. They said people who drive in old cars in the back seat are expend­able. [laughter] We don’t give a damn about them. Whereas, people in the back seat of new cars are much more valu­able.


Q: That was the auto industry’s take.


MUELLER: Obvi­ously, the reason for the differ­ence was that the cost of retro­fit­ting is very high. So they said it’s not worth protect­ing people in old cars because it costs too much to protect them. It’s much cheaper in new cars because if you’re build­ing a car, you can get it all built into the design plan. And once it’s done, it’s prob­ably a fairly marginal increased cost. So what they’re saying is people who drive around in old cars in back seats, too bad. They made that decision because they don’t have an infin­ite amount of money.


The thing that’s really ironic about this is that the Depart­ment of Home­land Secur­ity does good risk analysis of natural disasters. . Tornadoes some­times come through Oklahoma and there are people living in the trailer parks who can’t get to shel­ters because trailer parks don’t have shel­ters. Maybe we should build shel­ters for them. So they’ve done analysis. How frequently would big tornadoes come through and how many people would is save if they enough warn­ing and could get into these shel­ters? How much do the shel­ters cost? And they may or may not decide to build the shel­ters. That’s the kind of analysis that should be done. And DHS is state of the art for natural hazards, but basic­ally not at all, as far as we can see, for terror­ism.


STEW­ART:  A cost/bene­fit analysis is stand­ard proced­ure for a whole range of private sector and public sector decision making. It’s actu­ally required. I mean the Australian govern­ment, the U.S. govern­ment, the Office of Manage­ment Budget, the GAO, all mandate that any new govern­ment regu­la­tion must be accom­pan­ied by a cost/bene­fit analysis, every one. And most coun­tries in the world do that.


So when we began think­ing about this we were okay, surely it’s been done. Surely the DHS or the TSA have looked at this. And we can’t find a single example of the cost/bene­fit analysis on any regu­la­tions that they have mandated. The closest we’ve got was for the full body scan­ners, the AITs. There must have been 30 to 40 pages look­ing at the cost. They’d worked out how much does the power cost? How much does each employee cost? Enorm­ous detail on the cost­ing. Then bene­fits was one sentence that said bene­fits are too diffi­cult to quantify at this stage. That was it. So it was all on the cost, which is the easy bit.


Q: So what are the imped­i­ments to a more rational approach within govern­ment? You talk about the threat matrix and the pres­sure that puts on policy makers involved in counter terror­ism.


Mueller: We deal with that in the last chapter of the book concern­ing respons­ible counter terror­ism policy. It seems to be the way to look at this is to look to the Consti­tu­tion for a second here. A very popu­lar docu­ment in this room I imagine. Prac­tic­ally the first words in the Consti­tu­tion are “preserve domestic tran­quil­ity.” If you read Hobbes, obvi­ously, that’s why people have govern­ments. So the main reason for govern­ment, the found­a­tional reason, certainly one of the found­a­tional reas­ons for having govern­ment at all is to provide for public safety, domestic tran­quil­ity. People want to be safe. They’re will­ing to pay. They’re will­ing pay in civil liber­ties. They’re will­ing to pay in taxes and so forth. They want to be able to walk down the street and not get hit over the head. They want their chil­dren to be able to get to school safely.


The prob­lem is that govern­ments also have a finite budget. If you have an infin­ite budget, everything changes. But they only have so much money to spend. They can’t prevent every possible hazard from happen­ing. They have to make unpleas­ant choices. To do that irre­spons­ibly, which is basic­ally we’ll spend a billion dollars to save this life, even though it only costs a hundred thou­sand dollars to save other lives, is basic­ally irre­spons­ible and repre­hens­ible.


It is true that public opin­ion is extremely uptight about terror­ism. We have a whole chapter in the book on that. And we’ve done some recent stuff as well on it. As Cass Sunstein, the major risk analyst and lawyer at Harvard Law School points out, it’s perfectly reas­on­able for govern­ments to pay atten­tion to people’s feel­ings, and concerns, and so forth, but that does­n’t mean that the decisions should be made on the basis of their mistakes. And if they have a mistaken idea about how bad a hazard is, you should­n’t give into it. It should be, as Mark has put it, risk neut­ral on that.


And that’s not happen­ing. They’re basic­ally doing things like let’s put the AITs in. We know exactly what they’re going to cost. We don’t know if there’s any bene­fit at all. But we’ll just do it. Or a guy gets on with a bomb in his shoes so there­fore we’ll say every­body has to take off their shoes. So it’s some­what manage­ment by in-box. It’s basic­ally very irre­spons­ible. What they’ll say is, as Mike sort of sugges­ted, well if I don’t do this, I’ll lose my job. First place, it’s not clear that’s true. There was a big attack in San Bern­ardino, but I don’t know that anybody that lost their job in the police or anything. Or in Belgium there were big screw ups in police depart­ment it looks like. But people aren’t losing their jobs. It’s not clear that if some­thing happens you’re going to lose your job.


But the issue, it seems to me, is that if you’re not will­ing to make decisions that might be job threat­en­ing, you should­n’t take the job in the first place. So I have very low patience for the idea that I’ll lose my job. That’s actu­ally corrupt. It’s steal­ing money. I’m getting money to send my kids to college by making decisions which actu­ally reduce safety, do not increase safety in the most effi­cient way. In other words, people are dying because of my decisions. That’s not differ­ent from taking money out of the till it seems to me. If a solider signs up and then someone starts shoot­ing at him and he says I didn’t sign up to be a soldier to get shot at. They should say you should have thought of that before you took the bloody job. Or a fire­fighter who refuses to go into a burn­ing build­ing.


Simil­arly, it seems to me, if you’re not will­ing to make a decision, and you enter a job that might require making job-threat­en­ing decisions for the mission of the job, which is public safety, the reason you’re there, you should­n’t take it at all. There’re a lot of safe jobs. You can be a plumber, a college professor, all kinds of things where you don’t have to worry about that stuff. [laughter] So it strikes me that it’s not a good ration­al­iz­a­tion at all. And basic­ally an immoral one.


Q: I like the way you put in the book that there’s illu­sion and delu­sion within the intel­li­gence community. The illu­sion part of that is the threat matrix that you have this. But there’s also delu­sion. You quote George Tenet, former CIA director as saying that even years later, he thinks their assess­ments about the numbers of al-Qaeda cells inside the United States were accur­ate, even though as you’ve sugges­ted, they’ve never come out. Often through­out the book, you go through where FBI or NSA offi­cials are testi­fy­ing about the nature of the threat or the effect­ive­ness of their counter meas­ures that later prove to be untrue. Do you think that they are delud­ing them­selves about the threat? How do you assess it?


MUELLER:  Some of our big prob­lems is A, they’re not stupid and B, they’re not evil. It would be so easy if they were stupid or evil. Instead, they are dedic­ated public servants over­all. I think Mark would agree with that. They’re respons­ible. They’ve listened to what we’ve said, and they ask intel­li­gent ques­tions. They say, boy this is really inter­est­ing, we have to think about this. And then, of course, they never invite us back. So there’s a flaw in the system there some­what.


I think they basic­ally do believe it. They’re scared. The threat matrix that Mike has mentioned is that after 9/11, the idea is we should have this big piece of paper which basic­ally has every threat. Actu­ally, the fact that they call it a threat as opposed to a lead or a tip is inter­est­ing. They call it a threat. So like this guy who said I’m going to blow up, the mwah ha ha guy. And it goes into this threat matrix. And it goes to the head of the CIA, head of the FBI, and ulti­mately the Pres­id­ent every day.


Let me give you a quote from Jack Gold­smith here. He quotes George Tenet. And he talks about read­ing this stuff every day. Spend­ing an hour or some­thing look­ing at it, and there’s all these things that are really horrible. He said, “Virtu­ally every day, Tenet says, you would hear some­thing about a possible impend­ing threat that would scare you to death.” And then Gold­smith from Harvard comes in some­what later. He’s sort of not part of this, but he’s sympath­etic. He says, “That captures the atti­tude of every person I knew who regu­larly read the threat matrix.”


Every day you’re read­ing these things and some of these things are really horrible. There’s going to be a nuclear weapon going off in Manhat­tan and so forth. He concludes, “The want of action­able intel­li­gence”—that none of this intel­li­gence was action­able, basic­ally, because there was noth­ing there—“combined with the know­ledge of what might happen, produced an aggress­ive panic atti­tude that assumed the worst about the threats.” This is what Mark was talk­ing about, the exag­ger­a­tion of threats.


So it has a sort of psycho­lo­gical impetus. Then the other thing that Mike mentioned is that when Tenet came out with his auto­bi­o­graphy in 2007, he said my gut tells me, my oper­a­tional instinct tell me, I have no evid­ence, but there must have been a bunch of other al-Qaeda types infilt­rated at the time of 9/11, before 9/11. There’s tons of evid­ence that that’s simply not true, includ­ing testi­mony from Khalid Sheikh Mohammed, for example. He also repeated it on a 60 Minutes show. So what he’s saying is I’m scar­ing you to death based entirely on noth­ing, my oper­a­tional intu­ition. He does say that and you might say, well his intu­ition might be espe­cially good. But I doubt it. Of course, that’s 2007 and none of those people have shown up in the inter­ven­ing near decade.


Q: And when the govern­ment offi­cials describe the terror­ist threat, they say it’s evolving, that’s ingeni­ous, that it’s conniv­ing. But when you analyzed the 62 cases that were present of plots in the United States, what did you find about the char­ac­ter of the terror­ist master­mind?


MUELLER: We have a chapter on the myth of the master­mind. This is from the Depart­ment of Home­land Secur­ity National Infra­struc­ture Protec­tion Plan of 2009. This is a grand plan about how they’re going to protect people and so forth. It would be really import­ant to assess how good the enemy is. This is their entire state­ment about this, only two sentences, rather long ones though. Maybe it’s only one sentence. It’s only one sentence, very long sentence.


 “The enemy is relent­less, patient, oppor­tun­istic, and flex­ible, shows an under­stand­ing of the poten­tial consequence of care­fully planned attacks on economic, trans­port­a­tion, and symbolic targets, seri­ously threatens national secur­ity, and could inflict mass casu­al­ties, weaken the economy, and damage public moral and confid­ence.” Now, if the next sentence said, “on the other hand most of them are knuckle­heads,” at least you’d have some sort of balance. But that second sentence is never there.


So when I’m assign­ing the students to do this the cases, I said one of the things that I want you to do is tell me what these guys were like in these 60 or 70 cases. What did they do? What did they smell like? Where’d they come from? Were they on drugs? Were they crim­in­als? Were they college profess­ors? Whatever. Just where’d they come from, what’d they smell like? What kind of adject­ives would you use? In the case stud­ies, these are the kind of adject­ives that come up repeatedly: incom­pet­ent, inef­fect­ive, unin­tel­li­gent, idiotic, ignor­ant, inad­equate, unor­gan­ized, misguided, muddled, amateur­ish, dopey, unreal­istic, moronic, irra­tional, fool­ish, and gull­ible.


Basic­ally, a summary by Brian Jenkins a counter terror­ism guy at Rand who has been in this field for 50 years. He says, and it’s a really good summary: “Their numbers remain small, their determ­in­a­tion limp, and their compet­ence poor.” But you’d never know that from the DHS state­ment.


Q: They would argue though, number one is that it does­n’t take a genius to do horrible damage.


MUELLER: Lee Harvey Oswald was one case.


Q: So one of the concepts that they have is this idea of radic­al­iz­a­tion. That once some­body artic­u­lates a partic­u­lar ideo­logy, that they are on a path to viol­ence that must be inter­dicted with all sorts of differ­ent means. And the FBI in partic­u­lar using inform­ants to move those people toward the terror­ist attack. What does the data show about that?


MUELLER: They’re pretty good at infilt­rat­ing inform­ants. The consid­er­able major­ity of the cases had FBI oper­at­ives as part of the plot. They weren’t just watch­ing it, but they actu­ally joined it and play acted. And in some people’s minds, entrapped—but not neces­sar­ily in a legal sense, I guess.


I also asked the students when they’re doing the case stud­ies, tell me what motiv­ated these guys. Where did it come from? They use the word radic­al­iz­a­tion frequently, but over­whelm­ingly, the radic­al­iz­a­tion, at least in these cases, is because of hostil­ity to Amer­ican foreign policy. One of the guys in the Boston Mara­thon case wrote on the inside of the boat. It says, “You’re out to destroy Islam. You’re attack­ing, you’re killing my people. Your soldiers are killing and raping Iraqi women.” Etcet­era.


In other words, what they think is that what’s been happen­ing in the Middle East, that’s their main motiv­a­tion, outrage. It’s about a drone strike. That guy Zazi from New York, he’s an Afghan Amer­ican, and he read in the news­pa­per about  either a drone strike or maybe just an air strike, that killed a whole bunch of Afghans.  That ticked him off. I’m not justi­fy­ing their reac­tion neces­sar­ily. I’ve been pretty intensely hostile to the Amer­ican foreign policy as well, but I’m not going to throw bomb about it. They say repeatedly, that’s what motiv­ated us. And it rarely makes it into the press. Just like the degree to which terror­ism presents a threat, the expec­ted number of how many people are going to be killed per year and so forth does­n’t enter into the press.


Q: Or in govern­ment radic­al­iz­a­tion stud­ies, the govern­ment funded radic­al­iz­a­tion stud­ies, they often don’t mention those elements of the process. Let’s talk a little bit about the press and how that molds public opin­ion around these issues.


MUELLER: The press is basic­ally acting ration­ally. If you talk about terror­ism, people look at the story. If you can get ISIS into any kind of story, that’s really great because people are going to read it. I think they’re acting ration­ally in the sense that they’re trying to sell their product. They want it to have people not turn to another chan­nel. They want to have people click on the story. Click bait. So they basic­ally feed into it. I think it’s irre­spons­ible in the sense that they don’t say, “well we just had this happen, but the chance of this happen­ing is one in 80 million” or some­thing like that. I think they play to it and they exacer­bate it perhaps, but it’s really does seem to be bottom up. The people are just plain scared. That’s what happened when I did this analysis of public opin­ion.


I can summar­ize that. I was some­what surprised at the results as I thought there would be an erosion of fear over the years since 9/11. So I looked at 20 or 30 public opin­ion trend lines since 9/11. There are two patterns. One pattern,  is shown in the ques­tion “do you think there’ll be another attack killing large numbers of people.” Not just an attack, but killing large numbers of people in the near future. That’s been asked consist­ently since 9/11. In Septem­ber 2011, the percent­age of people saying that it was likely or some­what likely was about 70 percent. Currently, it’s about 70 percent.


It’s gone up and down a bit. It’s a bit higher now because of ISIS. But even before ISIS, it was still about 70 percent. No change at all. Despite the fact that we got Bin Laden. Despite the fact there’s been no more 9/11s. Despite the fact that up until Paris, there was a ten year period of no major attacks in Europe. Despite the fact that the chance of being killed is one in 4 million or one in 80 million. It’s still there.


The other pattern is there’s a spike up at 9/11, and then it came down, but stayed the same. There’s one ques­tion, it’s really quite a good ques­tion, “Do you worry that you or a member of your family could be harmed by a terror­ist?” So it’s a nice personal ques­tion, it’s not complic­ated. Do you worry about that? When Timothy McVeigh did his terror­ism, that’s when they asked the ques­tion first, 1995. He blew up the Federal Build­ing in Oklahoma City. It stood at about 40 percent. Forty percent of people said they worried some­what or very. Then it went into real decline in the next few years, and then they stopped asking the ques­tion. There was obvi­ously a shock by the Oklahoma City thing, but it was wear­ing off over the next two or three years.


Then at 9/11, it went from about 20 percent up to 60 percent. By the end of the year, it had gone back down to about 40 percent. But it hasn’t really declined since that time, even before the rise of ISIS. It goes up and down: if there’s a bomb­ing in London it spiked up and so forth. But there’s been basic­ally no erosion. If you ask people do you feel safer than before 9/11, there’s been no change, basic­ally. If anything, people feel a little bit less safe. If you ask who’s winning the war on terror­ism? About the same over all that time.


I think it’s fair to argue that the media are not hyping nearly as much as they did in the first years. Nor are public offi­cials. You’ve got Obama, who actu­ally said, a year ago or so, that ISIS does not present an exist­en­tial threat to the United States. The pres­id­ent of the United States actu­ally saying some­thing like that. Banal and obvi­ous, it might be, but other offi­cials have been saying that for a long time that it is an exist­en­tial threat. Anyway, the general amount of alarmism coming out of the govern­ment is, I think, defin­itely declin­ing from the early years when they were hyster­ical all the time. There was going to be a big attack before the 2004 elec­tion, there’s going to be a 9/11 in the next few minutes. There’s going to be nuclear devast­a­tion one place or another within the next ten years, and so forth. You don’t see that kind of offi­cial alarmism as much, I think it’s fair to say, as before. Nonethe­less, public opin­ion hasn’t changed at all.


STEW­ART: And the media are not that crit­ical. They love to bandy about the word “master­mind” and “the genius bomb maker” and all these other things. Yet, if you were to treat terror­ism a bit like crim­in­als, most crim­in­als we know are not really smart. They get caught. That’s why so many movies about bank heists that go wrong because it’s really comedy.


The other thing is that when there’s a plot that’s been foiled, the media report what the terror­ists planned to do, which is you know, toppling a build­ing, taking out a subway, not what the capa­city actu­ally was.


Q:  And not just topple the build­ing, but have the build­ing create a tsunami.


STEW­ART: Right, it’s going to slide down a hill. But the media are not very ques­tion­ing about things. So you see on TV when they inter­view a counter terror­ism exper­t—­like after Brus­sels, this expert was saying London and Berlin should be worried and this and that. And the inter­viewer was just lapping it up. Yet every time I give an inter­view about the work that we’re doing, they’re saying all those risks, they can’t be that low. How did you get that? What’s this? What time period? They’re very partic­u­lar. They’re trying to find a hole in our argu­ment. If I was to say we should be worried about terror­ism and we should be protect­ing Sydney and Melbourne and Canberra, then they’ll just lap it up. It would­n’t be partic­u­larly crit­ical.


Q: Doing a proper risk assess­ment is half the story, right? The other half is then look­ing at your enforce­ment meth­ods and eval­u­at­ing them. So when the govern­ment eval­u­ates its meth­ods, what does it find?


STEW­ART: I’m not really sure they eval­u­ate their meth­ods partic­u­larly well. [laughter] They can eval­u­ate the costs reas­on­ably well. That’s the approach that we’re find­ing is that risk assess­ment is not to tell you an answer, not to tell you what’s right and wrong. It’s really just to give you some insights into how your system is work­ing. If we look at aviation secur­ity, at the TSA which has at the moment 21 layers of secur­ity from canines to bomb detect­ors for the luggage, to check point screen­ing, to air marshals. What happens before you get to the airport, what happens at the airport, what happens on the plane, there’s 21 layers.


There’s been no system­atic eval­u­ation of how those layers actu­ally work. Do you need 21? Maybe 18 is enough. Maybe you need 30. Just look­ing in detail and asking ques­tions. What threat are they trying to deter? Are they there to deter a threat? Are they there for secur­ity theater? Are they there to actu­ally disrupt a plot? Just asking those funda­mental ques­tions helps set a train, a thought process, about how to model the system and how effect­ive it’s going to be. What we’re find­ing is that in many cases, you can achieve a higher reduc­tion of risk at a lower cost. If you’re just a bit smarter about what you do and invest more money in meas­ures that we can see it work­ing and maybe divert resources from other areas that aren’t work­ing. You can actu­ally save money and increase public safety.


Q: One of the programs you analyze is the NSA’s section 215 of the Patriot Act, the tele­phone metadata program where all our tele­phone metadata is swept up. The govern­ment’s original claims about the effect­ive­ness of this program melted away as more public aware­ness.


MUELLER: Another way of saying that is that maybe they were lies? Maybe you don’t want to say that.


Q: How diffi­cult is a program that is terribly expens­ive, you analyzed the cost of it.


MUELLER: It’s not terribly expens­ive, but the ques­tion is it does­n’t seem to do any good. We looked at it and we said well, it’s kind of a no brainer. There’s no bene­fit and does cost some­thing. So how do you cost/bene­fit analysis? So we said let’s pretend that it might do some good in the future. It still does­n’t come out look­ing very good.


STEW­ART: The main thing is that we want to model how good it behaves in a way we know is relat­ively robust. So when we do our cost/bene­fit analysis, we’re not trying to say that the cost/bene­fit ratio is exactly this number. What we tend to say is this meas­ure to be cost effect­ive, how much do you have to reduce the risk for that meas­ure?


So if you find that you need to reduce the risk by 99 percent, well we don’t think that’s really going to happen. Or you may say how high does the attack frequency have to be for a meas­ure to be cost effect­ive? So I can’t say that the prob­ab­il­ity of an attack is exactly five percent, or ten percent, or 15 percent. But I know it’s not 600 percent. If the number comes out to be that if a meas­ure is cost effect­ive only if expect­ing six, or ten, or 30 attacks a year, then that gives you like a threshold. Then it’s up to the secur­ity experts to rethink where they think that threat lies. But if noth­ing else, we simply don’t think the threat is that high.


MUELLER: One method is to do break even analysis, which Mark just alluded to. The way you do that is you say “how many attacks would there have had to have been to justify these expendit­ures?” We look at the increase in counter terror­ism expendit­ures within the United States since 9/11. The ques­tion is how many attacks, maybe like the Times Square attack or the Boston bomb­ing attack, how many attacks like that would have had to have been preven­ted, protec­ted against, or deterred in order to justify the expendit­ures? It comes out to be one every day or one every week or some­thing like that for the whole period since 9/11.


So to justify the expendit­ures, you have to be able to say, yeah, but they would have deterred that many attacks, preven­ted that many attacks. And that’s very hard to do. But is’s a nice clean way to set the issue up. Similar with the FBI, how many plots would they have had to foil that would have other­wise been success­ful? The results are similar. There’s an awful lot they would have had to have done to be success­ful. And it’s implaus­ible that they have done so.


Q: You would hope now, 15 years out from this horrible event that we’d start to apply this kind of reas­on­able approach to counter terror­ism. What would be the best advice you’d give as far as what you’ve looked at, what is actu­ally effect­ive in redu­cing the threat, or what we know in that area?


STEW­ART: In the imme­di­ate after­math of 9/11, it made a lot of sense to put in a lot of secur­ity, really quickly to reas­sure the public and restore confid­ence. That made a lot of sense.


MUELLER: And before you really knew how big the threat was, right? Err on the safe side.


STEW­ART: Yeah. So you want to put a lot of resources in there, like a surge. Basic­ally you want a surge. But after fifteen years you might think, well maybe we can put off the accel­er­ator a little bit. Given there’s a bit more time to now reflect on what the threat is, the nature of the threat and how it’s evolving. John and I have spent a lot of time look­ing at aviation secur­ity. Because that’s some­thing that all of us can see, many of us go through many secur­ity proced­ures. The TSA have these 21 layers. We look at some of those layers and some of them are very expens­ive, partic­u­larly the federal air marshal service that has about two to four thou­sand air marshals at the moment. It cost about 900 million dollars per year from the TSA.


MUELLER: Which is impress­ive I must say.


STEW­ART: And it costs the airlines a few hundred million dollars because they have to provide free seats to these air marshals at very short notice, only two hours’ notice. So actu­ally bump­ing people in first class or putting them in coach. So the total cost is more than a billion dollars a year. The air marshals, four thou­sand sounds like a lot, but there’s some­thing like 80 thou­sand flights a day in the United States. So you do the math and they’re on about five percent of flights, maybe not even that. A very, very low propor­tion of flights. Some flights have up to six air marshals if it’s a large aircraft. Others have two.


So when you factor in the very high cost, the very low propor­tion of flights, they’re only there to prevent a hijack­ing. They’re not really there to stop an IED, a bomb on a plane. So they’re there for one threat, which we think has passed because since they hardened cock­pit doors post 9/11, and now passen­gers and crew would fight back if there’s a hijack­ing where before you were pass­ive and would wait for nego­ti­ations. What we saw happen­ing for the fourth plane that crashed, passen­gers and crew are going to fight back like you would­n’t believe. You talk to pilots and they’re going to fight for their lives.


So having air marshals might have made sense early on to reas­sure the public.


But when we do our risk assess­ment, they reduce the risk of a hijack­ing by less than one percent. It’s a very small number. The cost is like a billion dollars, so bene­fit to cost ratio is pretty easy to figure out. Its bene­fit to cost ratio is less than 0.1. So every dollar you’re spend­ing, you’re only getting ten cents in bene­fits. And that’s being gener­ous to the program. If I was to think what the real number would be, it would be a lot lower than that.


So we’ve been advoc­at­ing wind­ing down the program. Still have some on the books because you may still need them as a deterrent. And you invest some of that money else­where in the aviation sector. Very few other coun­tries have air marshal programs. Australia has one. Don’t want to say too much with the camera on. But being mind­ful of the cost and being mind­ful of how they manage their marshal program. In Australia, it’s managed by the Australian Federal Police. So they’re serving police officer­s—a bit like the FBI. They’re the federal agency for invest­ig­a­tion. So it’s a rota­tion. They spend six months to a year as an air marshal and then they go back and do some­thing that’s going to be a lot more excit­ing than sitting on a plane all day when you can’t drink, you can’t watch the movies.


MUELLER: You can’t drink, of course.


STEW­ART: And you can’t sleep. So it’s not a very attract­ive career. So in Australia, we can mix and match numbers when the threat rises and lowers. Air marshals in Australia have decreased in number. In the U.S., because it’s a separ­ate service, it’s a Federal Air Marshal Service with train­ing facil­it­ies and all sorts of…


MUELLER: Gymnas­i­ums.


STEW­ART: There are careers at stake. So no one wants to really jeop­ard­ize their own career.


MUELLER: Can I just add on that? The air marshals’ thing is about the closest you get to a total no brainer. It’s an extremely expens­ive program. It does­n’t do much good. You can take that money, just a portion of that money and put it in other meas­ures that do better and just get rid of it or reduce it enorm­ously. We’ve actu­ally had one congressper­son, John Duncan of Tennessee, who actu­ally came out and said, actu­ally citing our work a little bit, that we ought to get rid of the air marshals. So when Mark and I were in Wash­ing­ton last fall, I think it was, we got over to the congress­man’s office and talked to his chief aide and asked him, it’s a great idea, anything we can do to help with that, it’s a really good proposal, you should do it. He said Duncan was unable to get any other congress­man to sign on to his bill. None. Because they were all afraid some­thing would happen and they’d be blamed. So they’re unwill­ing. This program’s ludicrously expens­ive, does­n’t do much good at all, and it’s a no brainer to point that out, and you can’t get any move­ment on it.


Q: I know we’ve talked about talked about this SPOT program. TSA’s program of beha­vi­oral detec­tion. Same thing.


MUELLER: And several other things like that.