Transcript of John Mueller and Mark G. Stewart

John Mueller is a political scientist at Ohio State University and at the Cato Institute. He is the author, co-author, or editor of 18 books and hundreds of scholarly and popular articles. Mark G. Stewart is Professor of Civil Engineering at The University of Newcastle, Australia. He has 30 years of experience in probabilistic risk and vulnerability assessment of security systems, has published more than 400 articles, papers, and reports. Together they have written two books, Terror, Security, and Money: Balancing the Risks, Benefits, and Costs of Homeland Security; and Chasing Ghosts: The Policing of Terrorism.

Mike German, a fellow at the Brennan Center for Justice at NYU Law School, interviewed Mueller and Stewart on March 31, 2016. The following is an edited transcript of that interview.

Q:  My name is Mike German. I’m a fellow with the Brennan Center for Justice’s Liberty and National Security Program. Today we’re very pleased to have Mark Stewart, who is a professor of civil engineering at the University of New Castle, Australia. Not often in our discussions do we have somebody from the engineering field. But that’s why it’s so important that you’re here. And John Mueller, who is an old friend of mine, is a political scientist from Ohio State University and also a senior fellow with the Cato Institute in D.C. Both have published numerous works and this is their second effort together, called “Chasing Ghosts: The Policing of Terrorism.”

It’s a fascinating read, written with a lot of good humor, but serious data analysis as well. It’s an attempt, long overdue, to bring reason and scientific rigor to an assessment of our law enforcement, homeland security and intelligence efforts to counter the terrorist. So maybe that’s where we should start, talking about what is the threat. You start the book with a quote from George Bush, where he reminisces about a meeting with the FBI a few weeks after 9/11 where they said there are 331 al-Qaeda operatives within the United States. What happened to those 331 operatives?

MUELLER: No one knows. They never showed up. What’s interesting is that Bush, writing this book in 2010 or 2011, never goes back and tries to say what happened to those people. By 2005, the FBI, in fact, was saying that there are no al-Qaeda cells in the United States at all. And that continues to be true. Actually in 2002, this 331 number got bigger when intelligence people were telling reporters who have good in’s to the intelligence community that there were between two and four thousand al-Qaeda operatives in the United States, ready to go.

It was a complete fantasy. Of course, they might still be here, and they’re just being very quiet. Maybe they’re waiting for the apocalypse or they’re too busy watching pornography and doing drugs or something. But certainly, obviously, if they’d done anything, we’d know it. And/or the FBI or somebody would root them out. So somehow they were seeing ghosts. They were seeing all these people who were not there. And that continues.                          

Q: And it’s not that there isn’t a threat. You actually document in the book 62 cases.

MUELLER: Right. There’s more since the book came out. ISIS and everything.

Q: Tell us about that. You actually have that on a separate web-book.

MUELLER: Oh, yeah, one of the things that I’ve done, so you can’t blame Mark for this, is put together a book of all the cases of terrorism in which Islamic terrorists, since 9/11, have seemingly targeted or actually targeted the United States whether they are based in the United States or abroad. So it doesn’t deal with cases of people going abroad to fight either with al-Qaeda, or Taliban, or ISIS, but only those focused on the inside. It’s up to well over 60 case now. Ohio State has very good honor students and they wrote these cases up, and I would edit them. Anyway, it’s published on the web as a web-book of up to about 830 pages.

Q: Let’s just briefly talk about the response. You talk about some of the data that the Washington Post brought out about ”Top Secret America.” And if you break it down with the terrorist plots that you were able to find, I think you say in the book that “the U.S. has created or re-organized two counter terrorism organizations for every person arrested for plotting a terrorist act in the U.S.”

MUELLER: The wonderful book called “Top Secret America” by Dana Priest and Bill Arkin from the Washington Post that came out about 2010. Just tons and tons of information about how many people are working counter terrorism there are. One calculation is that the apparatus has launched far more covert operations in the aftermath of 9/11 than it had during the whole Cold War. There’s just this fantastic amount of effort. So the issue in our book in many respects is, are they finding enough terrorists to justify these fantastic costs?

I’ll just give you one example. In fact, we have to update the book now, it turns out. A book called “The Threat Matrix” by Garrett Graff, a reporter, says that in any one day the United States is chasing down about 5,000 terrorism leads—every single day. If each lead, on average, takes two days to clear, they have chased over 10 million.

But Peter Bergen just came out with a book and he talked to another counterterrorism agent who said that there’s actually 10 thousand leads that they’re following up every single day. Every single day your tax money is paying for this. We certainly know how many they’ve found, because they brought to trial or found on lesser charges or whatever. And the total number of people that this apparatus has come up with, terrorists, is certainly less than a thousand. Probably well less. And that would include ones also trying to go overseas to fight. So that leaves a fantastic number of false positives.

Q:  When we talk about these tips, there’s also a policy choice that the FBI in particular made that you call the 9/11 Commission Syndrome. Why don’t you tell us about that?

MUELLER: After 9/11, my namesake, Robert Mueller, the Director of the FBI mandated that every terrorism lead has to be followed up. Before, what they would do is they would triage. Some of this stuff is real garbage. In fact, I can give you a case of one lead.

It’s a “threat” from the Philippines to attack the United States unless blackmail money was paid. So they checked it out, and if you ask what was that threat, well, it’s an email they got saying, “Dear America, I will attack you if you don’t pay the 999999999 dollars. Mwah ha ha ha.” That’s the threat. But they had to follow it up. They got the Philippine authorities to go and visit his parents. So far, he hasn’t been heard from again.

I also talked to an FBI guy, a top guy, at a conference less than a year ago. I wanted to make sure that was still the case. So I said do you still follow up every single lead, every single tip including things like this “mwah ha ha” one? And he said – I didn’t put this word in his mouth, it came out of him, twice in fact. He said, “Unfortunately, yes.” So even in the FBI there’s some realization that they’re expending terrific amount of man power, $3 billion a year, chasing a vast number of false positives. You can spend your whole career in the FBI and never find a single thing chasing false positives constantly.

Q: And we actually have, I mean part of the apparatus that we built, this counter terrorism apparatus, includes this warning system that collects these potential tips. The mass surveillance programs the National Security Agency engages in, but if you’ve ridden the subway up, you’ll see all over the place, “See Something and Say Something.” What does that do?

MUELLER: Well, it generates a huge number of tips. It’s “If you see something, say something ®” — you can’t use that phrase without getting the permission of the New York Police Department. However, if you say “See something, say something,” you don’t have to get their permission.

A reporter from the New York Times asked them, how many terrorist arrests has this led to? The answer a few years ago was zero. They’ve gone through tens of thousands of these tips since the program started. The federal government is spending hundreds of thousands of dollars every year simply to publicize the “If You See Something, Say Something” banner. And no one is saying that it doesn’t do any good, nothing’s come from it. And one of the things that we’re trying to do in the book is try to evaluate how successful these programs have been and then compare them to what their costs are to see if they’re cost effective overall. Mostly the programs don’t come out looking too good.

Q: When we talk about costs, obviously here at the Brennan Center, at it’s Liberty and National Security program, we focus a lot on the cost to our individual privacy, the cost to civil liberties, the cost of government overreach in prosecution, investigation. But you suggest in the book that that’s not enough. That that won’t convince people who have a fear of terrorism that it’s greater than the threat actually poses. Talk a little bit about the public fear of terrorism and whether it’s based on reality.

MUELLER: Well, I guess the main message from the book for this group is that you can’t only say that these measures have privacy concerns. You also have to look at how much good have they done. You can’t let Dick Cheney say that counterterrorism and torture, and the other counterterrorism things have saved thousands, now he’s saying hundreds of thousands, of lives. Then you say, yeah, but they’ve intruded on people’s privacy. It’s not a very effective argument.

So what you have to do, basically, is look at what is the threat? And how much are these security measures basically reducing it? That’s the main thing we’re trying to do using certain standard methods of cost/benefit analysis.

It is also related to litigation, judicial decisions. For example, in December 2013, there are two decisions about Snowden, the metadata program, and whether it’s constitutional or not, or legal or not. One decision said it was not legal and mentioned in the judgement that the metadata program hasn’t caught anybody. It was supposed to be just a legal judgement. But it uses the program’s lack of success as ammunition to say it isn’t legal.

The other judgment starts out saying we live in a deeply dangerous world and NSA tells me about how zillions of people have been caught and so forth, and then concludes that the program is legal. So the legal judgement is being inflected by whether the program is effective or not. This issue is one that has to be dealt with directly.

I looked at all the Frontline programs about the Snowden thing, and nowhere in any of those programs did they talk about whether the metadata program or any of the NSA programs actually are successful. You can’t just talk about the cost, you have to talk about the benefits. If there are no benefits and some costs, then obviously it’s a no-brainer. But you can’t deal without that. Also, the movie, the Snowden movie, basically never talks about whether these surveillance measures have been effective.

You can’t simply say it’s an incursion on privacy, that it’s unreasonable search and seizure. People then say “well, it saved a hundred thousand lives, that proves it wasn’t unreasonable, right? They caught a lot of people, so that sounds damn reasonable to me.” Unless you’re able to deal with that argument, which this book tries to do, you don’t have as much to stand on it seems.

Stewart: And there’s a lot of worst case thinking in this area. So if you can imagine it, then the voters believe, it can happen. That therefore leads to very risk adverse decision making, because they’re worried about the next attack can be catastrophic, thousands of lives could be lost. I’ve heard senior officials in Australia say that the next attack in Australia would be catastrophic to the country. Given we’ve only had two, maybe three in the last fifteen years, and in each one, one person was killed. So instead of looking back and saying …

MUELLER: And one was killed by the police, too.

STEWART: Yes, that’s four. But no one actually looks back and says, what is the data? I mean, there’ve been plots against the New York subway and other transit systems, and the authorities say, “well we stopped this plot and saved tons of lives.” Well, the only time there’s another attack on the subway before was in London, and that cost about 40 lives, not hundreds of lives. So they’re always exaggerating the threat all the time. If you think you’re in a state of war, then that justifies the loss of a lot of civil liberties.

Q: Let’s talk about how we do risk assessment everywhere and obviously…

MUELLER: That’s Mark’s thing.

Q: …your contribution to this effort. It’s hard to imagine. How the government would like to have this argument is: “we can’t afford to have even one terrorist slip through the cracks.” And that justifies the expenditure of billions if not trillions of dollars. How does somebody who works in risk analysis look at that?

STEWART: Well, that would make sense if you’re thinking that one terrorist could kill ten thousand people. There is data that most terrorist attacks in the west kill only one or two people. There are exceptions. But they’re not going to threaten the existence of a nation state. They are not going to be existential. So when you do risk assessment, and that’s why I’m involved in this because I’ve done a lot of risk assessment in natural hazards: how high should we build a levy bank for floods, or how strong should you design a building against cyclones. You never aim for zero risk, because that’s just impossible. There’s always that balance between how much you spend and what safety to get. You try to find that balance, recognizing that perfect safety just can’t be achieved.

So the first step, you would ask yourself, what risks do you think are acceptable, or you’d tolerate? For example, when you’re looking at nuclear safety, environmental legislation, things which are pretty emotive issues, the general consensus is that society tends to accept the risk of death of no more than one in a million per year, which is a bit lower than the risk of being struck by lightning, much, much lower than the risk of being killed in car wrecks, or something like that. So that one in a million has been a benchmark across a whole range of risks.

Q:  And there are a whole range of risks. Not just natural disasters or criminal acts.

STEWART: Yeah, and that’s not where we might wind up, but that’s a good starting point. If the risk of death is like one in a thousand per year, then everyone would agree that that’s far too high and you must do something. So therefore, the first step would be, we’re not as interested in terrorism, what is the risk? What is the risk of being killed by a terrorist?

MUELLER: That’s the basic question, and it’s never asked.

STEWART:  And it’s not a difficult question to answer. I could do it in about half a day. It wasn’t really difficult because there aren’t that many attacks so it’s fairly easy to do the calculations. You don’t need to write a large computer program and get a PhD on this, it’s just basic stuff. Maybe don’t put that in the film. [laughter]

Yeah, you want to keep a job.

MUELLER: He slaves for the cause.

STEWART: So if you look at the last forty years in the states, including what happened on 9/11, the annual risk of being killed by a terrorist in the U.S. in the last forty years is about one in four million.

MUELLER: Per year.

STEWART: Per year. If you look at the risk of death from terrorism post 9/11, in the U.S. it’s one in 90 million per year in the US. If we look at Australia, it’s about one in eight million per year, and that includes the attacks on Bali in 2002, because that was like on our doorstep, so. If you look at Australia post 9/11, the annual risk of death is about one in 15 million. They’re very small numbers. John and I present at lots of conferences and because of my background in risk, I present at risk conferences, particularly security risk conferences. There’s two or three days, everyone talking about risk. They’re waving their arms about, you know threats are evolving, cyber and everything else.. I am the only speaker to quantify, to put out a single number, just the numbers I just discussed. And I’ve had people amazed. They’re quite surprised.

MUELLER: They’ve never heard it before.

STEWART:  The whole idea of acceptable risk, they haven’t really thought about that notion before. To me, those numbers are a starting point for a discussion. I would properly interpret those numbers to say the risks are very low. So it shows the police and counter terrorism measures in the West, is working pretty well, otherwise the risk would be much higher, there’d be a lot more successful attacks. On the other side I’d argue it probably shows maybe the threat isn’t as high as what we think it is because there aren’t that many people really plan to commit acts of terrorism. And the true answer is probably a bit of both. And there could be other reasons as well. But that’s the point of having a dialog and a discussion about what’s working, what isn’t working. You can look at the data in more detail to see what years are worse than others, maybe linked to different changes in government policy. You can delve into it in much more detail and get a lot more insights from it. We just don’t see that happening.

Q: The government might come back and say all you have to do is turn on the news and you’ll see that terrorism is actually an increasing phenomenon. So if you go back 40 years, maybe it wasn’t as bad and now it’s much worse so therefore justifies these expenditures.

MUELLER: Yeah, that’s not particularly true. If you look at the United States, there was a lot of terrorism in the seventies and eighties, so the actual numbers… The same with Europe. There’s terrorism because of the ETA in Spain and IRA in England.

STEWART: It begs the question that should be asked. If government came back and said please explain this. That’s great, then we can discuss it and say different time periods went up and down and these are some trends, what we’re seeing, what we’re not seeing. You don’t want to rely too much on just a single metric, but it helps inform what potential solutions should be. It’s really just part of a discussion. If you’re going to spend hundred and twenty billions dollars a year in the U.S. on domestic counter terrorism, you’d think you’d have that sort of discussion.

Q: Right.

MUELLER: And in fact that’s a trillion dollars, more than a trillion dollars since 9/11. I’ll give you just one example, it’s actually two examples. They did the same thing twice. We were at a conference on aviation. So they sent these TSA types there. I simply asked, “the only way to make airplanes completely safe is to ground the airlines. I assume you don’t want to do that. So therefore, you have to take a certain amount of risk.”

Then I said basically, “what’s an acceptable risk?” Suppose if you get on an airplane, your chance of being killed by a terrorist—including 9/11 in the calculation by the way, which is of course a special outlier—is one in a million, is that acceptable? How about one in five million? How about one in 40 million? How about one in 80 million, which is basically what it is, including 9/11. If you get on an airplane, your chance of being killed by a terrorist is about one in 80 million per flight. Is that acceptable? They look at me like—they have no answer to it because they’ve never even considered the question.

That’s where they should really start. I mean the problem is you can say is that safe enough? How safe do you want it to be? Maybe it’s not safe enough. Maybe you want it to be one in 130 million. If you want it to be zero, however, there’s only one solution.

STEWART: And if you want to start, and that discussion ends with how does the system work? In engineering, they like to have models about how buildings behave. The first thing is, let’s try to represent how the system works, as best we can. To see which factors influence the outcomes. We’re not seeing any of that really happening at all. For example, in the aviation area, the focus is really on how to protect planes and airports particularly. They’re not really thinking about what are some of the opportunity costs, right? And as John alluded to, driving is a thousand times more risky than flying. Flying is extremely safe. Driving is one of the most dangerous activities we can do.

So directly after 9/11 when people were scared to fly and there was so much emphasis on security at airports, and big queues, and it could take you an hour to get through the checkpoint, people dropped off. “Bugger this, I’ll drive.” So after 9/11 there has been a clear spike in traffic deaths, between 200 and 400 per year. This isn’t our work, this is the work of others. That put down to more people decided for shorter destinations, it’s more convenient to drive.

So here you have a public policy that encourages people to drive rather than fly that’s costing lives, not saving lives.

Q: So that needs to be part of the cost element in the risk analysis?

MUELLER: The irony is that it’s there in the rest of the government. For example, you have a safety measure, you say “let’s require there be seatbelts in the back seats of cars.” Well, maybe that’s a good idea, maybe not. We have to talk about it. The first thing you’d want to do is find out how many people are in backseats and get killed in automobile crashes. We have tons of data on that. If it turns out like three people a year are killed in crashes in the back seat, you probably don’t want to spend a whole lot of money dealing with it. You can spend the money on something else. They did decide, finally, to require them in cars. Because there must have been enough deaths to justify the measure.

However, they decided that new cars had to have seat belts in the back, but not old cars. Now, one possibility is they made a moral calculation. They said people who drive in old cars in the back seat are expendable. [laughter] We don’t give a damn about them. Whereas, people in the back seat of new cars are much more valuable.

Q: That was the auto industry’s take.

MUELLER: Obviously, the reason for the difference was that the cost of retrofitting is very high. So they said it’s not worth protecting people in old cars because it costs too much to protect them. It’s much cheaper in new cars because if you’re building a car, you can get it all built into the design plan. And once it’s done, it’s probably a fairly marginal increased cost. So what they’re saying is people who drive around in old cars in back seats, too bad. They made that decision because they don’t have an infinite amount of money.

The thing that’s really ironic about this is that the Department of Homeland Security does good risk analysis of natural disasters. . Tornadoes sometimes come through Oklahoma and there are people living in the trailer parks who can’t get to shelters because trailer parks don’t have shelters. Maybe we should build shelters for them. So they’ve done analysis. How frequently would big tornadoes come through and how many people would is save if they enough warning and could get into these shelters? How much do the shelters cost? And they may or may not decide to build the shelters. That’s the kind of analysis that should be done. And DHS is state of the art for natural hazards, but basically not at all, as far as we can see, for terrorism.

STEWART:  A cost/benefit analysis is standard procedure for a whole range of private sector and public sector decision making. It’s actually required. I mean the Australian government, the U.S. government, the Office of Management Budget, the GAO, all mandate that any new government regulation must be accompanied by a cost/benefit analysis, every one. And most countries in the world do that.

So when we began thinking about this we were okay, surely it’s been done. Surely the DHS or the TSA have looked at this. And we can’t find a single example of the cost/benefit analysis on any regulations that they have mandated. The closest we’ve got was for the full body scanners, the AITs. There must have been 30 to 40 pages looking at the cost. They’d worked out how much does the power cost? How much does each employee cost? Enormous detail on the costing. Then benefits was one sentence that said benefits are too difficult to quantify at this stage. That was it. So it was all on the cost, which is the easy bit.

Q: So what are the impediments to a more rational approach within government? You talk about the threat matrix and the pressure that puts on policy makers involved in counter terrorism.

Mueller: We deal with that in the last chapter of the book concerning responsible counter terrorism policy. It seems to be the way to look at this is to look to the Constitution for a second here. A very popular document in this room I imagine. Practically the first words in the Constitution are “preserve domestic tranquility.” If you read Hobbes, obviously, that’s why people have governments. So the main reason for government, the foundational reason, certainly one of the foundational reasons for having government at all is to provide for public safety, domestic tranquility. People want to be safe. They’re willing to pay. They’re willing pay in civil liberties. They’re willing to pay in taxes and so forth. They want to be able to walk down the street and not get hit over the head. They want their children to be able to get to school safely.

The problem is that governments also have a finite budget. If you have an infinite budget, everything changes. But they only have so much money to spend. They can’t prevent every possible hazard from happening. They have to make unpleasant choices. To do that irresponsibly, which is basically we’ll spend a billion dollars to save this life, even though it only costs a hundred thousand dollars to save other lives, is basically irresponsible and reprehensible.

It is true that public opinion is extremely uptight about terrorism. We have a whole chapter in the book on that. And we’ve done some recent stuff as well on it. As Cass Sunstein, the major risk analyst and lawyer at Harvard Law School points out, it’s perfectly reasonable for governments to pay attention to people’s feelings, and concerns, and so forth, but that doesn’t mean that the decisions should be made on the basis of their mistakes. And if they have a mistaken idea about how bad a hazard is, you shouldn’t give into it. It should be, as Mark has put it, risk neutral on that.

And that’s not happening. They’re basically doing things like let’s put the AITs in. We know exactly what they’re going to cost. We don’t know if there’s any benefit at all. But we’ll just do it. Or a guy gets on with a bomb in his shoes so therefore we’ll say everybody has to take off their shoes. So it’s somewhat management by in-box. It’s basically very irresponsible. What they’ll say is, as Mike sort of suggested, well if I don’t do this, I’ll lose my job. First place, it’s not clear that’s true. There was a big attack in San Bernardino, but I don’t know that anybody that lost their job in the police or anything. Or in Belgium there were big screw ups in police department it looks like. But people aren’t losing their jobs. It’s not clear that if something happens you’re going to lose your job.

But the issue, it seems to me, is that if you’re not willing to make decisions that might be job threatening, you shouldn’t take the job in the first place. So I have very low patience for the idea that I’ll lose my job. That’s actually corrupt. It’s stealing money. I’m getting money to send my kids to college by making decisions which actually reduce safety, do not increase safety in the most efficient way. In other words, people are dying because of my decisions. That’s not different from taking money out of the till it seems to me. If a solider signs up and then someone starts shooting at him and he says I didn’t sign up to be a soldier to get shot at. They should say you should have thought of that before you took the bloody job. Or a firefighter who refuses to go into a burning building.

Similarly, it seems to me, if you’re not willing to make a decision, and you enter a job that might require making job-threatening decisions for the mission of the job, which is public safety, the reason you’re there, you shouldn’t take it at all. There’re a lot of safe jobs. You can be a plumber, a college professor, all kinds of things where you don’t have to worry about that stuff. [laughter] So it strikes me that it’s not a good rationalization at all. And basically an immoral one.

Q: I like the way you put in the book that there’s illusion and delusion within the intelligence community. The illusion part of that is the threat matrix that you have this. But there’s also delusion. You quote George Tenet, former CIA director as saying that even years later, he thinks their assessments about the numbers of al-Qaeda cells inside the United States were accurate, even though as you’ve suggested, they’ve never come out. Often throughout the book, you go through where FBI or NSA officials are testifying about the nature of the threat or the effectiveness of their counter measures that later prove to be untrue. Do you think that they are deluding themselves about the threat? How do you assess it?

MUELLER:  Some of our big problems is A, they’re not stupid and B, they’re not evil. It would be so easy if they were stupid or evil. Instead, they are dedicated public servants overall. I think Mark would agree with that. They’re responsible. They’ve listened to what we’ve said, and they ask intelligent questions. They say, boy this is really interesting, we have to think about this. And then, of course, they never invite us back. So there’s a flaw in the system there somewhat.

I think they basically do believe it. They’re scared. The threat matrix that Mike has mentioned is that after 9/11, the idea is we should have this big piece of paper which basically has every threat. Actually, the fact that they call it a threat as opposed to a lead or a tip is interesting. They call it a threat. So like this guy who said I’m going to blow up, the mwah ha ha guy. And it goes into this threat matrix. And it goes to the head of the CIA, head of the FBI, and ultimately the President every day.

Let me give you a quote from Jack Goldsmith here. He quotes George Tenet. And he talks about reading this stuff every day. Spending an hour or something looking at it, and there’s all these things that are really horrible. He said, “Virtually every day, Tenet says, you would hear something about a possible impending threat that would scare you to death.” And then Goldsmith from Harvard comes in somewhat later. He’s sort of not part of this, but he’s sympathetic. He says, “That captures the attitude of every person I knew who regularly read the threat matrix.”

Every day you’re reading these things and some of these things are really horrible. There’s going to be a nuclear weapon going off in Manhattan and so forth. He concludes, “The want of actionable intelligence”—that none of this intelligence was actionable, basically, because there was nothing there—“combined with the knowledge of what might happen, produced an aggressive panic attitude that assumed the worst about the threats.” This is what Mark was talking about, the exaggeration of threats.

So it has a sort of psychological impetus. Then the other thing that Mike mentioned is that when Tenet came out with his autobiography in 2007, he said my gut tells me, my operational instinct tell me, I have no evidence, but there must have been a bunch of other al-Qaeda types infiltrated at the time of 9/11, before 9/11. There’s tons of evidence that that’s simply not true, including testimony from Khalid Sheikh Mohammed, for example. He also repeated it on a 60 Minutes show. So what he’s saying is I’m scaring you to death based entirely on nothing, my operational intuition. He does say that and you might say, well his intuition might be especially good. But I doubt it. Of course, that’s 2007 and none of those people have shown up in the intervening near decade.

Q: And when the government officials describe the terrorist threat, they say it’s evolving, that’s ingenious, that it’s conniving. But when you analyzed the 62 cases that were present of plots in the United States, what did you find about the character of the terrorist mastermind?

MUELLER: We have a chapter on the myth of the mastermind. This is from the Department of Homeland Security National Infrastructure Protection Plan of 2009. This is a grand plan about how they’re going to protect people and so forth. It would be really important to assess how good the enemy is. This is their entire statement about this, only two sentences, rather long ones though. Maybe it’s only one sentence. It’s only one sentence, very long sentence.

 “The enemy is relentless, patient, opportunistic, and flexible, shows an understanding of the potential consequence of carefully planned attacks on economic, transportation, and symbolic targets, seriously threatens national security, and could inflict mass casualties, weaken the economy, and damage public moral and confidence.” Now, if the next sentence said, “on the other hand most of them are knuckleheads,” at least you’d have some sort of balance. But that second sentence is never there.

So when I’m assigning the students to do this the cases, I said one of the things that I want you to do is tell me what these guys were like in these 60 or 70 cases. What did they do? What did they smell like? Where’d they come from? Were they on drugs? Were they criminals? Were they college professors? Whatever. Just where’d they come from, what’d they smell like? What kind of adjectives would you use? In the case studies, these are the kind of adjectives that come up repeatedly: incompetent, ineffective, unintelligent, idiotic, ignorant, inadequate, unorganized, misguided, muddled, amateurish, dopey, unrealistic, moronic, irrational, foolish, and gullible.

Basically, a summary by Brian Jenkins a counter terrorism guy at Rand who has been in this field for 50 years. He says, and it’s a really good summary: “Their numbers remain small, their determination limp, and their competence poor.” But you’d never know that from the DHS statement.

Q: They would argue though, number one is that it doesn’t take a genius to do horrible damage.

MUELLER: Lee Harvey Oswald was one case.

Q: So one of the concepts that they have is this idea of radicalization. That once somebody articulates a particular ideology, that they are on a path to violence that must be interdicted with all sorts of different means. And the FBI in particular using informants to move those people toward the terrorist attack. What does the data show about that?

MUELLER: They’re pretty good at infiltrating informants. The considerable majority of the cases had FBI operatives as part of the plot. They weren’t just watching it, but they actually joined it and play acted. And in some people’s minds, entrapped—but not necessarily in a legal sense, I guess.

I also asked the students when they’re doing the case studies, tell me what motivated these guys. Where did it come from? They use the word radicalization frequently, but overwhelmingly, the radicalization, at least in these cases, is because of hostility to American foreign policy. One of the guys in the Boston Marathon case wrote on the inside of the boat. It says, “You’re out to destroy Islam. You’re attacking, you’re killing my people. Your soldiers are killing and raping Iraqi women.” Etcetera.

In other words, what they think is that what’s been happening in the Middle East, that’s their main motivation, outrage. It’s about a drone strike. That guy Zazi from New York, he’s an Afghan American, and he read in the newspaper about  either a drone strike or maybe just an air strike, that killed a whole bunch of Afghans.  That ticked him off. I’m not justifying their reaction necessarily. I’ve been pretty intensely hostile to the American foreign policy as well, but I’m not going to throw bomb about it. They say repeatedly, that’s what motivated us. And it rarely makes it into the press. Just like the degree to which terrorism presents a threat, the expected number of how many people are going to be killed per year and so forth doesn’t enter into the press.

Q: Or in government radicalization studies, the government funded radicalization studies, they often don’t mention those elements of the process. Let’s talk a little bit about the press and how that molds public opinion around these issues.

MUELLER: The press is basically acting rationally. If you talk about terrorism, people look at the story. If you can get ISIS into any kind of story, that’s really great because people are going to read it. I think they’re acting rationally in the sense that they’re trying to sell their product. They want it to have people not turn to another channel. They want to have people click on the story. Click bait. So they basically feed into it. I think it’s irresponsible in the sense that they don’t say, “well we just had this happen, but the chance of this happening is one in 80 million” or something like that. I think they play to it and they exacerbate it perhaps, but it’s really does seem to be bottom up. The people are just plain scared. That’s what happened when I did this analysis of public opinion.

I can summarize that. I was somewhat surprised at the results as I thought there would be an erosion of fear over the years since 9/11. So I looked at 20 or 30 public opinion trend lines since 9/11. There are two patterns. One pattern,  is shown in the question “do you think there’ll be another attack killing large numbers of people.” Not just an attack, but killing large numbers of people in the near future. That’s been asked consistently since 9/11. In September 2011, the percentage of people saying that it was likely or somewhat likely was about 70 percent. Currently, it’s about 70 percent.

It’s gone up and down a bit. It’s a bit higher now because of ISIS. But even before ISIS, it was still about 70 percent. No change at all. Despite the fact that we got Bin Laden. Despite the fact there’s been no more 9/11s. Despite the fact that up until Paris, there was a ten year period of no major attacks in Europe. Despite the fact that the chance of being killed is one in 4 million or one in 80 million. It’s still there.

The other pattern is there’s a spike up at 9/11, and then it came down, but stayed the same. There’s one question, it’s really quite a good question, “Do you worry that you or a member of your family could be harmed by a terrorist?” So it’s a nice personal question, it’s not complicated. Do you worry about that? When Timothy McVeigh did his terrorism, that’s when they asked the question first, 1995. He blew up the Federal Building in Oklahoma City. It stood at about 40 percent. Forty percent of people said they worried somewhat or very. Then it went into real decline in the next few years, and then they stopped asking the question. There was obviously a shock by the Oklahoma City thing, but it was wearing off over the next two or three years.

Then at 9/11, it went from about 20 percent up to 60 percent. By the end of the year, it had gone back down to about 40 percent. But it hasn’t really declined since that time, even before the rise of ISIS. It goes up and down: if there’s a bombing in London it spiked up and so forth. But there’s been basically no erosion. If you ask people do you feel safer than before 9/11, there’s been no change, basically. If anything, people feel a little bit less safe. If you ask who’s winning the war on terrorism? About the same over all that time.

I think it’s fair to argue that the media are not hyping nearly as much as they did in the first years. Nor are public officials. You’ve got Obama, who actually said, a year ago or so, that ISIS does not present an existential threat to the United States. The president of the United States actually saying something like that. Banal and obvious, it might be, but other officials have been saying that for a long time that it is an existential threat. Anyway, the general amount of alarmism coming out of the government is, I think, definitely declining from the early years when they were hysterical all the time. There was going to be a big attack before the 2004 election, there’s going to be a 9/11 in the next few minutes. There’s going to be nuclear devastation one place or another within the next ten years, and so forth. You don’t see that kind of official alarmism as much, I think it’s fair to say, as before. Nonetheless, public opinion hasn’t changed at all.

STEWART: And the media are not that critical. They love to bandy about the word “mastermind” and “the genius bomb maker” and all these other things. Yet, if you were to treat terrorism a bit like criminals, most criminals we know are not really smart. They get caught. That’s why so many movies about bank heists that go wrong because it’s really comedy.

The other thing is that when there’s a plot that’s been foiled, the media report what the terrorists planned to do, which is you know, toppling a building, taking out a subway, not what the capacity actually was.

Q:  And not just topple the building, but have the building create a tsunami.

STEWART: Right, it’s going to slide down a hill. But the media are not very questioning about things. So you see on TV when they interview a counter terrorism expert—like after Brussels, this expert was saying London and Berlin should be worried and this and that. And the interviewer was just lapping it up. Yet every time I give an interview about the work that we’re doing, they’re saying all those risks, they can’t be that low. How did you get that? What’s this? What time period? They’re very particular. They’re trying to find a hole in our argument. If I was to say we should be worried about terrorism and we should be protecting Sydney and Melbourne and Canberra, then they’ll just lap it up. It wouldn’t be particularly critical.

Q: Doing a proper risk assessment is half the story, right? The other half is then looking at your enforcement methods and evaluating them. So when the government evaluates its methods, what does it find?

STEWART: I’m not really sure they evaluate their methods particularly well. [laughter] They can evaluate the costs reasonably well. That’s the approach that we’re finding is that risk assessment is not to tell you an answer, not to tell you what’s right and wrong. It’s really just to give you some insights into how your system is working. If we look at aviation security, at the TSA which has at the moment 21 layers of security from canines to bomb detectors for the luggage, to check point screening, to air marshals. What happens before you get to the airport, what happens at the airport, what happens on the plane, there’s 21 layers.

There’s been no systematic evaluation of how those layers actually work. Do you need 21? Maybe 18 is enough. Maybe you need 30. Just looking in detail and asking questions. What threat are they trying to deter? Are they there to deter a threat? Are they there for security theater? Are they there to actually disrupt a plot? Just asking those fundamental questions helps set a train, a thought process, about how to model the system and how effective it’s going to be. What we’re finding is that in many cases, you can achieve a higher reduction of risk at a lower cost. If you’re just a bit smarter about what you do and invest more money in measures that we can see it working and maybe divert resources from other areas that aren’t working. You can actually save money and increase public safety.

Q: One of the programs you analyze is the NSA’s section 215 of the Patriot Act, the telephone metadata program where all our telephone metadata is swept up. The government’s original claims about the effectiveness of this program melted away as more public awareness.

MUELLER: Another way of saying that is that maybe they were lies? Maybe you don’t want to say that.

Q: How difficult is a program that is terribly expensive, you analyzed the cost of it.

MUELLER: It’s not terribly expensive, but the question is it doesn’t seem to do any good. We looked at it and we said well, it’s kind of a no brainer. There’s no benefit and does cost something. So how do you cost/benefit analysis? So we said let’s pretend that it might do some good in the future. It still doesn’t come out looking very good.

STEWART: The main thing is that we want to model how good it behaves in a way we know is relatively robust. So when we do our cost/benefit analysis, we’re not trying to say that the cost/benefit ratio is exactly this number. What we tend to say is this measure to be cost effective, how much do you have to reduce the risk for that measure?

So if you find that you need to reduce the risk by 99 percent, well we don’t think that’s really going to happen. Or you may say how high does the attack frequency have to be for a measure to be cost effective? So I can’t say that the probability of an attack is exactly five percent, or ten percent, or 15 percent. But I know it’s not 600 percent. If the number comes out to be that if a measure is cost effective only if expecting six, or ten, or 30 attacks a year, then that gives you like a threshold. Then it’s up to the security experts to rethink where they think that threat lies. But if nothing else, we simply don’t think the threat is that high.

MUELLER: One method is to do break even analysis, which Mark just alluded to. The way you do that is you say “how many attacks would there have had to have been to justify these expenditures?” We look at the increase in counter terrorism expenditures within the United States since 9/11. The question is how many attacks, maybe like the Times Square attack or the Boston bombing attack, how many attacks like that would have had to have been prevented, protected against, or deterred in order to justify the expenditures? It comes out to be one every day or one every week or something like that for the whole period since 9/11.

So to justify the expenditures, you have to be able to say, yeah, but they would have deterred that many attacks, prevented that many attacks. And that’s very hard to do. But is’s a nice clean way to set the issue up. Similar with the FBI, how many plots would they have had to foil that would have otherwise been successful? The results are similar. There’s an awful lot they would have had to have done to be successful. And it’s implausible that they have done so.

Q: You would hope now, 15 years out from this horrible event that we’d start to apply this kind of reasonable approach to counter terrorism. What would be the best advice you’d give as far as what you’ve looked at, what is actually effective in reducing the threat, or what we know in that area?

STEWART: In the immediate aftermath of 9/11, it made a lot of sense to put in a lot of security, really quickly to reassure the public and restore confidence. That made a lot of sense.

MUELLER: And before you really knew how big the threat was, right? Err on the safe side.

STEWART: Yeah. So you want to put a lot of resources in there, like a surge. Basically you want a surge. But after fifteen years you might think, well maybe we can put off the accelerator a little bit. Given there’s a bit more time to now reflect on what the threat is, the nature of the threat and how it’s evolving. John and I have spent a lot of time looking at aviation security. Because that’s something that all of us can see, many of us go through many security procedures. The TSA have these 21 layers. We look at some of those layers and some of them are very expensive, particularly the federal air marshal service that has about two to four thousand air marshals at the moment. It cost about 900 million dollars per year from the TSA.

MUELLER: Which is impressive I must say.

STEWART: And it costs the airlines a few hundred million dollars because they have to provide free seats to these air marshals at very short notice, only two hours’ notice. So actually bumping people in first class or putting them in coach. So the total cost is more than a billion dollars a year. The air marshals, four thousand sounds like a lot, but there’s something like 80 thousand flights a day in the United States. So you do the math and they’re on about five percent of flights, maybe not even that. A very, very low proportion of flights. Some flights have up to six air marshals if it’s a large aircraft. Others have two.

So when you factor in the very high cost, the very low proportion of flights, they’re only there to prevent a hijacking. They’re not really there to stop an IED, a bomb on a plane. So they’re there for one threat, which we think has passed because since they hardened cockpit doors post 9/11, and now passengers and crew would fight back if there’s a hijacking where before you were passive and would wait for negotiations. What we saw happening for the fourth plane that crashed, passengers and crew are going to fight back like you wouldn’t believe. You talk to pilots and they’re going to fight for their lives.

So having air marshals might have made sense early on to reassure the public.

But when we do our risk assessment, they reduce the risk of a hijacking by less than one percent. It’s a very small number. The cost is like a billion dollars, so benefit to cost ratio is pretty easy to figure out. Its benefit to cost ratio is less than 0.1. So every dollar you’re spending, you’re only getting ten cents in benefits. And that’s being generous to the program. If I was to think what the real number would be, it would be a lot lower than that.

So we’ve been advocating winding down the program. Still have some on the books because you may still need them as a deterrent. And you invest some of that money elsewhere in the aviation sector. Very few other countries have air marshal programs. Australia has one. Don’t want to say too much with the camera on. But being mindful of the cost and being mindful of how they manage their marshal program. In Australia, it’s managed by the Australian Federal Police. So they’re serving police officers—a bit like the FBI. They’re the federal agency for investigation. So it’s a rotation. They spend six months to a year as an air marshal and then they go back and do something that’s going to be a lot more exciting than sitting on a plane all day when you can’t drink, you can’t watch the movies.

MUELLER: You can’t drink, of course.

STEWART: And you can’t sleep. So it’s not a very attractive career. So in Australia, we can mix and match numbers when the threat rises and lowers. Air marshals in Australia have decreased in number. In the U.S., because it’s a separate service, it’s a Federal Air Marshal Service with training facilities and all sorts of…

MUELLER: Gymnasiums.

STEWART: There are careers at stake. So no one wants to really jeopardize their own career.

MUELLER: Can I just add on that? The air marshals’ thing is about the closest you get to a total no brainer. It’s an extremely expensive program. It doesn’t do much good. You can take that money, just a portion of that money and put it in other measures that do better and just get rid of it or reduce it enormously. We’ve actually had one congressperson, John Duncan of Tennessee, who actually came out and said, actually citing our work a little bit, that we ought to get rid of the air marshals. So when Mark and I were in Washington last fall, I think it was, we got over to the congressman’s office and talked to his chief aide and asked him, it’s a great idea, anything we can do to help with that, it’s a really good proposal, you should do it. He said Duncan was unable to get any other congressman to sign on to his bill. None. Because they were all afraid something would happen and they’d be blamed. So they’re unwilling. This program’s ludicrously expensive, doesn’t do much good at all, and it’s a no brainer to point that out, and you can’t get any movement on it.

Q: I know we’ve talked about talked about this SPOT program. TSA’s program of behavioral detection. Same thing.

MUELLER: And several other things like that.