Skip Navigation
Fellows

Tightening the National Security Ratchet

In national security policy, U.S. intelligence agencies often implement new measures following an emergency without evaluating whether the measures work or improve our national security policy.

May 14, 2015

Cross-posted on Just Secur­ity.

A ratchet is a device that employs mech­an­ical imped­i­ments to allow move­ment in only one direc­tion. As such, it is a useful meta­phor for national secur­ity policy, where restrict­ive meas­ures taken in response to emer­gen­cies quickly become normal­ized, prevent­ing a return to the pre-emer­gency state. As new threats inev­it­ably emerge, new restric­tions are employed, creat­ing ever-tight­en­ing secur­ity prac­tices, regard­less of their neces­sity or effic­acy. One imped­i­ment prevent­ing a release of the national secur­ity ratchet is the fail­ure to prop­erly eval­u­ate the effect­ive­ness of new powers gran­ted. Another is excess­ive secrecy that agen­cies manip­u­late to avoid account­ab­il­ity. Two recent eval­u­ations of the FBI coun­terter­ror­ism activ­it­ies demon­strate these phenom­ena.

The first is a report by the 9/11 Review Commis­sion published last March which exam­ines the FBI’s progress in imple­ment­ing the 2004 recom­mend­a­tions of the 9/11 Commis­sion (a differ­ent group) and trans­ition­ing into a “threat-based, intel­li­gence driven organ­iz­a­tion.” [Full disclos­ure: the 9/11 Review Commis­sion inter­viewed me during its eval­u­ation.] Despite laud­ing the FBI’s efforts to reform itself after the 9/11 terror­ist attacks, the Review Commis­sion’s report “high­lights a signi­fic­ant gap between the artic­u­lated prin­ciples of the Bureau’s intel­li­gence programs and their effect­ive­ness in prac­tice.” One might think that such a disap­point­ing find­ing after 11 years of concer­ted effort toward reform would lead the Review Commis­sion to call for a seri­ous re-eval­u­ation of the FBI’s post-9/11 coun­terter­ror­ism tactics and author­it­ies. But that’s not how the national secur­ity ratchet works. The national secur­ity ratchet always demands the agen­cies be given more author­ity, with the hope they’ll use it better.

In fact, the govern­ment rarely employs stand­ard social science research meth­od­o­lo­gies to eval­u­ate the effect­ive­ness of its secur­ity meth­ods. Dr. Cynthia Lum of George Mason University’s Center for Evid­ence-Based Crime Policy discusses a system­atic review of terror­ism stud­ies she conduc­ted for the Camp­bell Collab­or­a­tion, and her book, Evid­ence-Based Coun­terter­ror­ism Policy


Only seven of the 20,000 terror­ism stud­ies Lum iden­ti­fied eval­u­ated the effect­ive­ness of coun­terter­ror­ism meas­ures using moder­ately rigor­ous meth­ods, leav­ing the govern­ment and the public with little evid­ence about what works and what does­n’t. The reas­ons agen­cies fail to test their meth­ods are complex, but Lum suggests it may rise from the fear of discov­er­ing that inves­ted efforts and resources may have been wasted.

The 9/11 Review Commis­sion invest­ig­a­tion simil­arly failed to util­ize rigor­ous social science research meth­ods, examin­ing only a stat­ist­ic­ally insig­ni­fic­ant five FBI coun­terter­ror­ism invest­ig­a­tions: David Head­ley’s parti­cip­a­tion in the 2008 attacks in Mumbai, India; Nidal Hasan’s attack at Ft. Hood, Texas; Fasail Shahz­a­d’s failed attempt to bomb Times Square; Najibul­lah Zazi’s inter­dicted plan to attack the New York City subway; and Tamer­lan Tsarnaev’s Boston Mara­thon attack. Remark­ably, though the FBI success­fully preven­ted only one of these attacks, the Review Commis­sion declared the exist­ing legal author­it­ies provided in the Patriot Act and the Foreign Intel­li­gence Surveil­lance Act, among others, were “essen­tial to the invest­ig­a­tions in each case.” The Review Commis­sion then argued the FBI “should ensure Congress is aware of the crit­ical value of these programs as it considers retain­ing, refin­ing, and expand­ing the Bureau’s author­it­ies as the threat evolves” (my emphasis). In other words, despite its find­ing that the FBI failed to satisfy the 9/11 Commis­sion’s recom­mend­a­tions, it concluded Congress should continue giving the FBI more author­ity, in the hope that it will become better.

Other evid­ence presen­ted in the Review Commis­sion’s report tends to argue against the effect­ive­ness of these expan­ded surveil­lance author­it­ies. The Review Commis­sion quoted the Webster Commis­sion’s find­ing that the FBI’s invest­ig­a­tion of Nidal Hasan was hampered by “a ‘crush­ing’ volume of data.” Simil­arly, the Review Commis­sion’s discus­sion of the FBI’s fail­ure to heed repeated tips that David Head­ley was a terror­ist raised “the import­ant ques­tion faced by all intel­li­gence agen­cies … [of] how to scan and assess volu­min­ous amounts of collec­ted inform­a­tion stra­tegic­ally … .” And despite the FBI’s expan­ded invest­ig­at­ive author­it­ies and elec­tronic drag­nets, Head­ley, Zazi, and Shahzad were able to travel over­seas to terror­ist train­ing camps and back to the US without detec­tion. Yet ques­tions about whether the FBI’s expan­ded intel­li­gence collec­tion was over­whelm­ing agents with useless data and false leads went largely unex­amined.

Like­wise unex­amined were the negat­ive consequences of the FBI’s expan­ded author­it­ies to inter­fere with the privacy and civil liber­ties of millions of ordin­ary Amer­ic­ans not suspec­ted of any wrong­do­ing. Continu­ing to do more of the wrong things won’t make the FBI better. The Review Commis­sion did not exam­ine whether cheaper, more focused, and less intrus­ive meth­ods might have been equally or even more effect­ive.

Lum discusses why requir­ing evid­ence-based coun­terter­ror­ism research is essen­tial to effect­ive policy:


Lum argues that evid­ence-based research provides an object­ive means to enable fiscal respons­ib­il­ity and demo­cratic account­ab­il­ity, essen­tial elements of effect­ive governance.

The second recent report examin­ing FBI coun­terter­ror­ism meth­ods reveals how govern­ment agen­cies use secrecy to main­tain or expand new author­it­ies despite evid­ence of inef­fect­ive­ness. The New York Times recently obtained a less-redac­ted copy of a 2009 combined inspect­ors general report regard­ing what it called the “Pres­id­ent’s Surveil­lance Program” (PSP), essen­tially the extra-legal elec­tronic surveil­lance programs later author­ized by the FISA Court under new inter­pret­a­tions of certain Patriot Act provi­sions, and by Congress through the FISA Amend­ments Act.

The report made public in 2009 included a section assess­ing the effect­ive­ness of the PSP. The Justice Depart­ment Inspector General mentioned an internal FBI survey eval­u­at­ing the program, quot­ing FBI lead­er­ship as saying that based on the results of the survey they determ­ined that the program was “of value.” The report said other agents were “gener­ally support­ive” of the programs “as one tool among many” that could “help move cases forward.” The IG, however, expressed a healthy skep­ti­cism of these views, docu­ment­ing how the excess­ive secrecy surround­ing the programs led to some FBI complaints that the PSP-derived tips did not provide enough inform­a­tion to appro­pri­ately prior­it­ize the leads. But a reader of the 2009 report would likely conclude that FBI offi­cials had reason to believe the PSP was valu­able to their coun­terter­ror­ism mission.

It turns out the FBI did some­thing extremely unusual and actu­ally scien­tific­ally eval­u­ated how help­ful the leads were to coun­terter­ror­ism invest­ig­a­tions. The newly declas­si­fied version of the IG report describes a 2006 FBI assess­ment that involved 30 FBI analysts and an FBI stat­ist­i­cian who tracked a stat­ist­ic­ally signi­fic­ant number of PSP leads dissem­in­ated through 2005 to eval­u­ate their worth. They found only 1.2 percent made a “signi­fic­ant contri­bu­tion” to FBI coun­terter­ror­ism efforts. Moreover, several of the signi­fic­ant tips related to subjects of ongo­ing FBI invest­ig­a­tions, rais­ing the ques­tion of whether a more focused approach would have proven a more effi­cient use of resources. A second study focus­ing on leads coming from the PSP email metadata drag­net found no signi­fic­ant contri­bu­tion to any FBI terror­ism invest­ig­a­tion. The results of these surveys were redac­ted from the 2009 public report, not to protect national secur­ity, but appar­ently to shield these programs from account­ab­il­ity. FBI lead­er­ship’s determ­in­a­tion that the PSP leads were “of value” seems partic­u­larly specious and self-serving given this hard data, but this claim was actu­ally modest compared to what the Justice Depart­ment later told the FISA Court about the value of just one aspect of the PSP, the domestic tele­phone metadata collec­tion.

In 2006, the FISA Court author­ized the domestic tele­phone metadata portion of the PSP under a broad new inter­pret­a­tion of Section 215 of the Patriot Act, but imposed specific minim­iz­a­tion proced­ures limit­ing the govern­ment’s use of the data. In 2009, however, the govern­ment advised the FISA Court that it had routinely viol­ated these proced­ures. Judge Reggie Walton deman­ded more inform­a­tion and stricter compli­ance with the FISA Court mandated proced­ures, but ulti­mately allowed the program to continue based on “the govern­ment’s explan­a­tion, under oath, of how the collec­tion of and access to such data are neces­sary to analyt­ical meth­ods that are vital to the national secur­ity of the United States.” Later analysis would determ­ine that the tele­phone metadata program never iden­ti­fied a single terror­ist or stopped a terror­ist plot.

So even in the rare instances where agen­cies prop­erly eval­u­ate their programs, secrecy allows them to mislead their over­seers and avoid the public account­ab­il­ity that might release the national secur­ity ratchet.

Dr. Lum and her colleagues at the Center for Evid­ence-Based Crime Policy are devel­op­ing tools to object­ively eval­u­ate law enforce­ment policies and prac­tices. If Congress is seri­ous about making the FBI become an intel­li­gence-led organ­iz­a­tion, it should compel the FBI to embrace evid­ence-based research meth­ods for eval­u­at­ing its coun­terter­ror­ism and law enforce­ment policies. Secur­ity resources should go to programs that work, not to protect­ing programs that don’t.