Skip Navigation
Analysis

Lessons on How to Conduct Credible Post-Election Audits

Contrasting the reviews of Arizona’s 2020 election and the 2020 Census illuminates how reliable audits can work.

Close-up shot of audit boxes
Courtney Pedroza/Stringer

In Janu­ary, a small Flor­ida-based company was found in contempt of court and fined tens of thou­sands of dollars, after which it promptly dissolved. For Cyber Ninjas, this was only the latest chapter in a tumul­tu­ous year since it was contrac­ted to eval­u­ate the 2020 elec­tion results in Mari­copa County, Arizona. Its review made national news for vari­ous stumbles — but these missteps can prove instruct­ive for recog­niz­ing future sham reviews, espe­cially when contras­ted with other, more cred­ible efforts.

Ten days before Cyber Ninjas released its final report, the Amer­ican Stat­ist­ical Asso­ci­ation’s Census Qual­ity Indic­at­ors Task Force released its qual­ity eval­u­ation of the 2020 Census to compar­at­ively little fanfare. While both the census task force and Cyber Ninjas purpor­ted to provide inde­pend­ent over­sight of a tech­nical process crucial to a func­tion­ing demo­cracy, the former consisted of experts conduct­ing a fair, informed, and trans­par­ent audit, while the latter lacked each of these neces­sary compon­ents of cred­ib­il­ity. As the next elec­tion cycle looms and politi­cians around the coun­try continue to pursue partisan elec­tion reviews, the differ­ences between these two efforts can shed valu­able light on how cred­ible audits can and should work. Let’s look at a few of the biggest differ­ences. 

Expert­ise

The U.S. Govern­ment Account­ab­il­ity Office requires that audit­ors possess tech­nical expert­ise that allows them to accur­ately identify prob­lems — or a lack thereof. This subject matter expert­ise ensures any review by audit­ors is meth­od­o­lo­gic­ally sound and cred­ible.

Partisan Arizona Review

Cyber Ninjas had no back­ground in elec­tion reviews — with not a single mention of audit­ing on its website prior to the Arizona review — but was chosen over a Boston-based firm that has conduc­ted 200 elec­tion audits. Predict­ably, its report was riddled with errors. Many of Cyber Ninjas’ policy recom­mend­a­tions were already in place. For instance, it recom­men­ded that “voting machines… always have a paper backup of all ballots,” even though they already do in every county in Arizona. Imply­ing that policies like these do not exist misin­forms Arizonan voters and contrib­utes to declin­ing confid­ence in elec­tions.

Cyber Ninjas also lacked an under­stand­ing of prob­ab­il­ity and how those concepts play out in voter data. That precluded its abil­ity to accur­ately assess any poten­tial numer­ical discrep­an­cies in the elec­tion results.

Census Task Force

The census task force, on the other hand, was composed of cred­ible experts repres­ent­ing a broad range of relev­ant back­grounds, from academic research­ers to former Census Bureau direct­ors and scient­ists. Several are accom­plished stat­ist­i­cians and demo­graph­ers, and some have served as expert witnesses in federal litig­a­tion. These experts were assembled by the Amer­ican Stat­ist­ical Asso­ci­ation, a relev­ant, cred­ible, profes­sional organ­iz­a­tion with a centur­ies-long commit­ment to gath­er­ing, clas­si­fy­ing, and report­ing U.S. stat­ist­ics.

Objectiv­ity

The GAO also outlines a baseline stand­ard of objectiv­ity for audit­ors, a criterion that is excep­tion­ally import­ant in a hyper-polar­ized polit­ical envir­on­ment. If the review appears biased, the public will be less likely to trust the results.

Partisan Arizona Review

Even if Cyber Ninjas could conduct an impar­tial assess­ment, its public support for former pres­id­ent Donald Trump and the Big Lie under­mined their review’s cred­ib­il­ity, ensur­ing its percep­tion as a partisan effort. (This percep­tion was only deepened by its appar­ent lack of interest in audit­ing the elec­tion results until Arizona had been called for Pres­id­ent Biden.) CEO Doug Logan had made his pro-Trump and pro-conspir­acy views known on social mediaauthored a memo advising U.S. senat­ors to object to certi­fy­ing the 2020 elec­tion, and he worked closely with Sidney Powell and other Trump allies who sought to prevent Pres­id­ent Biden from taking office. Money poured into the company’s review from sources with obvi­ous ulterior motives, such as Over­stock CEO Patrick Byrne, who infam­ously berated Trump offi­cials that they weren’t doing enough to attempt to over­turn the elec­tion results. This clearly partisan activ­ity creates the impres­sion that the review was not conduc­ted in good faith.

Census Task Force

The task force was formed in response to concerns about the 2020 Census’s data qual­ity before the census was completed or the results were known, rather than after it turned out in a way that partisan actors didn’t approve. The task force contin­ued its work despite a change in polit­ical admin­is­tra­tions mid-stream and without regard to the polit­ical outcomes of the count. This helped bolster the sense that it was impar­tially pursu­ing its goal to “produce a set of scien­tific­ally-sound, publicly avail­able stat­ist­ical indic­at­ors” on the qual­ity and accur­acy of the 2020 census.

In a further demon­stra­tion of its objectiv­ity, the task force addressed conflicts of interest as they emerged. When Pres­id­ent Biden appoin­ted task force co-chair Robert Santos to be director of the Census Bureau, for instance, Santos imme­di­ately resigned from the task force, allow­ing it to main­tain its impar­ti­al­ity and conduct its review separ­ately from the Bureau.

Trans­par­ency and Secur­ity

Trans­par­ency is neces­sary for public trust because it holds external review­ers account­able for their actions and encour­ages object­ive results. A trans­par­ent review process should also include proto­cols that ensure data confid­en­ti­al­ity and secur­ity while clearly telling the public what can and cannot be shared.

Partisan Arizona Review

Despite prom­ising to conduct “the most trans­par­ent elec­tion audit in Amer­ican history,” Cyber Ninjas fought to keep its review proced­ures secret and restric­ted press access to the process. Ballots were left unat­ten­ded and spot­ted near pens with blue ink, expos­ing them to tamper­ing.

Census Task Force

The Census Task Force struck a finer balance between data confid­en­ti­al­ity and process trans­par­ency. The Census Bureau selec­ted three task force members with special sworn status to access confid­en­tial, internal 2020 census data, which ensured that their review meth­ods complied with strong confid­en­ti­al­ity laws. The task force was also trans­par­ent about its member­ship, assess­ment meth­od­o­logy, and outputs, as well as its eval­u­ation’s limit­a­tions. 

The task force’s report advoc­ated for increased trans­par­ency within the Census Bureau, recom­mend­ing that the bureau make its qual­ity indic­at­ors publicly avail­able. They also encour­aged other qual­i­fied external research­ers to conduct inde­pend­ent analyses. 

Address­ing Limit­a­tions 

As outlined in the GAO’s best prac­tices for identi­fy­ing sources of evid­ence, cred­ible external reviews must also refrain from cast­ing base­less asper­sions, use clear language about uncer­tain find­ings, and modify the scope or meth­od­o­logy of an audit in the face of insuf­fi­cient evid­ence.

Partisan Arizona Review

When Cyber Ninjas did not have enough evid­ence to make clear and cred­ible claims, it made unfoun­ded alleg­a­tions, such as “23,334 ballots impacted,” or impli­cit accus­a­tions, without provid­ing proof of error or bad intent. In its exec­ut­ive summary, the company recom­men­ded that its find­ings be referred to the Arizona attor­ney general without making any case for crim­inal charges in the report. When it did include caveats to their sensa­tion­al­ist claims, they were buried in foot­notes.

Census Task Force

When the census task force did not have suffi­cient data, it said so clearly. In its Septem­ber 2021 report, it concluded that the “indic­at­ors released to date by the bureau do not permit a thor­ough assess­ment of the 2020 census data qual­ity.” This trans­par­ency is not a weak­ness — it bolstered the report’s cred­ib­il­ity.

•  •  

Cred­ible elec­tion audits are a crit­ical secur­ity meas­ure. Outside experts have an import­ant role to play in govern­ment over­sight, which is why the Census Bureau has consist­ently welcomed qual­ity eval­u­ations from vari­ous expert organ­iz­a­tions. But guard­rails are needed to ensure nongov­ern­ment entit­ies follow exist­ing profes­sional stand­ards and avoid another Cyber Ninjas–­type trav­esty.

Now more than ever, as several state legis­latures take steps to codify these charades that waste taxpayer money and under­mine public confid­ence in elec­tions, we must be able to tell the differ­ence between profes­sional over­sight that supports good governance and partisan sham reviews.