Skip Navigation

When it Comes to Justice, Algorithms are Far From Infallible

Algorithms are playing an increasingly important role in police departments and courtrooms across the nation. But “objective” algorithms can produce biased results.

  • Erica Posey
March 27, 2017

Early on in Tues­day’s confirm­a­tion hear­ing, Neil Gorsuch sugges­ted that the judi­ciary may be in danger of auto­ma­tion. When asked how polit­ical ideo­logy can affect judi­cial decision-making, Judge Gorsuch joked that “they haven’t yet replaced judges with algorithms, though I think Ebay is trying, and maybe success­fully.” The joke fell flat, but Judge Gorsuch isn’t completely wrong – though Ebay does­n’t seem to have anything to do with it.

Algorithms already play a role in courtrooms across the nation. “Risk assess­ment” soft­ware is used to predict whether or not an offender is likely to commit crimes in the future. The soft­ware uses personal char­ac­ter­ist­ics like age, sex, socioeco­nom­ics, and family back­ground to gener­ate a risk score that can influ­ence decisions about bail, pre-trial release, senten­cing, and proba­tion. The inform­a­tion fed into the system is pulled from either defend­ant surveys or crim­inal records.

Algorithms also help determ­ine who ends up in the courtroom in the first place. Police are invest­ing in “predict­ive poli­cing” tech­no­logy — power­ful soft­ware that uses data on past crime to fore­cast where, when, and what crimes might occur. Police use the predic­tions to make deploy­ment decisions. Some soft­ware even claims to predict who may be involved in a future crime. A pilot program in Chicago used soft­ware to identify roughly 400 people likely at high risk of being involved in viol­ent crime in the next year. Law enforce­ment noti­fied the indi­vidu­als and followed up with them in an attempt to cut the city’s crime rate. Facial recog­ni­tion algorithms are already used with surveil­lance foot­age, and emer­ging tech­no­logy will allow real-time facial recog­ni­tion with police body cameras.

Proponents of the tools laud the soft­ware’s poten­tial to cut costs, drive down prison popu­la­tions, and reduce bias in the crim­inal justice system. The expens­ive and preju­di­cial outcomes of our human-driven crim­inal justice system are well docu­mented. As Judge Gorsuch lamen­ted, “I’m not here to tell you I’m perfect. I’m a human being, not an algorithm.”

Unfor­tu­nately, the algorithms aren’t perfect either. A ProP­ub­lica analysis of a widely-used risk assess­ment algorithm found that only 20% of people the soft­ware predicted would commit viol­ent crimes went on to do so in the two years after the assess­ment was conduc­ted. When all crimes – includ­ing misde­mean­ors – were taken into account, the algorithm was only slightly more accur­ate than a coin flip in predict­ing recidiv­ism rates. Worse still, it was nearly twice as likely to misla­bel black defend­ants as high risk than white defend­ants.

“Object­ive” algorithms, rely­ing on biased input data, can produce biased results. Predict­ive poli­cing systems primar­ily analyze past poli­cing data to develop crime fore­casts; the algorithms may be more skilled at predict­ing police activ­ity than crime. As crim­inal justice professor Chris­topher Herrmann noted, “at best, these predict­ive soft­ware programs are begin­ning their predic­tions with only half the picture,” given that only 50% of crimes are ever repor­ted to the police. Facial recog­ni­tion algorithms fall prey to similar issues. Stud­ies suggest– and commer­cial applic­a­tions reveal – that facial recog­ni­tion algorithms tend to be less accur­ate for women and people of color, likely because the original data used to train the soft­ware didn’t include suffi­cient examples of minor­ity faces.

Addi­tion­ally, the tools them­selves aren’t well under­stood. The algorithms are often gener­ated by private compan­ies that protect their soft­ware as intel­lec­tual prop­erty. There are relat­ively few inde­pend­ent assess­ments of their valid­ity, so judges and attor­neys lack the inform­a­tion they’d need to adequately under­stand or chal­lenge their use. A public defender in Cali­for­nia struggled to even access the surveys taken by her clients to calcu­late their risk scores. One Wiscon­sin judge over­ruled a plea deal because of the defend­ant’s high risk score, only to later reduce the sentence on appeal after hear­ing testi­mony from the risk score’s creator that clari­fied its mean­ing.

There are ways to mitig­ate these concerns, and proponents note that algorithms can still perform equally as well or better than human judg­ment. Why demand perfec­tion from computers when the human altern­at­ive is also imper­fect?

But Judge Gorsuch’s testi­mony high­lights perhaps the biggest issue with using algorithms in a crim­inal justice context. Computers and algorithms are popularly perceived to be infal­lible and unbiased. History warns of the dangers of using math and stat­ist­ics to lend credence to racism. One shoddy inter­pret­a­tion of 1890s census data drew a false caus­a­tion between black­ness and crimin­al­ity. The study, hailed as object­ive because of the data source and perceived neut­ral­ity of the immig­rant author, was used to justify Jim Crow laws and insur­ance discrim­in­a­tion. Imper­fect algorithms predict­ing crimin­al­ity can provide a veneer of impar­ti­al­ity to a system of insti­tu­tion­al­ized bias.

We are not far from a future in which a person may end up in prison after an algorithm sends cops to their neigh­bor­hood: an algorithm iden­ti­fies their face as having an outstand­ing warrant: and an algorithm tells a judge that the person is at high risk of commit­ting further crimes. We need to make sure our soci­ety – and espe­cially our judi­ciary – fully under­stands these tools, and takes their limit­a­tions into account.