Skip Navigation
Analysis

Court: Public Deserves to Know How NYPD Uses Predictive Policing Software

The New York State Supreme Court ordered the NYPD to produce substantially more records about their predictive policing program.

January 26, 2018

The Bren­nan Center won a signi­fic­ant victory in the fight for govern­ment trans­par­ency and surveil­lance over­sight last month, when the New York State Supreme Court (the state’s trial court) ordered the NYPD to produce records about test­ing, devel­op­ment, and use of predict­ive analyt­ics tools in response to the Bren­nan Center’s ongo­ing Article 78 litig­a­tion.

The litig­a­tion stemmed from a Free­dom of Inform­a­tion Law (FOIL) request submit­ted by the Bren­nan Center in June 2016 for records about predict­ive poli­cing. Predict­ive poli­cing uses computer model­ing to try to anti­cip­ate crime and to manage when and where police officers are deployed. Algorithms analyze histor­ical data to predict where certain crimes may occur and some­times even who may be involved in a future crime. 

Predict­ive poli­cing soft­ware has been enthu­si­ast­ic­ally adop­ted by law enforce­ment agen­cies in recent years, cham­pioned by proponents as a tool to cut costs, improve police effect­ive­ness, and even poten­tially improve fair­ness in poli­cing. Yet most of these tools rely on histor­ical poli­cing data to gener­ate their predic­tions; in the absence of mean­ing­ful over­sight and trans­par­ency, the soft­ware may instead simply recre­ate and obscure the origins of racially biased poli­cing. Unfor­tu­nately, the nature of the tools often frus­trates efforts to hold police account­able. Most of the soft­ware involves the use of a black-box algorithm, often purchased from a third-party commer­cial soft­ware provider, to gener­ate predic­tions from histor­ical data.  The use of private compan­ies can some­times intro­duce addi­tional complic­a­tions to trans­par­ency, intro­du­cing ques­tions of trade secret and intel­lec­tual prop­erty inform­a­tion.

The Bren­nan Center filed the original FOIL request in the interest of better under­stand­ing and inform­ing the public about the use of these systems. The request sought docu­ments includ­ing:

  • commu­nic­a­tions with private developers of predict­ive poli­cing soft­ware;
  • histor­ical inputs and outputs of the soft­ware;
  • records on test­ing and util­iz­a­tion of the soft­ware;
  • audit logs; and
  • policies and proced­ures govern­ing the use of predict­ive poli­cing.

If obtained, these records would help eval­u­ate the costs and bene­fits of predict­ive poli­cing systems and assess whether the system was subject to suffi­cient over­sight and account­ab­il­ity. Yet the NYPD issued blanket deni­als of both the initial request and a subsequent appeal, forcing the Bren­nan Center to file suit in Decem­ber 2016. After months of good faith nego­ti­ations, in which the Bren­nan Center narrowed its request to exclude the algorithm itself as well as the most recent six month’s worth of inputs and outputs, the NYPD produced some docu­ments with signi­fic­ant redac­tions, but contin­ued to with­hold most of the records reques­ted by the Bren­nan Center. On August 30, 2017, the court heard oral argu­ments.

In briefs and at oral argu­ment, the NYPD fought to keep these outstand­ing docu­ments secret. Their attor­ney claimed that releas­ing vendor corres­pond­ence would reveal trade secret inform­a­tion and jeop­ard­ize future commu­nic­a­tions with poten­tial vendors, arguing that the NYPD had agreed to keep product test­ing and perform­ance inform­a­tion strictly confid­en­tial. (Actu­ally, the non-disclos­ure agree­ments signed between the third-party vendors in this instance and the NYPD only covered NYPD inform­a­tion. There was no coun­ter­vail­ing agree­ment for NYPD to main­tain the confid­en­ti­al­ity of the vendors.) Addi­tion­ally, they claimed that disclos­ure of histor­ical output data and notes about the types and sources of data used could enable indi­vidu­als to recre­ate the predic­tions of the NYPD and some­how game the system.

The Court was uncon­vinced. Find­ing that the NYPD had failed to offer any expert testi­mony to support its claims about trade secrets and secur­ity, the Court ordered the NYPD to produce email corres­pond­ence with predict­ive poli­cing soft­ware vendors with only limited identi­fy­ing inform­a­tion redac­ted; histor­ical output data from the exist­ing predict­ive poli­cing system up through June 27, 2017; and notes from the NYPD’s Assist­ant Commis­sioner of Data Analyt­ics who developed the predict­ive poli­cing algorithms currently in use. The NYPD was also ordered to produce the summary of results of trials that the depart­ment conduc­ted of several vendors’ products; the judge will review those records to determ­ine whether any part of them is covered by a delib­er­at­ive process priv­ilege, and then presum­ably order disclos­ure of the remainder. These mater­i­als should offer addi­tional insight into both the exist­ing system and the specific­a­tions that the NYPD discussed with vendors previ­ously under consid­er­a­tion. 

In addi­tion, the Court ordered the NYPD to expand its search for respons­ive docu­ments to include the Coun­terter­ror­ism Bureau, and to submit an affi­davit about the results no later than the middle of March. The NYPD searched only three depart­ments for respons­ive docu­ments and issued a conclus­ory state­ment that the search was complete, claim­ing that those depart­ments were the only ones that could reas­on­ably be expec­ted to have relev­ant inform­a­tion. The Coun­terter­ror­ism Bureau was never searched. Yet the NYPD has said publicly that the predict­ive poli­cing system is housed within the depart­ment’s Domain Aware­ness System, which is itself located in the Coun­terter­ror­ism Bureau. 

Finally, in response to the Bren­nan Center’s request for policies govern­ing the predict­ive poli­cing system, the NYPD produced a single privacy policy docu­ment from 2009. This 8-year-old policy, the Public Secur­ity Privacy Guidelines, governs the use of the Domain Aware­ness System, which encom­passes “tech­no­logy deployed in public spaces as part of the coun­terter­ror­ism program of the NYPD’s Coun­terter­ror­ism Bureau.” The policy by its terms has little applic­a­tion to predict­ive poli­cing, which is not a “tech­no­logy deployed in public spaces” and purports to be a tool for stra­tegic decision making in poli­cing rather than a mech­an­ism for coun­terter­ror­ism, indic­at­ing that the NYPD has no policy in place that expli­citly governs the use of predict­ive poli­cing or the shar­ing and reten­tion of the data produced.

The Guidelines also require the NYPD to perform audits of the Domain Aware­ness System; if the policy applies to the predict­ive poli­cing system, that means the Depart­ment would be required to audit the system as well. The NYPD offered no audit docu­ments, however, after confirm­ing that it had done a dili­gent search for respons­ive records, which suggests either that it is not comply­ing with its own policy or that it is continu­ing to with­hold relev­ant mater­i­als. At the same time, the NYPD produced a paper co-writ­ten by Evan Levine, the NYPD’s Assist­ant Commis­sioner of Data Analyt­ics, describ­ing the predict­ive poli­cing system and Domain Aware­ness System, which cites stat­ist­ics about the effect­ive­ness of their predict­ive poli­cing algorithm. The NYPD must have run tests on their programs to come up with those stat­ist­ics, but no records of these tests were ever released to the Bren­nan Center.

In the course of the August 2017 hear­ing, the NYPD’s attor­ney sugges­ted that the police depart­ment regu­larly ignores FOIL requests until the requester gives up or files suit. While the Bren­nan Center is pleased that the Court ulti­mately gran­ted the major­ity of the original request, the general lack of trans­par­ency around law enforce­ment prac­tices, partic­u­larly with respect to the use of predict­ive poli­cing soft­ware, is deeply concern­ing. Citizens have the right to know about the tools, costs, and stand­ard prac­tices of law enforce­ment agen­cies that police their communit­ies. In this case, it took a year and a half and a lawsuit to ulti­mately get even a portion of the inform­a­tion that should have been made avail­able to the public at the outset. As Judge Barbara Jaffe wrote in her opin­ion on the Bren­nan Center’s FOIL litig­a­tion, “govern­mental trans­par­ency is a tran­scend­ent virtue.” The Bren­nan Center is commit­ted to continu­ing in its fight for govern­ment trans­par­ency, and looks forward to provid­ing analysis of the docu­ments produced as a result of this Order.