The Dangers of Unregulated AI in Policing

Across the country, police departments have adopted automated software platforms driven by artificial intelligence (AI) to compile and analyze data. These data fusion tools are poised to change the face of American policing; they promise to help departments forecast crimes, flag suspicious patterns of activity, identify threats, and resolve cases faster. However, many nascent data fusion systems have yet to prove their worth. Without robust safeguards, they risk generating inaccurate results, perpetuating bias, and undermining individual rights.

Police departments have ready access to crime-related data like arrest records and crime trends, commercially available information purchased from data brokers, and data collected through surveillance technologies such as social media monitoring software and video surveillance networks. Police officers analyze this and other data with the aim of responding to crime in real time, expeditiously solving cases, and even predicting where crimes are likely to occur. Data fusion software vendors make lofty claims that their technologies use AI to supercharge this process. One company describes its tool as “AI providing steroids or creating superhuman capabilities” for crime analysts.

The growing use of these tools raises serious concerns. Data fusion software allows users to extract volumes of information about people not suspected of criminal activity. It also relies on data from systems that are susceptible to bias and inaccuracy, including social media monitoring tools that cannot parse the complexities of online lingo, gunshot detection systems that wrongly flag innocuous sounds, and facial recognition software whose determinations are often flawed or inconsistent — particularly when applied to people of color.

Police use of technology can have beneficial uses as well. Body worn cameras help shed light on police interactions with civilians. AI-enhanced license plate readers (LPRs) decrease errors, helping police better match license plates associated with stolen vehicles and other crimes. Yet police departments are adopting ever more powerful data fusion tech with little testing, proof of efficacy, or understanding of the dangers posed to civil rights and civil liberties.

Public information about newer data fusion tools is scant, but two police tools — and the concerns frequently raised about them — offer insight into this technology’s capabilities and implications: predictive policing programs, which purport to predict where and when a crime is likely to occur and even who is likely commit it; and social media analysis software, which allows police to mine people’s online presence and identify connections, locations, and potential threats. Both technologies pose substantial risks, from perpetuating bias to chilling and even enabling the targeting of constitutionally protected speech and activity; independent studies assessing how much they contribute to public safety are scarce.

AI-enabled data fusion capabilities amplify these concerns. Without oversight and transparency measures in place, the use of these tools risks infringing on civil rights and civil liberties by magnifying bias and inaccuracies and encouraging bulk data collection. 

Historically, practical considerations limited law enforcement’s ability to collect and keep dossiers on individual citizens. Surveillance devices were more rudimentary and largely did not facilitate the surreptitious collection of detailed data that could reveal intimate personal information. And most data that police departments collected was connected to criminal activity. (The multiple high-profile examples of police of targeting racial justice activists for ostensibly radical activities is a notable exception.)

The “smart policing” approach emerged in the late 20th and early 21st centuries. The CompStat system, first pioneered by the New York City Police Department (NYPD) in the 1990s, uses crime-related data to map out where police officers should focus their resources. Almost all police departments across the country have adopted it, and Brennan Center research suggests it is associated with decreases in both property crimes and violent crimes — including homicides.

Predictive policing tools were popularized during the late 2000s and early 2010s. These first-generation data fusion tools combine and analyze historical data to predict when and where criminal activity might occur and who might be involved. Proponents of predictive policing programs argue that the technology can predict future crimes more accurately and objectively than police officers. But community groups have organized against use of these tools, objecting to their lack of transparency and potential for perpetuating bias. These concerns, combined with independent audits documenting inaccuracies and misuse of the programs, have led some departments to stop using them.

These programs were followed by data fusion tools, which assemble live data streams to alert police to crimes and help them respond in real time. The NYPD’s Domain Awareness System (DAS), for example, is a citywide database that stitches together license plate reader data, crime data, facial recognition technology, and gunshot detection sensors, and video feeds that incorporate video analytics software. In addition to mapping out crime hotspots, DAS notifies police of gunshot detection alerts, current and predicted locations of vehicles on watchlists, and suspicious activity identified through video analytics. Programs similar to DAS have emerged around the country, in the form of “smart cities” — municipalities using data fusion tools and other new technologies with the stated goal of making cities more sustainable and efficient while also opening the door to pervasive surveillance.

Over time, police data fusion systems expanded to include functions that are only tangentially related to criminal activity. Social media monitoring vendors, for example, claim that they can determine users’ sentiments, assign them risk scores, and map out their online relationships, including whether people have attended the same protests as individuals who pose a supposed threat to public safety or are linked to them in some other way. 

Technology vendors tout AI-driven data fusion tools as a one-stop shop for data analytics. Whereas traditional police databases provide departments with information that officers could assess to develop a theory or reach a conclusion, today’s data fusion tools automatically generate conclusions for police, supplying those determinations without context or explanation. Several red flags arise around this new model.

These systems are fed large amounts of data, much of it unvetted or inherently subjective (such as content coming from social media). Moreover, given these systems’ opacity, officers have no way to know how they reach their determinations, which compounds the risk of unidentifiable errors and actionable but potentially problematic — or altogether inaccurate — conclusions. In other words, AI-powered data fusion tools may lead police to act on false or misleading outputs, misdirecting resources and potentially infringing on the rights of innocent individuals. Police departments that lack strategies and procedures to evaluate and mitigate these risks may fail to adopt safeguards prior to deployment.

These AI-enhanced data fusion tools are rapidly becoming accessible even to less-resourced police departments. Systems like Cognyte’s NEXYTE “decision intelligence platform,” C3.ai’s C3 AI Law Enforcement “integrated intelligence platform,” Peregrine’s machine-driven “data integration platform,” and Flock Safety’s Nova “public safety data platform” are often less expensive and easier to implement than the NYPD’s DAS or the networks that smart cities require.

Most of these new tools are vendor-agnostic, meaning that departments can partner with multiple surveillance technology vendors while still accessing data in a single place. Some facilitate data sharing among departments.footnote1_0ND1PvIPqr99XzltmfsQdPnafpm3KrZL0Qld2aqjX0_l9VLNDjG3XU91 All can be categorized as artificial intelligence as defined in the National Defense Authorization Act (NDAA) for fiscal year 2019 — which notes that AI systems “perform[] tasks under varying and unpredictable circumstances without significant human oversight.”

Public information about these systems is limited. The Brennan Center reviewed public records, news stories, and vendor publications to identify approximately 40 police departments that may have used or received information produced by these tools; detailed information was available only from Peregrine and C3.ai. While early iterations of data fusion tech analyzed a single data stream (such as social media posts) or only a few data streams, the systems promise to fuse unlimited types of data. Peregrine claims that its platform can “integrate data of any type, from any source, at any scale.”

Police data fusion tech companies generally claim to be able to ingest and analyze video feeds, license plate reader data, social media, intelligence or suspicious activity reports, gunshot detection alerts, public records (such as marriage licenses and property records), criminal databases, and other unstructured data (like text, images, and video from police body cameras, dashcams, or stationary cameras often used in citywide camera networks).

Vendors contend that expanding the breadth of data enhances a data fusion system’s capacity to identify the strongest connections or threats. However, every data input carries a risk of inaccuracy or bias that can contribute to warped outputs; the huge amounts of data that these systems rely on compound that risk. And overconfidence in the system on the part of police officers may cause them to overlook or even ignore facts that contradict an algorithm’s determination.

Image

Traditionally, police departments manually consolidate information from various siloed databases and may not have easy access to external data sources. C3.ai offers centralized data sharing hubs that can amalgamate data from multiple police department customers. For example, one contract between C3.ai and San Mateo County, California, enabled the county’s 12 police departments to access each other’s data. A Forbes magazine profile on Peregrine described the company as hoping to “turbocharge local police department access to surveillance data,” giving less-resourced police departments the opportunity to utilize interagency data sharing through real-time crime centers using Peregrine tech.

While these data sharing capabilities may be a boon to police departments, the easy access to massive troves of data and ability to repurpose sensitive information far beyond the purpose for which it was initially collected can also facilitate intrusive searches and misuse of data for non–law enforcement purposes. 

Image

Today’s data fusion tools also differ from earlier technologies in that they claim a broad array of uses, including crime trend prediction, threat identification, sentiment analysis, risk score calculations, relationship mapping, and pattern and anomaly detection. These analytic capabilities are not altogether new, but the capacity to access them in tandem while drawing on such large volumes of data — far surpassing what individual officers or departments could collect or analyze on their own — magnifies the risks that outputs will be inaccurate, bake in bias, and enable indiscriminate surveillance. Indeed, guidance released by both the Biden and Trump administrations has categorized similar AI tools as having such significant implications for civil rights, civil liberties, and privacy that the federal government is prohibited from using them without critical safeguards in place, including training, pre-deployment testing, impact assessments, and ongoing monitoring.

Whether these new data fusion tools contribute measurably to public safety remains to be seen. The risks posed by the technologies from which they evolved, however, are well known — risks that these new systems seem likely not just to replicate but to supercharge. 

Predictive policing technologies and social media monitoring tools share certain characteristics with newer data fusion systems: They integrate various data points, attempt to predict crime, map out online networks, and conduct risk assessments. But compared to today’s rapidly evolving AI-enhanced data fusion technologies, they rely on far fewer data points, have more rudimentary technical capabilities, and are more limited in scope. Still, their known risks help to illuminate the hazards that new, unregulated data fusion technologies are likely to pose: perpetuating bias, reaching flawed or inaccurate conclusions, and facilitating indiscriminate surveillance and invasions of privacy.

Bias and Inaccuracy

When police feed inconsistent, unreliable, inaccurate, or biased data into predictive policing systems, even unintentionally, the resulting predictions are prone to reflect those issues. For example, historical crime data inputs often reflect systemic inequities. Brennan Center research has found a strong correlation between the racial and socioeconomic makeup of a community and misdemeanor arrest rates. In New York City, such arrests are concentrated in Black and Latino neighborhoods that have high rates of poverty, unemployment, and mental health issues. Additionally, police in major cities already spend considerably more time patrolling Black areas than non-Black areas with otherwise similar socioeconomic demographics and violent crime rates,  contributing to higher rates of arrest.

Predictive policing tools analyzing this arrest data may conclude that officers should disproportionately focus their efforts on Black and Latino neighborhoods, producing more arrests that can further drive inaccuracies in the algorithm. Newer data fusion tools are susceptible to the same defects: Like predictive policing tools, they lack built-in guardrails to prevent police departments from incorporating data that may reflect systemic inequities. And the vast amount of data that these new systems use to generate predictions can make problematic inputs even harder to identify.

Imperfect data collection and entry can also warp outputs. An audit by the Los Angeles Police Department (LAPD) inspector general found that “significant inconsistencies” in how police calculated and entered data into a now-inactive predictive policing program fueled biased predictions. Half the individuals flagged had few or no ties to arrests for the prioritized crimes. The department also lacked formal procedures detailing how the program was run.

Accurate data — and accurate data entry — directly affect the outputs of data fusion tools. As such, guidance and training are imperative. Because vendors and police departments have published little information about the tools, there is no way to know how many police departments using such systems have established appropriate guardrails. For example, in response to an open records request, the sheriff’s office in Orange County, California (which polices one of the nation’s most populous areas), admitted to having no policies, manuals, or procedures in place for its use of Peregrine’s data fusion platform. All these measures are essential to harm prevention.

Decisions about how to weight data inputs can contribute to inaccuracies as well. Chicago’s inspector general found that a predictive policing program meant to identify people likely to be involved in a shooting over-relied on arrest records — potentially elevating risk scores for people arrested for misdemeanors who had no connection to gun violence or had never been convicted. Because police accosted people with higher scores in the street and at their homes, the program “effectively punished individuals for criminal acts for which they had not been convicted.” 

Today’s data fusion tools can easily perpetuate and even amplify these issues, and uncovering such flaws may prove even more difficult. Cognyte’s NEXYTE, for instance, conducts risk scoring using AI and machine learning. The technical opacity that comes with these more advanced systems makes determining how they reach conclusions far more complicated than with a more rudimentary system — and almost certainly beyond the technical capabilities of most users, including police officers.

Limitations within the technologies’ capabilities can contribute to inaccuracies and bias too. Some data fusion vendors advertise that their systems are powered by natural language processing (NLP), a subfield of AI that allows computers to recognize, understand, and generate speech. Troublingly, NLP models have at times categorized words like Black, woman, and deaf as “toxic” and associated the word Muslim with violence.

They also have a poor record of conducting sentiment analysis (i.e., gauging someone’s opinion or views) and threat assessments. The technology struggles to interpret irony, satire, and humor and has proved ill-equipped to undertake context-specific tasks like identifying hate speech, terrorist propaganda, and harassment — labels about which even knowledgeable people disagree.

A simple test conducted by the ACLU demonstrates that ChatGPT, a predictive language tool, cannot reliably identify slang or online jargon. When asked to rank the suspiciousness of various statements — including “I wish the president was dead,” “I need to research ways to murder people,” and “Yesterday I bought fertilizer. Today I’m renting a truck.” — ChatGPT ranked the following as the most suspicious: “Free this week, for quick gossip/prep before I go and destroy America.” In fact, the British tourist who tweeted that out before a U.S. trip was using slang to say he planned to party here. His cheeky statement nevertheless had serious consequences when DHS agents, misinterpreting it much like ChatGPT, detained him for 12 hours and then put him on a plane back home.

Network analysis poses risks as well. Some social media monitoring vendors claim that their tools map out connections between users. These links can be misleading: They can reflect anything from close relationships to passing acquaintanceships and loose ties, and they can sweep up even people who simply have mutual friends or have encountered each other by chance online — nuances that algorithms and the police can easily miss.

A case study by one social media monitoring company, Voyager Labs, reveals how tenuous such connections can be. The company touted that its AI-driven platform VoyagerAnalytics had analyzed almost 4,000 Facebook friends of a user who urged his followers to infect Egyptian government officials with Covid-19 in March 2020. Voyager’s tool concluded that none of the friends could be tagged as an “extremist threat,” but it nevertheless suggested that the user’s indirect connections (i.e., friends of friends or followers) could have an “affinity” for “violent, radical ideologies” because they were connected to “known extremist accounts” — a determination the company did not explain. Despite the fact that no charges were filed and there was no evidence that the user actually posed a threat, the vendor produced materials insinuating that the user himself might be a violent extremist. The materials may have also prompted law enforcement to scrutinize other people in his network based solely on social media interactions rather than involvement in criminal activity. 

Image

Image

Many other types of data incorporated into data fusion systems may likewise be inaccurate or biased. For example, independent audits of ShotSpotter, one of the most widely adopted gunshot detection systems, found that 80 to 90 percent of alerts investigated by police could not be linked to a confirmed gun-related offense. And facial recognition technology is notoriously bad at identifying the faces of people of color: Black individuals account for at least eight out of ten people wrongfully arrested based on faulty matches.

Combining all these data streams into police-actionable outputs runs the risk of magnifying these effects while also increasing the difficulty of assessing which data has specifically contributed to inaccuracies. 

Indiscriminate Surveillance and Invasions of Privacy

AI-enabled data fusion tools, either by design or if misused, are likely to pose other hazards: intrusions into individuals’ privacy, violations of due process, and the facilitation of expansive surveillance that can border on harassment. Experiences with predictive policing are illustrative. For example, when the LAPD’s now-inactive predictive policing program Operation LASER identified certain individuals as “chronic offenders,” officers created fliers with their pictures and personal information and shared them department-wide. As a result, although some of the identified individuals had no active arrest warrant or any prior arrest for a violent crime, the system nevertheless flagged them for increased scrutiny, resulting in warning letters and even home visits from police. In a similar vein, the sheriff’s office in Pasco County, Florida, used a predictive policing tool to compile a list of people considered more likely to commit crimes, and then repeatedly sent deputies to their homes to cite them for minor offenses like missing mailbox numbers and overgrown grass. The office ultimately shuttered the program, finally admitting recently that it had violated residents’ constitutional rights to privacy, freedom of association, and due process.

AI-driven data analysis tools will vastly expand law enforcement’s ability to target people and extrapolate information about their movements, habits, and associations, along with personal information like health conditions and ideologies. For instance, Flock Safety, a high-profile provider of police surveillance tech, recently launched Nova, a data fusion tool that can reportedly correlate a person’s vehicle information with a slew of other data, including public records, information collected from data brokers (such as government IDs, contact information, and marriage licenses), and open-source information like social media profiles. Nova can also map out a vehicle owner’s relationships with other people, which could be combined with the company’s AI feature notifying officers about ostensibly suspicious driving patterns.

Flock Safety’s widely adopted license plate reader (LPR) systems (which can be used in conjunction with Nova) have raised such serious concerns that Congress opened an investigation into the company in August 2025. Recent reports reveal that a sheriff’s office in Texas lied about data captured from more than 88,000 Flock LPR cameras around the nation. The sheriff’s office claimed that the searches were in response to a family concerned about the safety of a missing family member suspected of self-managing an abortion. In fact, the woman was at the center of a “death investigation” for aborting her fetus.

Police departments across the country have also been conducting Flock Safety LPR database searches for immigration-related investigations on behalf of the federal government. Such searches violate state and local laws that prohibit collaboration with the federal government for immigration enforcement. Data fusion tools like Nova supersize the amount of data that law enforcement can rapidly access about a person from a single point, making individuals flagged by these systems — regardless of the circumstances — even easier to track down.

Data fusion system vendors claim that their tools can digest sensitive intelligence data and third-party intelligence reports (the latter likely originating from private cybersecurity and intelligence companies, according to Brennan Center research), as well as open-source intelligence and suspicious activity reports (SARs). This information is often gathered and produced through government intelligence programs that have a history of bias and of identifying, tracking, and targeting people based on constitutionally protected activity, such as attending a protest or practicing their religion. (Though vendors often advertise their products as criminal tools, some have ties to the U.S. military or federal intelligence agencies, with significant overlap between the products they offer to those customers and police departments.)

For instance, local law enforcement agencies compile SARs to document behavior indicative of potential terrorist activity. They share these reports with federal law enforcement agencies and fusion centers — state- and locally managed entities that integrate federal, state, and local law enforcement and other personnel to collect, analyze, and distribute a broad range of threat information. Although roughly a third of the reports sent to the federal government between 2010 and 2017 alleged a “nexus to terrorism,” this standard is overbroad.

Moreover, SARs may contain nonactionable or irrelevant information. One fusion center director told the Brennan Center that he would categorize any SAR that has “anything” to do with “foreign nationals” as a potential indicator of terrorism. And a 2013 government investigation found that more than 95 percent of SARs were never investigated by the FBI, suggesting that they offer little value. There is no evidence that the utility of SARs has improved since then; to the contrary, recent reviews show them to be rife with ethnic profiling and information not connected to criminal activity.

Police departments and fusion centers often compile open-source intelligence for “situational awareness,” too, such as information about protests collected from publicly available social media. A fusion center within the Boston Police Department has published intelligence reports describing antiwar and social justice groups as domestic extremists and used surveillance software to target Black Lives Matter activists and Muslims. The latest data fusion systems use this type of open-source intelligence alongside other forms of intelligence data in their analytic assessments. Their outputs therefore could direct police to conduct even more frequent or intensive monitoring of individuals and groups already disproportionately targeted, including for constitutionally protected activity.

Furthermore, people only tenuously connected to profiled individuals may be targeted. An open records request filed by the Brennan Center found that the Washington, DC, fusion center used a social media monitoring tool to identify accounts that were “geo-located in both Ferguson [Missouri] and Baltimore” during protests following the deaths of Michael Brown and Freddie Gray. The tool also identified an additional group of 58 social media users as having “common connections” with the accounts on the first list, even though they did not necessarily attend a protest. At least one social media monitoring company — Voyager Labs, which offers relationship mapping capabilities similar to those advertised by data fusion vendors (and which conducted the extremist case study described above) — is known to share data about such loose connections with law enforcement.

Data fusion systems can combine open-source intelligence like that gathered by the DC fusion center with other data points to help police departments create detailed profiles and map out relationships between protest movements, organizations, or activists and people distantly connected to them. These tactics have tangible repercussions: Surveillance makes people less likely to attend rallies, speak at meetings, or even associate with groups subject to enhanced scrutiny.

China’s surveillance and data analysis infrastructure offers a cautionary tale. Chinese police have capitalized on independent sources of surveillance data by using data fusion systems. The “Police Cloud” system compiles comprehensive, individualized records linked to people’s national ID numbers and uses analytics to scrutinize their associations, visualize trends, and keep track of individuals whom the government views as threats to government power. The system is part of a national project to allow police to monitor and track anyone and everyone — and effectively suppress crime and political dissent — under the guise of improving “social stability.”

Courts are starting to recognize the risks of unrestricted law enforcement data collection. Though the U.S. Supreme Court has yet to conclude that surveillance of this type directly violates the First or Fourth Amendments, a majority in its most recent case on this point, Carpenter v. U.S., concluded that the government’s acquisition of “deeply revealing” location data from a cell phone company constituted a search under the Fourth Amendment and therefore required a warrant. The Court observed that location data “provides an intimate window into a person’s life, revealing not only his particular movements, but through them his familial, political, professional, religious, and sexual associations.”

After Carpenter, one appeals court grappled with whether law enforcement use of hidden cameras on utility poles constitutes a search under the Fourth Amendment. There was no clear holding, but half the full court reasoned that the cameras require a warrant because they enable ongoing surveillance. The Colorado Supreme Court has recognized the same. And the Massachusetts Supreme Court has applied Carpenter to LPRs, ultimately holding that although limited use of LPRs does not implicate the Fourth Amendment, widespread use might. This reasoning should extend to data fusion systems that combine data points collected without a warrant, which reveal an even more intimate picture of someone’s life than data collected by a single surveillance tool.

As it stands, the patchwork of rulings on surveillance technologies and their narrow application in each case means that police departments are largely free to continue broadly collecting such information and using data fusion tools to streamline analysis of that data. Because these tools run on massive data sets, the companies marketing them encourage police to gather more data — like people’s whereabouts via vehicle surveillance systems, connections and patterns gleaned through public social media posts, and pictures and video of individuals’ faces — to feed and sometimes train their products.

Police departments (and data fusion companies themselves) may also buy biometric information, internet search histories, geolocation data, and other sensitive details from commercial data brokers, thus evading the warrant, court order, or subpoena that obtaining such information directly would require. Flock Safety briefly advertised that its Nova tool would incorporate data leaked from the dark web; the company has evidently abandoned those plans for now, but similar initiatives, if pursued, would expose personal data that was illegally obtained and likely unverified to law enforcement scrutiny. 

Lack of Transparency and Guardrails

Finally, history suggests that law enforcement will frequently use risky data fusion tools without crucial transparency measures and guardrails. Police departments have chronically failed to provide a public accounting of their predictive policing tools and other surveillance technologies. The NYPD, for example, revealed information about its predictive policing and social media monitoring programs only in response to lawsuits brought by the Brennan Center.

When police do share information, the data often reveals that these programs are far less effective than promised. For instance, an external audit of a predictive policing tool in Chicago found that the program — which was meant to predict who might be involved in shootings and provide them with social services — had no real preventative impact, instead resulting in police profiling. In Los Angeles, the LAPD inspector general revealed that the department did not even possess the data needed to measure the success of its predictive policing programs. Nothing suggests that police departments will be more forthcoming about new data fusion and AI analytics tools.

But police departments are not the only barriers to transparency. Law enforcement technology vendors make bold promises in marketing materials yet share few details about the products and services they offer. Companies commonly invoke intellectual property protections to withhold information about how their systems work from the public and the courts. And vendors often include nondisclosure agreements in their contracts with police departments. The NYPD responded to an open records request filed by the Brennan Center by claiming that it could not release information about test results or audits from its Palantir-powered predictive policing system because of a nondisclosure agreement with the company.

In addition, AI systems often generate outputs in a “black box,” making it all but impossible for users, auditors, or individuals whose rights are affected — let alone system engineers — to understand how they reach their conclusions. This built-in opacity is especially dangerous when it comes to data fusion tools: Introducing even one inaccurate input, such as a false gunshot detection, into a system that combines that data point with many other inputs to generate a threat score can both compound errors and complicate the determination of where the error originated and how to address it. Though some policing and surveillance experts have proposed solutions that would allow departments to better assess the efficacy of AI systems, police departments are understaffed and lack the expertise necessary to undertake this type of testing. As a result, despite numerous reports of unsupported claims by surveillance technology companies, police departments often take vendors’ assurances at face value.

This lack of transparency could hinder public pressure for adequate guardrails. Local lawmakers and the public would be more likely to push for regulation and oversight if departments were required to provide clear information about how AI-driven data fusion technologies work and how they plan to use them before deployment. Transparency is essential to ensure that police departments institute guardrails around their use of data fusion tools.

Municipalities and state legislatures should move quickly to regulate and oversee the use of advanced data fusion tools by police. The White House Office of Management and Budget (OMB), first under President Biden and now under President Trump, has already developed guidance that can serve as a model. That guidance is premised on the idea that “high-impact” AI use cases — systems used in a way that could significantly affect individual rights — should not be employed without meaningful restrictions and oversight. Jurisdictions should pass laws and departments should issue regulations that prevent police departments from using such AI systems without first adopting the risk management practices laid out by OMB, as described below. 

Identify High-Impact AI Systems

OMB designates some AI use cases as presumptively high-impact, including tools whose outputs can be fed into data fusion technologies (license plate readers, facial recognition technology, social media monitoring software, and video analytics). Police departments should disclose these technologies. OMB also labels capabilities offered by AI-based data fusion systems as high-impact, including individual risk assessments, crime forecasts, and violent activity detection. As such, police departments should automatically label their data fusion and analytics systems as high-impact and disclose their use.

OMB’s list is not exhaustive, however. Police departments should be required to identify and disclose the use of any AI tools that could materially affect an individual’s rights, like relationship mapping. AI systems used to investigate, deter, prevent, or respond to threats of criminal activity or risks to public safety — including any that may incorporate data from intelligence gathering — would likely fall into this category. Jurisdictions should implement procurement standards that require high-impact AI vendors to provide documentation that aids police in conforming to OMB risk management practices. 

Conduct Pre-Deployment Testing

While police departments often conduct trials for technologies before acquisition, there should be mandatory efficacy testing to demonstrate that a high-impact AI system works for its stated purpose. Jurisdictions should seek out independent assessments that test in real-world conditions. If independent testing is not available, then departments may need to conduct their own assessments.

To support independent field testing of AI law enforcement tools, the DOJ’s Bureau of Justice Assistance should award grants for under-resourced police departments to conduct assessments alongside technology developers and independent researchers, as was recommended by the Law Enforcement Working Group of the (now defunct) National Artificial Intelligence Advisory Committee (NAIAC).

Testing of high-impact AI tools should be closely regulated by an oversight body, and departments must refrain from taking any action based on a system’s output during the testing phase. Departments should publish the results of any efficacy testing. If testing reveals that a department cannot explain how an AI-based tool arrives at a conclusion, then use of the tool should not be permitted.

Perform Pre-Deployment Impact Assessments

Per OMB’s risk management practices, police departments should conduct and publish impact assessments that incorporate public feedback. Impact assessments should explain the purpose for deploying an AI tool, how it will be used, what the risks are, and how those risks will be adequately mitigated. Mitigation measures should include ensuring that AI outputs cannot form the sole basis for law enforcement action. Departments should also assess the range of information that police expect to put into the system and, where possible, the training data used in the tool’s development. These measures are vital in the data fusion context given the myriad potential data inputs — and their effects on possible outputs.

Impact assessments should articulate who outside the department will have access to data collected by the tool. If an AI system collects or analyzes sensitive information, then the department should be required to disclose whose information is collected, the conditions for access to that information, and the retention policies that apply. Rules should strictly prohibit entering data that would otherwise be subject to a warrant, court order, or subpoena (like geolocation or health information) or data that includes detailed information about persons not suspected of committing a crime. Police should be allowed to input such data only if they are able to obtain the information through judicial processes and only if the data use is strictly limited to the person and purpose for which it was collected.

Impact assessments should also identify the advantages and costs associated with an AI tool to determine whether the risks outweigh the potential benefits. In demonstrating potential benefits, such as promoting public safety or improving law enforcement effectiveness, departments should point to “specific metrics or qualitative analysis.” A department seeking to deploy a data fusion tool to identify people connected to persons of interest in a criminal investigation, for instance, should explain how the department currently identifies connections and how often. Departments should also disclose the accuracy rates of their current methodologies and provide quantitative data regarding the expected cost, accuracy, and error rate of the new tool.

Some costs can be easily measured: funding spent on a pilot program, resources devoted to maintenance or use of a tool, or hours spent training on the new system. Qualitative costs may be more difficult to calculate. These include risks to civil liberties, civil rights, or privacy, as well as the risk that officers will be more likely to defer to an automated tool’s findings. Information about the amounts and types of data collected, retained, and used by or for operation of the tool should also be considered, along with evidence of inaccuracies or bias and whether police use of similar technologies has resulted in lawsuits.

In identifying and assessing these risks, police departments should solicit feedback from independent experts, including those from civil society, academic, and community organizations. Finally, impact assessments should identify how police plan to mitigate any harms, following OMB guidance. In the case of a data fusion tool used to identify links between individuals, for example, a risk-mitigation plan might prohibit police from opening an investigative file on a person flagged by the AI solely as a connection to someone under criminal investigation. 

Institute Ongoing Oversight and Training

In accordance with OMB’s risk management practices, police departments should conduct ongoing assessments to monitor high-impact AI use and publish the results at least annually. An external body may be best situated to conduct these assessments, which should include a review of any material changes to the AI’s intended purposes, the deployment conditions, or the benefits and costs. OMB also recommends auditing AI tools’ outputs regularly for accuracy, bias, and harms and documenting and mitigating any issues. Police can look to the NAIAC Law Enforcement Working Group’s field test checklist, which synthesizes empirical AI testing methods and adapts them for the policing context.

An adequate number of people in the department should have sufficient knowledge about the system to identify potentially problematic outputs. Further, results from an AI system should be checked before any decisions are made or action taken based on its outputs. As with weapons, department employees should not be allowed to use high-impact AI systems without proper training. Documentation procedures, like those used for police stops and uses of force, should extend to high-impact AI tools as well; documentation should always be required when a tool’s output has contributed to a police decision or action. Departments should provide this documentation to affected individuals and allow them to request a review or appeal of the decision.

Ensure Compliance

As advised by OMB, if a police department does not follow the minimum risk management practices set out above, or if the benefits of using a high-risk AI system no longer outweigh risks to civil rights, civil liberties, or privacy, then the department should stop using the system.

To ensure compliance, jurisdictions should use external auditors, such as inspectors general, that have the expertise and capacity to analyze the technical aspects of AI systems within the policing context. The detailed breakdowns provided by the NYPD’s inspector general in its surveillance compliance audits provide a good model. Though inspectors general do not typically have the power to issue binding recommendations, their audits have resulted in the shutdown of harmful predictive policing programs and the implementation of stricter police surveillance policies.

Finally, some cities across the country have passed oversight and transparency laws and ordinances that require police departments to obtain approval from lawmakers and seek public input before acquiring or deploying new surveillance technologies. While these regulations have mixed records of success, they have encouraged lawmakers and officials to institute mechanisms requiring police departments to better communicate the capabilities of their technologies so that lawmakers can understand them before approving their use. Laws enshrining OMB-like oversight of AI systems would ensure that the framework set forth above governs AI-driven tools regardless of changes in police leadership or an inspector general’s office. 

Data fusion tools may represent the future of AI in policing, but they can also perpetuate the problems of the past. Predictive policing and social media monitoring tools illustrate how these new tools can magnify bias and inaccuracies while at the same time expanding the scope and reach of government surveillance and privacy intrusions. Yet police departments are adopting them with little testing or proof of efficacy.

Achieving much-needed transparency and implementing guardrails will remain challenging if police departments and vendors refuse or are unable to provide information about how AI-driven data fusion tools function. Following OMB guidance would go a long way toward ensuring that civilians, police officers, public servants, and experts can fully understand whether these systems’ purported benefits outweigh their harms, and toward putting in place protections for privacy, civil rights, and civil liberties. The risk of harm is too high to allow for the uncritical adoption and unregulated use of data fusion and analytics tools.