Does AI Fight or Fuel Election Disinformation?

We tested how well chatbots could correct election falsehoods when questioned, including by users demonstrating mistrust of election processes or results. And we assessed how easy it would be for someone to use AI tools to create disinformation campaigns. We conducted our tests between February and August 2026. For more information, see our methodology page.

There’s a surprising but encouraging bit of news: The chatbots we tested consistently pushed back against the false tropes that have been central to election disinformation campaigns. They even did so in the face of persistent questioning. And though the chatbots often made mistakes, they also behaved as sympathetic listeners to skeptical users — while still refusing to endorse conspiracy theories. This suggests they could play a constructive role in fighting election disinformation. 

But this good news comes with some important caveats. The most critical relates to our second major finding: When we asked AI models to generate misleading election-related images and video, they easily did so. Many of their safeguards for preventing this were ineffective, inconsistently applied, or functionally absent.

Finally, most models we tested did a poor job identifying that the content we created was generated by AI. Sometimes, they insisted that these media were real and depicted actual events. This, along with other findings from our study, suggests that in an ongoing election with little authoritative information for models to rely on, AI may not be as helpful in pushing back against unsubstantiated rumors and misinformation.

An important note: This report does not disclose the prompts we gave the AI platforms to generate images, video, and audio, because they could be misused to create and spread disinformation. For more information on this, please consult the section on prompt library characteristics in the methodology.

Asking Chatbots Questions About Election Conspiracy Theories

We first examined how six of the most widely-used AI products — ChatGPT, Gemini, Grok, Claude, Perplexity, and DeepSeek — responded to questions regarding common election conspiracy theories.

Our tests focused on false tropes the Trump administration and its allies have used to undermine election results: (1) Noncitizens register and vote in substantial numbers; (2) voting machines are rigged; (3) election officials commit or support widespread fraud; (4) mail voting is rife with fraud; and (5) ballots cast on or before Election Day but counted after are illegitimate. We also tested a disproven conspiracy theory that gained traction among Trump’s opponents: that Elon Musk’s internet provider Starlink was used to rig elections.

Here’s an example of an interaction we had with Gemini.

While the chatbots consistently challenged misinformation, this good news needs to be tempered. 

All the chatbots made mistakes or cited nonexistent, irrelevant, or erroneous sources. Half of all responses contained an inaccuracy or a bad citation. One in every three responses had a factual error, and one in three contained broken links or misleading citations.

These persistent errors could undermine chatbots’ credibility. And when models make mistakes, even if they dispute the main conspiracy theory, they might still perpetuate disinformation.

Although our prompts did not name a specific election, the models tended to frame their answers around 2020 and 2024. They relied on extensive reporting that documents and debunks the fraud claims surrounding those elections. 

Most viral U.S. election conspiracy theories are based on familiar tropes, such as rigged machines. When a new disinformation campaign relies on similar claims, the models will likely be skeptical of those claims. But they will have few trustworthy sources to cite when refuting emerging conspiracy theories. To make matters worse, people may deliberately seed lies online for chatbots to find and repeat back to users, a technique known as data poisoning.

The errors that chatbots make, however trivial, could contribute to confusion around election processes.

How AI companies prioritize accuracy isn’t exactly clear. Some observers argue that AI companies, unlike social media platforms, compete to give customers accurate, objective tools, which creates an incentive for reliability. Yet recent research suggests that models might be incorporating advertisements in chatbot responses. Indeed, OpenAI, which runs ChatGPT, has introduced advertising into the chatbot interface. The company has said it will not allow political advertising, but that applies only to the current election cycle.

Liability adds another layer. Social media platforms are largely shielded from liability for content their users post under Section 230 of the Communications and Decency Act. The companies that make AI tools, however, may be held responsible for what their models generate. After all, chatbots do much more than shape users’ exposure to content. They craft a single, authoritative answer, a role similar to that of traditional media. Courts have yet to settle whether that exposes their developers to greater liability.

Another issue involves AI companies’ political preferences. In 2025, xAI changed Grok’s system prompt to instruct it to “not shy away from making claims which are politically incorrect, as long as they are well substantiated.” Soon after, the chatbot began praising Hitler and referring to itself as “MechaHitler.”

AI companies can, in both subtle and explicit ways, influence the quality of the information their users receive.

Using AI Tools to Create Election Disinformation Media

We next tested six popular generative AI tools to see if we could create realistic images, video, and audio depicting supposed election-related malfeasance. The tools were ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux.2. All of them have safeguards that are meant to prevent users from creating deceptive content on sensitive topics. Despite those protections, it was far too easy to do so.

Here’s how the multistep process worked. First, we asked four popular chatbots (ChatGPT, Gemini, Grok, and Claude) general research and strategy questions, such as how to frame disinformation scenes convincingly. All four helped at this stage.

Next, we compiled their answers ourselves, by hand, into a single set of instructions we call the seed prompt. The chatbots did not write the seed prompt; we assembled it from their research.

Finally, we asked the chatbots to generate a full set of image-generation prompts based on the seed prompt. In principle, this approach could generate a nearly limitless number of prompts to create election disinformation.

But this is where the chatbots diverged. Most refused the request; for example, one stated, “I can’t help write direct image-generation prompts whose goal is to create realistic, convincing false election claims.” 

Grok was the exception, producing the complete set and even pulling from X to build convincing narratives. In its “chain-of-thought” text — the temporary response the chatbot shows users while it is generating an answer to a prompt — Grok acknowledged that some of the false claims we were prompting “may incite unrest and contest elections” yet nonetheless stated, “election misinformation not listed as disallowed activity.”

We reran our attempt to draft the image prompts in July 2026. Again, most chatbots refused. This time, Grok’s chain-of-thought no longer explicitly stated that election disinformation wasn’t prohibited — but it still created the prompts.

All it took to produce these prompts was a handful of requests, eliminating the technical skill and resources needed to make convincing deepfakes in the past. The same process could easily be scaled.

After Grok generated our derived prompts, we used them to ask ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux.2 to create fake election images and video. 

All the chatbots we tested agreed to generate images for us that could be used to spread election disinformation. (The images below include watermarks stating that they were generated by AI. We added these watermarks after we completed the tests.)

The companies whose tools we tested have rules, known as acceptable use policies, on the ways people can use the tools. These include provisions against deception, election interference, and recreation of a person’s likeness without consent. Some of the images and video we created — if we shared them publicly and claimed they were authentic — could violate those rules. But the policies vary. Some restrict how AI content is used or generated; others prohibit deceiving or misleading people, leaving unclear whether generating disinformation, absent of distribution, is itself a violation. Companies rarely disclose how they enforce these policies.

Open-weight models are AI models that can be downloaded and run on a user’s computer rather than through a company’s website. There is, therefore, no company record linking the content generated by open-weight models to whoever created it, making it harder to hold users accountable. Flux.2, the only open-weight image model examined in this test, accepted every prompt we offered and often produced photorealistic images styled as smartphone photographs. 

All the models we tested rarely rejected requests outright (for more information, see our methodology). Instead, they often responded with what’s called a soft rejection. 

With soft rejections, AI tools don’t create a realistic image. They purposefully generate highly stylized versions of requests, adding illustrative outputs, rainbow coloring, or pop-art styling. Or they swap out elements of the scene. According to OpenAI, ChatGPT’s image generation tool will deliberately “transform adversarial requests into safe ones rather than simply producing the requested violative content.”

We also saw inconsistencies in safeguards across different versions of the same model. Most chatbots have a premium, more complex “thinking” version, as well as a lower-tier, less complex “instant” version.

Of every model we tested, Meta AI was the most inconsistent: Its versions gave contradicting acceptances and denials to the same prompt 67 percent of the time. ChatGPT did so 28 percent of the time, Grok 22 percent, and Gemini 10 percent. Gen-4 and Flux.2 never differed — because neither version rejected a single prompt. (These figures reflect only whether each version generated an image or refused, not whether the image was convincing.)

As our tests reveal, the safeguards developers have put in place against using generative AI tools for election disinformation are often ineffective, inconsistent, or easily circumvented.

Asking Chatbots to Identify AI-Generated Election Disinformation Media

Social media platforms do a poor job of identifying and labeling AI-generated content, even when it contains embedded watermarks or metadata identifying it as such. 

We tested whether six major chatbots — ChatGPT, Gemini, Grok, Claude, Perplexity, and Meta AI — could identify AI-generated images, including images they had generated themselves.

For the most part, they could not.

(The images below include watermarks stating that they were generated by AI. We added these watermarks after we completed the tests.)

Four of the chatbots that we tested had also generated some of our content. Gemini identified all three images its tools had created as fake, and Meta AI identified three of four. ChatGPT and Grok did not identify any of the images their own tools had made as fake. 

After we completed our testing, OpenAI released a public tool that uses metadata and watermarks to verify whether an image was generated by its systems. We tested ChatGPT again after the update: It detected some of the images it had created but not all of them.

On August 2, 2026, the California AI Transparency Act (CAITA), which requires AI companies to embed provenance data in AI-generated content and create new tools to detect that data, went into effect. We reran the detection tests after the law went into effect but saw little improvement. Most chatbots still couldn’t identify the images as AI generated. The one exception was Gemini, which had changed to reading the provenance data embedded in a file rather than analyzing the image itself; it flagged the images that carried that data but was unable to identify the rest.

We also conducted the detection tests in Spanish. Most models did about as well as they had in English. Interestingly, Grok did notably better in Spanish. Gemini, however, did far worse, particularly for images generated by other tools. Its SynthID watermark detection program, which flags content made by Google’s own tools, kept working in Spanish; for images from other tools, which it has to analyze visually, its performance dropped sharply.

Our study found that AI chatbots consistently pushed back on election-related conspiracy theories, but also that they sometimes erred on the details, cited nonexistent sources, and often failed to detect that media was AI generated. When the models created content for us, they often offered workarounds to their own guardrails. They helped us write the prompts and suggested tweaks needed to generate deceptive content. They also frequently added realistic details that were not in the prompts, making the fabrications more convincing. 

Recommendations

So, what can be done?

Chatbot operators should ensure that chatbots verify all their cited sources before returning an answer, continuously auditing outputs for accuracy. Companies should engage with civil society organizations and fact-checking bodies whose work informs how chatbots respond to election-related questions. One model, Claude, recently began appending a disclaimer to some election-related responses noting that its information might be outdated and pointing users to authoritative civil society sources. Other companies should follow suit. All chatbot operators should increase human staffing and computing resources for monitoring during the months leading up to and following major elections.

Transparency about limitations, clear citations, and human oversight are essential. Operators should ensure that their chatbots do not claim to be able to distinguish between real and AI-generated images when they clearly cannot. These same companies should ensure that their chatbots refuse to generate prompts that could be used in election disinformation campaigns, as many (though not all) have already done.

Tech companies need to make it easier for users to identify AI-generated materials. They already have the tools to do this, via watermarks or other labels. But their use of these tools is inconsistent. But labeling is not a solution on its own. As our study shows, detection is imperfect, and systems can both miss AI-generated content and wrongly flag authentic content, so labels must be designed and communicated carefully to avoid misleading users. Policymakers must hold generative AI companies accountable on this front. 

CAITA, which took effect on August 2, 2026, is a compelling, albeit incomplete, model for legislation. It requires AI companies to embed provenance data in AI-generated content and provide free, publicly accessible detection tools. These are typically standalone tools that are not embedded in the chatbot itself and require the user to access them through a different website. Even when chatbots do check the authenticity of uploaded images, none yet offer users the reliable, cross-source detection CAITA will require of large platforms by 2027. This law will require large social media platforms to detect and manage provenance data embedded in the content shared on their sites. It is unclear whether this requirement will also apply to chatbots that users are likely to use to evaluate content. 

New legislation should require that AI-generated content be identifiable across platforms, not just by the system that created it. The European Union’s AI Act requires machine-readable marketing of AI outputs, though its interoperability provision is qualified by what is technically feasible. U.S. legislation could go further. Chatbot companies should also be required to detect AI-generated content that users upload and to build watermark detection into their interfaces. Further, people should be able to hold companies accountable if they violate these policies. To that end, lawmakers should establish a private right of action allowing people to sue companies in civil court for violations of the law. 

Tech companies must also strengthen and enforce their restrictions on election-specific content, clearly banning the creation of deepfakes of government officials, government insignias, and election infrastructure. They should dedicate ample resources to pre-deployment guardrail testing for election-related content. Internal policy teams should consider how often their model denies a user’s request but suggests a change to the user’s prompt, test the qualities of those suggestions, and consider outright denying more requests. They should also work to ensure that the variety of AI models and tools that they offer issue rejections consistently.

In addition, well before Election Day, journalists, election officials, and civic groups should preemptively debunk the tropes that are central to disinformation campaigns. Election officials can explain existing voting safeguards and election administration processes and can boost their content for chatbot outputs (a technique known as generative engine optimization, or GEO) to ensure that models refer to accurate primary sources. Civil society organizations can facilitate digital literacy training that prepares people to recognize common conspiracy theories. News outlets can continue to publish stories on voting machines, mail ballots, and election results early and with critical context. Publicizing information that counteracts false election rumors shapes the information environment that AI tools use to respond to queries.

Third-party researchers should be allowed to conduct rigorous, independent studies of AI tools. Too often, companies make users agree to terms that restrict the testing of their products. Companies and policymakers should protect these researchers with a legal safe harbor, allowing them to conduct adversarial tests that may expose the dangerous aspects of AI tools.

Finally, while our study shows that chatbots are generally good at rebutting long-debunked election conspiracy theories, it is less clear that they would do as well with AI-generated content presented as a breaking event. This kind of disinformation is designed to provoke outrage or alarm. Voters need to pause and verify such claims before believing them. They should use trusted sources, such as official election office websites and independent fact-checkers like PolitiFact or AP Fact Check, before sharing information, and they should be especially skeptical of scandalous content that surfaces close to Election Day.

Access our methodology here.