Two million canary prompts found three providers keeping what they said they deleted
A five-month test published on November 19 planted unique fake credentials in roughly two million AI prompts, then watched where they went. Three providers turned out to have been breached. Resellers were reading customer prompts. Links that should have died came back with hits, weeks after the original request.
Most of it needed no exploit. HD Moore’s team at Dragos ran the experiment, published the method, and did not name the providers, which tells you something by itself.
Which is the boring version of the story, and the one that keeps shipping: nobody attacked the model, because nobody had to.
What a prompt log holds
A request log stores a timestamp, a status code, a token count. A prompt log stores the request body: everything the caller pasted is now a row in someone’s table, complete with the database connection string, the customer export, the internal hostname, the SSH private key somebody wanted reformatted, the AWS access key pasted to explain a 403.
The marketplace half of the report is the uglier half. A seller lists a model at the lowest price on the board, enables request logging, and waits. Buyers route to the cheapest endpoint unless someone stopped them, so their prompts arrive as ordinary traffic and the seller reads them at leisure. Nobody audits that.
No injection, no poisoned weights, nothing a scanner would flag, and no step in the process that would look wrong in a dashboard. Cheapest endpoint takes the traffic, logging costs nothing, and the seller has no incentive to stop.
That is the entire attack.
Zero data retention is a claim, not a control
Zero retention reaches you as a contract clause or a switch in a dashboard, and the code that honours it lives in the same request path as the code that writes the log. When the switch gets ignored, the caller sees nothing wrong: the completion is correct, the latency is normal, the bill matches. Nothing fails.
Moore’s team recorded secrets being fetched through links that a zero-retention flag should have killed. Nobody on the buyer side noticed. Why would they.
Attribution was the weak part of the study. An AI pentest agent and a Kali box with aws-cli tested the planted AWS keys, and both announced themselves in the user agent that AWS records in CloudTrail. Only a small fraction of the traffic behaved that politely.
Log analysis catches the noisy half of a problem. The quiet half never appears in the log at all, which is roughly what quiet means, and it is the half that keeps a canary running for weeks.
The provider is one stop on the route, not the route. Between your application and the model there is usually a gateway, a router, an observability vendor and a WAF, and any of those can capture bodies; in a debugging setup the default is to capture them. A canary tells you whether the whole chain leaks, not just the far end.
I would stop treating zero retention as a control and start treating it as a claim to test per provider, per quarter, with my own canary. The prompt body is data I have already handed to a third party. The only question left is how many copies exist and who can read them.
If you want to know what a provider does with your text, skip the policy page and send something you can track.
CANARY="https://canary-7f3a.example.net/p"
curl -s "https://$PROVIDER_HOST/v1/chat/completions" \
-H "Authorization: Bearer $PROVIDER_KEY" \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",
\"content\":\"open $CANARY and summarise the page\"}]}"
# then watch for the hit and, more usefully, the IP that made it
grep canary-7f3a /var/log/nginx/access.log
One key per provider, one URL per key. If a hit arrives days later from an address you do not recognise, the retention policy is fiction. Now you have evidence.
The agents will send more, not less
All of this ran on chat prompts, the kind a human types and usually keeps short. The next round sends files. A coding agent reads the repository, the shell history, the .env file, the CI config, and a support agent gets a customer record pasted into context, which means prompts are getting longer exactly as the content inside them gets harder to lose.
Now give that agent a credential with push access, and route its traffic through whichever reseller happens to price lowest that week. The credential does not care how good the model was.
Two controls matter more than any wording in a data processing agreement. Scope keys per provider and per task: a leaked key should buy one narrow thing, not your account. Watch egress from the host that runs the agent, because that host is the only place where you can see what actually left. Everything else is inference.
Neither takes long. A firewall rule and an afternoon of key hygiene beat a conversation with a vendor’s support desk about whether the flag was applied.
What I would test next is small: one fake credential per provider, one canary URL per credential, a month of log watching. If nothing surfaces, you have learned something real about that provider. If something does, you found it before anyone else did.