Most of the criticism aimed at Blue Voice will focus on the wrong thing. The company just raised $6 million to build what reporting has described as a “Harvey for police officers” — an AI assistant giving cops real-time policy guidance in the field. Critics will argue about whether AI belongs in policing at all. That debate matters, but it isn’t the one I find most urgent. My concern is narrower and, I think, more immediate: this is an AI system whose outputs shape legally consequential decisions, deployed in an environment where the people it interacts with have every incentive to manipulate it.
That is not a philosophical problem. It’s a security problem, and it’s one the industry has not solved.
What makes this deployment different
Harvey, the legal AI product Blue Voice is being compared to, operates in an office. A lawyer reads its output, applies judgment, and takes as long as needed. If the model produces something wrong, there are review layers, billing records, and a partner who signs off. The failure mode is slow and correctable.
An officer asking a policy question during an active encounter has none of that. The decision window is seconds. There is no second reader. And the output isn’t a research memo — it’s guidance that may inform whether someone gets searched, detained, or worse. The gap between “model was wrong” and “rights were violated” collapses to nearly zero.
Security people have a term for this: the blast radius. In most enterprise AI deployments, a bad answer costs time or money. Here, a bad answer costs something you can’t refund.
The attack surface nobody is scoping
Think about what an AI assistant in the field actually consumes as input. Almost certainly speech. Possibly transcripts of the encounter, dispatch data, records lookups, or contextual information about location and prior calls. Every one of those channels is untrusted input, and untrusted input reaching a language model is the oldest unsolved problem in this field.
Consider a few scenarios that follow directly from how these systems work:
- Voice-channel injection. If the assistant is listening to an encounter, anyone within earshot is effectively typing into the prompt. A person who understands the system could speak phrases crafted to steer its guidance.
- Poisoned retrieval. Policy guidance systems almost always pull from a document corpus. Whoever can edit that corpus — a department administrator, a vendor, a compromised account — can change what the model says the law permits. There’s no visible difference between a legitimate policy update and a malicious one.
- Confident wrongness. The failure that needs no attacker at all. Legal reasoning depends on jurisdiction, on case law that shifts, on facts the model can’t see. A model that hedges is useless in a five-second window. A model that doesn’t hedge is dangerous.
- Authority laundering. This one worries me most. “The AI told me it was permitted” is a phrase that will show up in a courtroom. When a system’s output can be cited as justification, that output becomes a target worth attacking.
Six million dollars buys a product, not a threat model
I want to be fair here. A seed round is a seed round. Nobody should expect a formally verified system at this stage, and the founder deserves credit for identifying a real gap — officers genuinely do make policy calls without reliable reference material.
But the funding environment around this is instructive. The same week’s news included Jeff Bezos’s Prometheus raising $12 billion for an “artificial general engineer,” and Flock’s CEO calling for “compromise” as his surveillance company absorbs public backlash. That second story is the relevant one. Flock built infrastructure first and is negotiating trust afterward. It’s a bad sequence, and it’s the default sequence in this industry.
Policing technology has a specific history of doing exactly that: ship, deploy, and address the objections once contracts are signed. Adding a language model to that pattern doesn’t change the pattern; it just makes the failure harder to audit, because now the reasoning happens inside a model nobody can inspect.
What I’d want to see
The questions I would ask Blue Voice are not hostile ones. They’re the questions any team building AI into a high-stakes workflow should have answers to. What is the input validation story for the voice channel? Who can modify the policy corpus, and is there an audit trail? Does the system log its own guidance in a way that survives litigation? Are there hard refusals — categories where the assistant declines to answer rather than guessing? Has anyone red-teamed it with an adversary who wants to change its answers, not just an evaluator checking accuracy?
None of that is exotic. It’s baseline practice for any AI deployment where the output has teeth. The problem is that “AI safety” in commercial products usually means content filtering, and adversarial input handling gets treated as a later problem.
An AI assistant advising officers on the limits of their authority is not a later problem. Build the threat model now, while the product is still small enough to change.
🕒 Published: