Where classification physically executes decides whether your text ever leaves the machine, and it is the single most consequential difference between these products. Everything else in an evaluation follows from that answer.
The six questions
Architecture is what you cannot change after signing.
Most evaluation grids compare feature names, which every vendor can match. These six get at architecture, and architecture is what you cannot change after signing.
- 01
Where does classification execute?
On the endpoint, in your cloud, or in the vendor's. If the prompt is sent somewhere for inspection, the content now exists in that system.
- 02
What is stored, and is it reversible?
A masked snippet and a finding, or the message. If anything is reversible, who holds the key.
- 03
Which surfaces are not covered?
Insist on the list. Browser, desktop apps, file uploads, images, agents and IDE assistants all differ.
- 04
Can it act, or only alert?
Warning before the send and reporting afterwards are different products. Ask what happens when detection is slow.
- 05
What is the false positive rate?
On what corpus, measured how, and published or asserted. This is the number that decides whether people keep the tool.
- 06
What evidence does it produce?
For an auditor, a works council, and an incident. Coverage per device matters more here than a dashboard.
How to test it
Run it on something you would not want leaving.
A demo on the vendor's data proves the vendor can find the vendor's data. Bring a document of your own, with the identifiers your organisation actually holds, and watch what the console receives.
Then capture the traffic. If the product claims the text never leaves the device, that claim is checkable in an afternoon with a packet capture and a string search, and the answer is binary.
What not to weight
Detector count is the least useful number on the page.
Every vendor can add a regular expression. The count of built-in detectors tells you almost nothing, because the hard part is context: whether a sixteen-digit number next to the word example is a card number, and whether four identifiers together make a record.
Coverage claims deserve the same scepticism. A surface reported as quiet might be quiet because nothing happened or quiet because nothing was watched, and only a product that reports per surface can tell you which.
What it costs to run
Ask what the tuning takes, not just what the licence costs.
Licence cost is the number in the proposal and it is rarely the number that decides whether the deployment survives its first year.
The costs that arrive afterwards are packaging and deployment through whatever management tooling you run, the consultation your jurisdiction requires before anything is switched on, and the tuning loop: somebody, regularly, deciding which rules are too noisy and which categories are missing. Ask how long a vendor's existing customers spend on that in a typical month, and ask whether changing a rule requires professional services or is something your own team can do at two in the afternoon.
The last one is more predictive than it sounds. A product where tuning is a support ticket ends up untuned, which means it ends up noisy, which means people stop reading it. Precision is not only a detection property; it is also a function of how easily the people who live with it can adjust it.
The reference call
Ask a reference which rules they turned off.
Vendor-supplied references are selected and will say good things, which is fine. The value is in the questions that are awkward to answer well rather than in the overall verdict.
Four are worth the call. Which detectors did you turn off, and how long did that take to decide, which gets at precision in a way no datasheet does. What broke when a provider changed their interface, and how did you find out, which gets at whether coverage is maintained or merely claimed. What did your works council or legal team object to, which surfaces the rollout problems before you meet them. And what do you still not have visibility of, which is the question that produces the most honest answer of the four.
Ask how long from signature to first useful report. The gap between procurement and anything appearing in a console is where these projects quietly stall, and it is rarely the software that is slow: it is packaging, approvals and whichever consultation the organisation owes its staff before anything is switched on.
Questions
What is the single most important question?
Where classification physically executes. On the endpoint means the content never reaches the vendor. In the vendor's cloud means it does, whatever the retention policy says afterwards. Every privacy, works council and regulatory conversation downstream turns on this one answer.
How do I verify the on-device claim rather than trust it?
Capture the traffic during a test send and search it for a string you planted in the prompt. If the string appears, classification is not local. It is an afternoon of work and it settles the question better than any datasheet.
Is a higher detector count better?
Rarely. Detector count is easy to inflate and tells you nothing about precision. A product with eight well-validated detectors and good context handling will outperform one with forty regexes that fire on anything numeric.
Should we care about false positives at the evaluation stage?
It is the thing that decides whether the deployment survives. A tool that interrupts people wrongly teaches them to dismiss warnings, at which point you own the cost of the control and none of its benefit. Ask for the rate, the corpus, and whether it is published.
How long should a pilot run?
Long enough to see a normal month. Two to four weeks in log-only mode on a mixed group of about a hundred people gives you real numbers and no disruption, and the findings usually reshape the policy before anything is enforced.
What about tools built into the platforms we already own?
Worth using where they fit. The native controls in Microsoft 365 or Google Workspace sit close to data inside those tenants. They do not see a prompt typed into a third-party AI tool in a browser tab, which is the gap this category exists to cover.
Build or buy?
A regex pipeline is a weekend. Context handling, surface coverage, enforcement that does not break the tools, and the evidence trail are the years. We wrote out the honest version of that comparison separately.
What single question separates these products fastest?
Where classification physically executes. On the endpoint and in the vendor's cloud are different products with different risk profiles, and for a covered entity the second creates a disclosure of its own. Ask for the answer in writing, because it decides most of the rest.
How should we run a proof of concept?
On your own data, with a prepared set: ten real documents, three that look sensitive and are not, two that are sensitive only in context, and a screenshot. Watch what happens to the sample during the test as well as the verdict, because an evaluation that uploads your sample is itself a disclosure.
What should we ask a reference customer?
Which detectors they turned off, what broke when a provider changed its interface and how they found out, what their works council or legal team objected to, and what they still have no visibility of. The last one produces the most honest answer.
Keep reading
- Glossary AI DLP Data loss prevention applied to the prompt layer, and why files, email and network traffic are the wrong place to look. 4 min read
- The comparison AI DLP vs CASB vs endpoint DLP What a CASB, endpoint DLP and a web gateway each inspect, and the surface that falls between all three. 6 min read
- Build or buy Build or buy AI DLP The prototype is a weekend. The parts that take years are context, surface coverage, enforcement that does not break the tools, and the evidence trail. 6 min read
- Glossary Coverage gap A surface a control does not inspect. The dangerous ones are silent: indistinguishable from a surface with nothing on it. 3 min read