How to evaluate AI agent security platforms.
The agent-security market spans discovery, posture management, build-time controls, runtime controls, response, DLP, tool allowlists, identity, and MCP controls — and the categories overlap. This is a vendor-neutral checklist for cutting through that: the questions that actually separate the tools, the tradeoffs to weigh, and where each type of tool stops. Use it to evaluate anyone, including us.
A vendor honest about its boundaries is easier to trust on its claims.
First, get specific about the problem
Before comparing tools, name the moment you're securing. Agent risk isn't one thing:
- My agent reads untrusted content and could be steered by a hidden instruction.
- My agent has tools (MCP or otherwise) and could misuse them.
- My coding or ops agent can see secrets and files.
- My agent acts with my credentials and could leak them.
- I need to prove what an agent did.
- I need to deploy this safely across a team, an enterprise, or many customers.
Different tools address different moments. A tool that's excellent at discovery may do nothing at runtime; a content filter may not govern a tool call. Match the tool to the moment.
The checklist
1. Inspection point — where does it actually watch?
On the device where the agent acts, or in a central cloud service? On the wire between agent and model, at build time, or after the fact? Does it see tool calls, retrieved content, and network egress, or only the prompt and response? An agent's power is local and its attacks land at the action.
2. Enforcement — can it stop an action, or only report it?
Does it block, hold, or contain an action before it happens, or detect and alert after? Can it enforce at the OS level, or only cooperatively? If it can't inspect a request, does it fail open (forward) or fail closed (block)? Detection tells you what happened; enforcement changes the outcome.
3. Per-identity control — same action allowed for one agent, blocked for another?
Can you bound each agent to its job (tools, data, effects), or is policy one-size-fits-all? Is the boundary enforced on the observed action, or does it trust the agent's stated intent? Can a compromised agent move its own boundary? A boundary the agent can see or move isn't a boundary.
4. Data handling — does your content leave your environment?
Does raw content (prompts, files, tool arguments) stay on your machine, or is it sent to a vendor to inspect? What leaves by default — content, or only metadata? Can content-carrying telemetry be turned on by a single party, or does it require explicit consent? A tool that has to receive your content to read it is a tool you're trusting with your data.
5. Evidence — can you prove what happened, and can an outsider verify it?
Is there a tamper-evident record of every decision? Can an auditor who distrusts the vendor verify integrity independently, with no network and no vendor service? Is the evidence tied to the actual action? Trust you can check beats trust you're asked to extend.
6. Proof — can you validate the vendor's claims yourself?
Is there a public, reproducible benchmark you can run? Are the misses and false positives published, or only the wins? Can you run the same test against other tools and compare? A vendor that ships a clone-and-run benchmark, misses included, is making a checkable claim. Most don't.
7. Deployment — does it fit how you actually run?
One developer, a team, an enterprise fleet, or many customer tenants (MSP/MSSP)? How much setup and code change does it require? If it goes down, does your agent keep working? The best control you won't deploy protects nothing.
8. Honesty — does the vendor tell you where it stops?
Does it claim to solve everything, or is it clear about what it doesn't replace (guardrails, governance, secrets management, tool-side authz, sandboxing)? Are the platform limitations stated? A vendor honest about its boundaries is easier to trust on its claims.
Where each type of tool stops
- AI governance platforms define, validate, and document controls; they don't enforce an agent's action at the wire.
- Prompt guardrails filter inputs and outputs; they don't govern the tool call or the egress that follows.
- Agent frameworks / runtimes help you build and host agents; they aren't a security control over the agent's behavior.
- MCP gateways manage connections and protocol; they aren't inspecting agent intent and local data egress.
- EDR / XDR / SIEM watch endpoint and security events beside or after the agent's action; they weren't built for the agent decision path.
- Sandboxes isolate code execution; they don't necessarily inspect retrieved content, tool calls, or data egress.
Most real deployments combine several. The checklist above helps you see which moments you've covered and which you haven't.
How Crawdad answers the checklist
We'll be direct about where we land: Crawdad inspects and enforces on the device, on the wire, before the action, sees tool calls and retrieved content and egress, governs per-identity on the observed action, keeps raw content on your machine (metadata-only by default, dual-consent to change), produces a verifiable hash-chained record, and publishes a public, reproducible benchmark of 497 real attacks with the misses included. It runs from one developer to an MSP fleet, needs one env var and no code changes, and fails closed. And it does not replace guardrails, governance, secrets management, tool-side authorization, or sandboxing — it's the runtime-enforcement layer in a stack that includes them.
Every criterion above is one Crawdad answers — stated as neutral evaluation, not marketing.