Guide4 min read
The security reviewer's guide to AI agents
The questions to ask any agent vendor about data, permissions and logging, with what a good answer sounds like.

Most vendor questionnaires were written before software could take actions on its own. They ask about encryption at rest, data centres and incident response, which still matter. They rarely ask what the product can do without a person, or how you would find out if it did something unexpected. This guide adds the questions they miss, grouped into four areas, with what a good answer sounds like and where that answer should be written down.
Data
Start with the basics, then go one level deeper than usual. An agent reads more of your systems than a typical SaaS tool, so the scope of what it sees matters as much as where it is stored.
Ask these questions:
- Where is our data stored and processed, including any model providers you use?
- How long are prompts, drafts and source extracts kept, and can we shorten that?
- Is our data used to train or improve any model, yours or a third party’s?
- What happens to our data when the contract ends?
A good answer is specific. “Drafts and source extracts are kept for 30 days by default and you can shorten the period per workspace” is a good answer. “We take privacy seriously” is not. On training, the answer should be a plain no, with no exceptions for “service improvement”.
Ask for these answers in the contract or the data processing agreement, and treat a web page as a summary only. Pages change. Contracts are what you can enforce.
Permissions
This is where agent reviews differ most from ordinary SaaS reviews. You need to understand what the agent can reach and what it can change.
- What access does each connection need? A good vendor can start every connection read-only and explain exactly which scopes write access adds.
- Is write access granted per action? “Can update tickets” is too broad. “Can add an internal note to a ticket” is the level you want.
- Can permissions differ between agents? An HR agent and a security agent should not share a set of rights.
- How is user access managed? Expect SSO through your identity provider and SCIM provisioning, so leavers lose access when they leave.
If a vendor needs broad write access on day one to show value, that is worth noting in your review.
Actions
Ask what the agent can do without a person, and listen carefully to the answer. The safest design is one where the agent drafts and a named person approves every action before it runs. Some vendors describe “autonomous mode” or confidence thresholds that let actions through unattended. Ask who sets those thresholds, whether you can turn them off and whether an unattended action looks different in the log.
Then ask how you would find out if the agent did something unexpected. A good answer covers alerts, the log and a way to pause an agent immediately.
“The question that sorted vendors fastest was simple: show me an action that happened without a person. The good ones had nothing to show.”
Elin Sørensen, Security Lead, Northpass
Logging and evidence
Your auditors will ask you to prove what happened, so the log needs to answer that question on its own. Ask to see a real log entry from a test workspace during the review.
A useful audit log records:
- What the agent read to produce each draft
- The draft itself, and any edits a person made
- Who approved or rejected it, and when
- What action ran as a result, and in which system
You should be able to export the log in a standard format and keep it in your own systems for as long as your retention policy requires. Ask whether exports include deleted drafts and rejected actions, because those are often what an investigation needs.
Running the review alongside a pilot
A security review does not need to block all progress. A sensible pattern is to approve a short pilot of four to six weeks with read-only access to one or two systems, while the full review continues. The pilot gives you real logs to inspect, and the read-only scope means the risk during that time is limited to what the agent can see.
Write down the conditions for moving beyond the pilot before it starts: which questions must be answered in the contract, which permissions you will consider adding and who signs off. That way the end of the pilot is a checklist the team already agreed.


