AI System Security Assessment & Penetration Testing
Testing AI agents, LLM services and AI infrastructure as targets of attack
We put AI agents, LLM services and AI infrastructure to the test from the attacker's side first, and confirm the real risks, whether before launch or in service.
From the tools and permissions an agent uses to the models, data and infrastructure behind them, we look for weaknesses across the whole AI system.
Under attack, does our AI system stay inside the lines we have set?
Whether any path crosses the line, what it would mean if one does, and records that let you check it again.
Expert reviewed
When it fits
- You are bringing in an AI agent or launching an LLM service
You have added an AI chatbot to your service, or are bringing agents into your work, and want to know that they keep to your lines when someone attacks them.
- Agents are being given more tools and access
The more an agent can use, from email and calendars to internal systems and outside tools, the more you need evidence of how far its permissions reach and where a person still has to approve.
- You are connecting internal documents and data to AI
If retrieval-augmented answers now draw on your internal documents, we check that their contents and sensitive information never reach staff who should not see them.
- Several teams share training and inference infrastructure
Several teams share GPU clusters, HPC (high-performance computing) environments and storage, and you want to know that each team's training data and model files are properly kept apart.
The SAFE SQUARE approach
Lines first
We first write down, in plain sentences, the lines the AI must not cross, and judge everything by whether they hold under attack.
Knowing how agents are built
Drawing on SAFE SQUARE's own research into how agents plan, use tools and handle permissions and approvals, we map which tools an agent uses with which rights before we test it.
Beyond the answers
From what the model says, we follow on into the tools the agent uses, the systems it connects to, and the isolation of the training and inference infrastructure.
What we look at
Our focus is AI agents, LLM services and AI infrastructure, along with the places where they connect to one another and to your other systems.
AI agents
- The target
- AI agents that work through email, calendars, internal systems and outside tools, together with the environment they run in.
- What we can test
- The reach of tool and API permissions
- Where actions need a person's approval
- How outside input (documents, web pages, email) affects behaviour
- Data leaving through the agent
- The execution environment and its isolation
- Action records and traceability
- Authentication to connected systems
- An example line
- ‘The AI agent sends nothing outside without a person's approval.’
LLM services
- The target
- Services built on language models, such as chatbots, retrieval-augmented answers and summaries, with their models, data and integrations.
- What we can test
- Prompt injection, direct and indirect
- Exposure of sensitive data and system instructions
- Permissions on the documents it retrieves
- Output handling and its effect on connected systems
- Whether safety policies work as intended
- Misuse of usage and cost
- The model and data supply chain
- An example line
- ‘Staff get answers only from documents they are allowed to see.’
AI infrastructure (GPU and HPC)
- The target
- GPU clusters and HPC (high-performance computing) environments used for training and inference, with their schedulers and shared storage.
- What we can test
- Job schedulers and management interfaces
- Isolation between users and teams
- Permissions on shared storage and training data
- Access to model files
- The software stack, such as drivers and runtimes
- Networking between nodes
- Inference service APIs
- An example line
- ‘One team's training data and model files cannot be read or changed by another.’
The areas and lines are examples.
ScopeHere the AI system is what is being attacked. Evaluating and improving how well AI agents carry out security work belongs to Security Capability Development for People & AI. For researching and building AI models and agents for your security work, see Technical collaboration and joint research.
How it runs
Agree targets and lines
We agree with you the AI systems and integrations to test and the lines they must keep.
Map the system
We set out with you the tools and permissions the agent uses, how models and data flow, and the connected systems and execution environment.
Test and validate
Our experts make the attempts a real attacker would, and check by hand whether the lines you have set hold under attack.
Explain and re-check
We explain the findings, real impact, evidence and fixes to decision-makers and the technical team and, if you wish, check the fixes again the same way.
Keeping it safeOn a live service we set test accounts, scope and stop conditions first, so that real users and data are not affected. The principles every engagement follows, safe conduct and stop conditions, facts kept apart from assumptions, and how material is kept and destroyed, are set out under How we work.
What you receive
Lines held and lines crossed
For each agreed line, whether it held and, where it did not, through which input, permission and path it was crossed.
Findings and real impact
What each finding would mean for the business, such as data exposure, actions beyond permission or runaway cost.
Reproducible evidence
The inputs, conditions and records needed to confirm each result again.
What to fix and check again
What to fix first, and what to check again after a model or integration changes.
A workplace assistant agent that answers from internal documents and can send email
Connected to: internal document search, sending email, calendar lookup
| Line | Result | Evidence |
|---|---|---|
| Answers use only documents the user may see | Held | Documents are filtered by the user's permissions at the search step, so out-of-bounds documents were never used in answers. |
| No email leaves the company without approval | Crossed | Following an instruction hidden in a document received from outside, the agent had a path to email an outside address without approval. |
| System instructions and integration keys stay hidden | Needs work | Part of the system instructions showed up in answers; the integration keys did not. |
- Real impact
- A single document received from outside could send internal content out of the company.
- How to fix
- Put a person’s approval on anything that leaves the company, such as sending email, and handle what the agent reads in documents separately from its instructions. Keep only what may be seen in the system instructions.
When the model, its instructions or its integrations change, the same lines are worth checking again.