SSQ.AISSQ.AI
English
help!

AI System Security Assessment & Penetration Testing

Testing AI agents, LLM services and AI infrastructure as targets of attack

We put AI agents, LLM services and AI infrastructure to the test from the attacker's side first, and confirm the real risks, whether before launch or in service.

From the tools and permissions an agent uses to the models, data and infrastructure behind them, we look for weaknesses across the whole AI system.

Under attack, does our AI system stay inside the lines we have set?

Whether any path crosses the line, what it would mean if one does, and records that let you check it again.

Expert reviewed

INPUT Outside input AGENT AI agent TOOLS Tools & access OUTSIDE Sent outside Hidden prompt Sends without approval Line: nothing leaves without a person's approval Weakness found Crossed the line Line held
We confirm whether an agent that follows a hidden instruction can cross the line you set through one of its tools, and check that the line holds on the other paths.

When it fits

  • You are bringing in an AI agent or launching an LLM service

    You have added an AI chatbot to your service, or are bringing agents into your work, and want to know that they keep to your lines when someone attacks them.

  • Agents are being given more tools and access

    The more an agent can use, from email and calendars to internal systems and outside tools, the more you need evidence of how far its permissions reach and where a person still has to approve.

  • You are connecting internal documents and data to AI

    If retrieval-augmented answers now draw on your internal documents, we check that their contents and sensitive information never reach staff who should not see them.

  • Several teams share training and inference infrastructure

    Several teams share GPU clusters, HPC (high-performance computing) environments and storage, and you want to know that each team's training data and model files are properly kept apart.

The SAFE SQUARE approach

Lines first

We first write down, in plain sentences, the lines the AI must not cross, and judge everything by whether they hold under attack.

Knowing how agents are built

Drawing on SAFE SQUARE's own research into how agents plan, use tools and handle permissions and approvals, we map which tools an agent uses with which rights before we test it.

Beyond the answers

From what the model says, we follow on into the tools the agent uses, the systems it connects to, and the isolation of the training and inference infrastructure.

What we look at

Our focus is AI agents, LLM services and AI infrastructure, along with the places where they connect to one another and to your other systems.

AGENT

AI agents

The target
AI agents that work through email, calendars, internal systems and outside tools, together with the environment they run in.
What we can test
  • The reach of tool and API permissions
  • Where actions need a person's approval
  • How outside input (documents, web pages, email) affects behaviour
  • Data leaving through the agent
  • The execution environment and its isolation
  • Action records and traceability
  • Authentication to connected systems
An example line
‘The AI agent sends nothing outside without a person's approval.’
LLM

LLM services

The target
Services built on language models, such as chatbots, retrieval-augmented answers and summaries, with their models, data and integrations.
What we can test
  • Prompt injection, direct and indirect
  • Exposure of sensitive data and system instructions
  • Permissions on the documents it retrieves
  • Output handling and its effect on connected systems
  • Whether safety policies work as intended
  • Misuse of usage and cost
  • The model and data supply chain
An example line
‘Staff get answers only from documents they are allowed to see.’
GPU

AI infrastructure (GPU and HPC)

The target
GPU clusters and HPC (high-performance computing) environments used for training and inference, with their schedulers and shared storage.
What we can test
  • Job schedulers and management interfaces
  • Isolation between users and teams
  • Permissions on shared storage and training data
  • Access to model files
  • The software stack, such as drivers and runtimes
  • Networking between nodes
  • Inference service APIs
An example line
‘One team's training data and model files cannot be read or changed by another.’

The areas and lines are examples.

ScopeHere the AI system is what is being attacked. Evaluating and improving how well AI agents carry out security work belongs to Security Capability Development for People & AI. For researching and building AI models and agents for your security work, see Technical collaboration and joint research.

How it runs

  1. Agree targets and lines

    We agree with you the AI systems and integrations to test and the lines they must keep.

  2. Map the system

    We set out with you the tools and permissions the agent uses, how models and data flow, and the connected systems and execution environment.

  3. Test and validate

    Our experts make the attempts a real attacker would, and check by hand whether the lines you have set hold under attack.

  4. Explain and re-check

    We explain the findings, real impact, evidence and fixes to decision-makers and the technical team and, if you wish, check the fixes again the same way.

Keeping it safeOn a live service we set test accounts, scope and stop conditions first, so that real users and data are not affected. The principles every engagement follows, safe conduct and stop conditions, facts kept apart from assumptions, and how material is kept and destroyed, are set out under How we work.

What you receive

Lines held and lines crossed

For each agreed line, whether it held and, where it did not, through which input, permission and path it was crossed.

Findings and real impact

What each finding would mean for the business, such as data exposure, actions beyond permission or runaway cost.

Reproducible evidence

The inputs, conditions and records needed to confirm each result again.

What to fix and check again

What to fix first, and what to check again after a model or integration changes.

A sample resultFictional example
Target

A workplace assistant agent that answers from internal documents and can send email

Connected to: internal document search, sending email, calendar lookup

LineResultEvidence
Answers use only documents the user may seeHeldDocuments are filtered by the user's permissions at the search step, so out-of-bounds documents were never used in answers.
No email leaves the company without approvalCrossedFollowing an instruction hidden in a document received from outside, the agent had a path to email an outside address without approval.
System instructions and integration keys stay hiddenNeeds workPart of the system instructions showed up in answers; the integration keys did not.
Real impact
A single document received from outside could send internal content out of the company.
How to fix
Put a person’s approval on anything that leaves the company, such as sending email, and handle what the agent reads in documents separately from its instructions. Keep only what may be seen in the system instructions.

When the model, its instructions or its integrations change, the same lines are worth checking again.

  1. Tell us what worries you, or what you want checked. A few lines are enough.

  2. We reply with a proposed scope, schedule and cost, and settle them with you.

  3. Once agreed, the work begins. Results and next steps come to you as it runs.