AI Red Teaming Roadmap
AI red teaming is the work of testing AI systems, mostly applications built on large language models, the way an adversary would, but with permission and with the aim of making them safer. Security teams, AI engineers, trust and safety groups and independent consultants all do it. The job mixes security habits with an understanding of how models fail.
This roadmap starts with mindset, rules of engagement and just enough machine learning, then moves through threat modelling, the main attack classes, practical testing, defences and career development. Attacks are explained as concepts: what they are, why they work, how to test for them in an authorised engagement and how to reduce the risk. It deliberately contains no working attack recipes, because the skill that employers pay for is finding, measuring and fixing weaknesses.
When you finish you will be able to scope an engagement, build a threat model, write a repeatable evaluation harness, report findings clearly, and recommend layered defences. Use study plan to schedule the stages, and the AI Agents Roadmap and Prompt Engineering Roadmap roadmaps for the building side.
- Level
- Intermediate to advanced
- Time
- 10-14 weeks at about 8 hours a week
- Scope
- 63 topics in 8 stages
- Before you start
- Programming in Python Roadmap; Basic web and API knowledge, see API Design Roadmap; Familiarity with using chat models and their APIs; General security awareness such as authentication and access control
What you will be able to do
- Scope an AI red team engagement with clear rules and stop conditions
- Build a threat model for an LLM application with tools and retrieval
- Explain direct and indirect prompt injection, jailbreaks and data attacks, and why each works
- Write an automated evaluation harness that reports rates, not anecdotes
- Recommend layered defences, including privilege separation outside the model
- Report findings so engineers can reproduce and fix them
- Monitor a deployed system and feed incidents back into testing
The roadmap at a glance
- Stage 1Foundations and MindsetBeginner · 1 week · 7 topics
- Stage 2Machine Learning and LLM Basics for TestersBeginner · 1-2 weeks · 8 topics
- Stage 3Threat Modelling and Test PlanningIntermediate · 1-2 weeks · 8 topics
- Stage 4Prompt-Level AttacksIntermediate · 2 weeks · 9 topics
- Stage 5Data, Model and Infrastructure AttacksAdvanced · 2-3 weeks · 10 topics
- Stage 6Testing LLM Systems in PracticeIntermediate · 2 weeks · 9 topics
- Stage 7Defences and EvaluationAdvanced · 2 weeks · 8 topics
- Stage 8Community, Practice and CareerBeginner · Ongoing · 4 topics
Stage 1 Foundations and Mindset
Goal. Explain what AI red teaming is, who does it, what the rules of engagement are and how it fits with ordinary security thinking.
Introduction Core
AI red teaming is the practice of probing an AI system the way an adversary would, with permission, so that weaknesses are found and fixed before real attackers or real users find them. For language-model products this means testing what the model says, what it can be made to reveal, and what the software around it can be made to do.
The work sits between security testing and quality assurance. A tester needs enough machine learning knowledge to understand why models misbehave, and enough security habit to think in terms of assets, attackers and impact. This roadmap teaches both, in the order most people need them.
Practice. Write a one-paragraph description of a chatbot you use and list three things you would be worried about if you were responsible for it.
Why Red Team AI Systems? Core
Conventional testing checks that software does what the specification says. AI systems are different: their behaviour comes from training data and statistics, the input space is natural language, and the same input can give different answers. A specification cannot list every wrong behaviour, so the only way to learn how a system fails is to try to make it fail.
There is also a practical reason. Once a model is connected to documents, databases, email or code execution, a bad output stops being embarrassing and becomes an action. Red teaming measures that gap between intended and possible behaviour, gives product teams evidence, and supports the safety and compliance claims that customers and regulators now ask for.
Practice. List three failures of a model-based product that a normal unit test suite could not catch.
Role of Red Teams Core
A red team plays the attacker. A blue team defends and monitors. A purple arrangement has both sharing findings in near real time. In AI work the red team also covers harms that a classic penetration test ignores: biased or toxic output, leaking private data, giving dangerous advice, or being talked into breaking the product's own policy.
Typical outputs are a threat model, a set of reproducible findings ranked by impact, an evaluation suite that can be re-run after each fix, and advice on mitigations. A good red teamer is judged less by clever tricks than by whether the findings changed what the team built.
Practice. Draw a simple diagram of red, blue and product teams for an assistant you know and mark who owns each type of finding.
Ethical Considerations Core
Testing is only acceptable inside a written scope. Before starting, agree what systems and data are in bounds, what techniques are allowed, who will be told, how findings are stored, and who can stop the test. Never test a production system you do not have permission for, and never use real customer data when synthetic data will do.
Some content is harmful to produce or read. Plan for the wellbeing of testers who handle abusive material, limit how much harmful output is retained, and store evidence securely. When a finding shows a real danger, such as a method for serious harm, describe the class of weakness in the report rather than circulating working details. Honest reporting, minimal harm and respect for the scope are the habits that make the profession trustworthy.
Practice. Write a half-page rules-of-engagement template with scope, stop conditions, data handling and contacts.
Confidentiality, Integrity, Availability Core
The security triad gives a vocabulary for AI findings. Confidentiality is about information leaking: a system prompt, another user's conversation, training data or credentials. Integrity is about outputs and actions being trustworthy: an injected instruction changing what an agent does, or poisoned data skewing a classifier. Availability is about the service staying usable: expensive prompts that exhaust a budget, or loops that never finish.
When writing a finding, say which property is affected and for whom. A model that is rude is a quality issue. A model that emails a customer's data to a third party is a confidentiality failure with a clear owner and a clear fix.
Practice. Take five recent AI-incident headlines and classify each as confidentiality, integrity or availability.
AI Security Fundamentals Core
Several basic ideas carry through the rest of the roadmap. An asset is something worth protecting, such as data, model weights or the ability to take actions. An attack surface is every place an attacker can supply input or influence behaviour. A trust boundary is a line where data passes between parties with different privileges.
AI adds some twists. The model cannot reliably tell instructions from data in the same text stream. Behaviour is probabilistic, so a defence that works nine times in ten still fails at scale. And the supply chain is wide: pretrained weights, datasets, plugins, libraries and hosted APIs all become part of the system you are accountable for.
Practice. For an assistant that reads email, list its assets, inputs and trust boundaries in a table.
Risk Management Core
Not every weakness deserves the same effort. Risk is the combination of how likely something is and how bad it would be, in the context of the actual product. A jailbreak that makes a children's homework helper swear matters differently from one that makes a banking agent move money.
Rate findings with a simple, consistent scale, record the assumptions, and pair each with a treatment: fix, reduce, accept with sign-off, or transfer. Frameworks such as the NIST AI Risk Management Framework and ISO management standards give shared language, but the value comes from applying them to concrete systems and revisiting them after each change.
Practice. Score six hypothetical findings for likelihood and impact and justify the order in which you would fix them.
Stage 2 Machine Learning and LLM Basics for Testers
Goal. Describe how models are trained and used well enough to predict where they are fragile.
Neural Networks Core
A neural network is a large function built from layers of simple units whose weights are adjusted during training to reduce error on examples. After training, the weights hold what the model has learned, which is why they are valuable and why stealing them is an attack.
For a tester, three facts matter. Networks generalise from patterns, not rules, so inputs that look odd to a human can sit in regions the model handled badly. Small, carefully chosen changes to an input can change the output (the basis of adversarial examples). And the internals are hard to inspect, so testing is mostly behavioural.
Practice. Train a tiny classifier on a toy dataset and find an input it gets confidently wrong.
Supervised Learning Core
In supervised learning a model learns from labelled examples: emails marked spam or not, images marked with objects. Quality depends on the labels, so anyone who can alter labels or insert mislabelled examples can steer the model. This is the route for many data poisoning attacks.
Testers should ask where the labels came from, who can contribute them, how they are audited, and how the model is evaluated on data it has never seen. Test sets that leak into training make a model look better than it is.
Practice. Flip five percent of labels in a small dataset and measure how accuracy changes on a clean test set.
Unsupervised Learning Recommended
Unsupervised methods find structure without labels: clustering customers, detecting anomalies, learning embeddings of text. Security teams use them for anomaly detection, and retrieval systems use embeddings to find relevant documents.
The attack angles differ. Anomaly detectors can be evaded by moving slowly so abnormal behaviour looks normal, and embedding-based search can be manipulated by planting content that sits close to many queries. Because there is no ground truth label, it is also harder to say what a failure is, so tests need explicit success criteria.
Practice. Cluster a small text collection with embeddings and inspect which items land in surprising groups.
Reinforcement Learning Recommended
Reinforcement learning trains an agent by reward. Modern chat models use a version of it, learning from human or automated preference judgements to become more helpful and less harmful. Reward signals are imperfect proxies, so models can learn behaviour that scores well without being what the designers wanted, often called reward hacking.
For testers this explains two things. Safety behaviour is learned, not guaranteed, so it has gaps. And agents that act in an environment can find unintended shortcuts, which is why constrained tool permissions and monitoring matter more than trust in the training.
Practice. Describe a reward for a customer-service agent and list two ways it could be satisfied without helping the customer.
Generative Models Core
Generative models produce new content: text, images, audio, code. They learn a distribution from training data and sample from it. That brings specific risks: memorised training examples can be reproduced, outputs can be convincing but false, and content can be generated at a scale that suits abuse such as phishing or disinformation.
Different modalities share ideas but differ in detail. Image models can be steered by hidden text in pictures, and speech models can be fooled by audio that sounds ordinary to people. Know which modalities your target accepts, because each is an extra input channel.
Practice. List the input and output modalities of a product you use and one risk specific to each.
Large Language Models Core
A large language model predicts the next token given the text so far. Chat products wrap it with a system prompt, conversation history, tool descriptions, retrieved documents and the user's message, all flattened into one sequence. The model has no hard boundary between the developer's instructions and untrusted text inside that sequence, which is the root of prompt injection.
Other useful facts: the context window limits how much can be seen at once, sampling settings change how random answers are, and models are retrained or swapped over time, so a test result is only valid for a specific model version and configuration. Record both.
Practice. Send the same neutral question twenty times at two temperature settings and describe how the answers vary.
Prompt Engineering Core
Prompt engineering is the craft of writing instructions, examples and structure so that a model behaves as intended. Red teamers need it for two reasons. It shows what a well-built prompt can and cannot enforce, and it is how you build clear test cases and defensive prompts. The Prompt Engineering Roadmap roadmap covers it in depth.
The key security lesson is that a prompt is guidance, not an access control. Telling a model to keep something secret or never perform an action reduces the chance of it happening but gives no guarantee, so anything truly sensitive must be protected outside the model.
Practice. Write a system prompt for a support bot, then list which of its rules must also be enforced in code.
Model Vulnerabilities Core
Weaknesses of models themselves fall into recurring groups. Instruction confusion lets text from untrusted sources act like commands. Over-compliance means the model helps with requests it should refuse, and over-refusal the reverse. Hallucination produces plausible falsehoods. Memorisation can reveal training data. Sensitivity to wording makes behaviour inconsistent. Bias leads to unequal treatment.
Treat these as classes, not a checklist of exploits. For each class ask what the consequence would be in your specific product and what would reduce it, and let that guide which tests you write.
Practice. Pick one vulnerability class and write three different test ideas for it against a document-summarising assistant.
Stage 3 Threat Modelling and Test Planning
Goal. Scope an engagement, choose a testing approach and build a safe lab.
Threat Modeling Core
Threat modelling is a structured way to decide what to test. Draw the system: users, front end, model, tools, data stores, third-party services. Mark trust boundaries and where untrusted text enters. For each boundary ask what could go wrong using a method such as STRIDE for general security, and frameworks that catalogue AI threats such as MITRE ATLAS and the OWASP lists for LLM applications.
Finish with a short ranked list of scenarios, each with an attacker, a goal, a path and an impact. These become your test plan and, later, the structure of your report.
Practice. Threat-model a retrieval chatbot that can also open support tickets, and produce five ranked scenarios.
Black Box Testing Core
In black box testing you see only what an ordinary user or external attacker sees: inputs and outputs. It is the most realistic view of what an outsider can do and needs no cooperation from the developers, which makes it suitable for assessing third-party products where scope allows.
Its weakness is efficiency. You cannot see the system prompt, tool wiring or filters, so you must infer them from behaviour, and a lack of findings may only mean you have not looked in the right place. Report coverage honestly.
Practice. Write a black box test plan for a public assistant using only its visible behaviour and documented features.
Grey Box Testing Core
Grey box testing gives partial knowledge: architecture diagrams, the list of tools, example prompts, or a test account with some privileges. It is the most common arrangement in commercial engagements because it saves time while keeping an attacker's viewpoint.
The extra information lets you aim at the likely weak joints, such as a tool with write access or a retrieval source that anyone can edit, rather than guessing. Record exactly what you were told so results can be compared later.
Practice. Given a one-page architecture description, pick the three joints you would test first and say why.
White Box Testing Recommended
White box testing has full access: source code, prompts, configurations, logs and sometimes model weights. It allows code review of how model output is parsed, how tools are authorised and how secrets are handled, and, with weights, gradient-based analysis.
Most serious weaknesses in LLM applications are ordinary software problems around the model, such as missing authorisation checks or unsafe handling of output, and white box review finds them quickly. It also supports checking that defences really exist as designed, not just in a diagram.
Practice. Review a small agent codebase and list every place model output is used as input to something else.
Automated vs Manual Testing Core
Automated testing runs large sets of cases cheaply and repeatably, and is the right tool for regression checks, measuring rates across many variations, and testing after every model or prompt change. Manual testing brings creativity, context and judgement, and finds the new and multi-step problems that no script anticipated.
Use both. Humans explore and discover, then turn each discovery into an automated case. Automated results need human sampling, because judging whether an output is harmful or leaks something is itself error-prone.
Practice. Choose five test cases for a bot and decide which should be automated and which stay manual, with reasons.
Vulnerability Assessment Core
A vulnerability assessment is a broad, systematic sweep that lists weaknesses and rates them, as opposed to a red team operation that chains weaknesses toward a goal. For an AI product it covers the model's behaviour, prompts, tools, data sources, dependencies, infrastructure and access control.
Work through a consistent checklist, such as the OWASP Top 10 for LLM applications, record evidence for each item, and rate severity by impact in your context. The result is a prioritised backlog that engineering can act on.
Practice. Build a one-page assessment checklist from a public catalogue of LLM risks and apply it to a demo app.
Industry Standards Recommended
Several bodies publish material that gives red teams a shared language. The OWASP Top 10 for LLM applications lists common application risks. MITRE ATLAS maps adversary tactics against machine learning systems in the style of ATT&CK. NIST publishes an AI risk management framework and guidance on adversarial machine learning. ISO and IEC have management system standards for AI, and the EU AI Act sets legal duties for some systems.
Use them to structure scope, name findings consistently and show coverage to auditors. Do not treat them as complete, because attacks and categories evolve faster than documents.
Practice. Map three of your own findings to categories in two different frameworks and note where they fit poorly.
Lab Environments Core
Practise in places where mistakes hurt nobody. Run an open-weight model locally, or use your own accounts and keys with spending limits, and build small vulnerable-by-design applications that you control: a chatbot with a fake secret, a retrieval app over a folder you edit, an agent with harmless tools.
Isolate the lab from real credentials and networks, log every request, and keep it reproducible with containers or scripts so findings can be re-run. Deliberately vulnerable training apps and capture-the-flag style exercises are covered later in the community stage.
Practice. Create a container-based lab with a local model and a fake secret, and confirm nothing in it can reach the internet.
Stage 4 Prompt-Level Attacks
Goal. Explain each class of prompt-level attack, why it works, how to test for it with permission and how to reduce it.
Prompt Hacking Core
Prompt hacking is the umbrella term for manipulating a model through its input to make it behave in ways its operator did not intend. It includes prompt injection, jailbreaking and prompt leaking, which differ in who the victim is. Injection targets the application and its owner, jailbreaking targets the model's safety training, and leaking targets the confidentiality of the instructions.
Keep the categories separate in reports, because the fixes differ. Hardening a policy against jailbreaks does nothing for an injected instruction inside a web page, and vice versa. In an engagement, describe the class, the conditions under which it worked and the impact, not a reusable phrase.
Practice. Write a table with three columns, victim, goal and likely fix, for injection, jailbreaking and prompt leaking.
Prompt Injection Core
Prompt injection happens when text from an untrusted source is treated by the model as an instruction. It works because the model receives developer instructions and outside text as one stream of tokens and has no dependable way to tell them apart. Training makes the model prefer the system prompt, but this is a tendency, not a rule.
The impact depends on what the model can do. With no tools and no private data, injection mostly changes the reply. With access to files, email or APIs it can read secrets or take actions as the user. This is why the most useful defences limit the model's authority instead of trying to spot every malicious sentence.
To test responsibly, use harmless markers: ask whether an instruction hidden in data can change a benign, observable behaviour such as adding a harmless word to the output, and record the rate across variations.
Practice. Build a summariser with a fake secret and measure how often a harmless marker placed in the input document changes its output.
Direct Injection Core
In direct injection the attacker is the user, typing instructions into the chat that try to override the developer's configuration. The threat is mainly to the operator: bypassing business rules, extracting hidden instructions, using the service for something it was not paid for, or generating content that harms the brand.
Because the attacker already has the keyboard, the real question is what a user is entitled to. Any action or data the bot can reach must be authorised against that user's own rights in ordinary code, so that a successful manipulation gains nothing the user could not do anyway. Test by checking whether authorisation lives in the prompt or in the backend.
Practice. List every capability of a support bot and mark whether the user could legitimately do it without the bot.
Indirect Injection Core
Indirect injection places the instruction in content that the model reads later, rather than in the chat: a web page, an email, a shared document, a calendar invite, a code comment, an image, a tool result. The victim is a different person from the attacker, which makes it the more serious class for agents and retrieval systems.
It works because the system fetches text on the user's behalf and puts it next to trusted instructions. A single poisoned page can then try to steer the agent, for instance to disclose conversation content or call a tool. Defences combine treating all retrieved text as untrusted data, removing or flagging instruction-like content, restricting tools after untrusted content is read, and requiring user confirmation for consequential actions.
Practice. Plant a harmless marker instruction in a document in your own lab corpus and check whether a retrieval assistant obeys it.
Jailbreak Techniques Core
A jailbreak is an attempt to get a model to produce content its safety training is meant to prevent. Published research groups the methods into families: role-play and persona framing, hypothetical or fictional wrappers, splitting a request into harmless-looking parts, encoding or translating the request, gradual escalation over many turns, and automated search for inputs that tip the model over. New variants appear regularly and old ones get patched.
For a professional the useful skill is not collecting phrases but measuring. Define a policy, build a set of prohibited-request categories using sanitised stand-ins, run families of strategies at scale, and report the attack success rate with confidence intervals for a given model version. Mitigations include stronger safety training, input and output classifiers, and refusing to rely on the model as the only barrier.
Practice. Define a three-category policy for a lab model and write an evaluation plan that reports attack success rate per strategy family.
Safety Filter Bypasses Core
Many products add filters around the model: classifiers on input, classifiers on output, keyword lists, moderation APIs. Filters fail in predictable ways. They may check only one language, ignore text inside files or images, miss content split across messages, or inspect the prompt but not the tool output.
Testing here is about coverage of the filter, not about shocking content. Check every channel that reaches the model, compare input and output filters on equivalent content, test non-text modalities, and confirm that filtered results are logged. The usual remedy is layered defence: if one layer misses, another still has a chance, and the system fails safe when a filter errors.
Practice. List every input channel of a product and mark which ones pass through a filter.
Code Injection Core
When a model's output is used to build something executable, such as a database query, shell command, HTML page or template, ordinary injection bugs reappear with the model as the unwitting author. A model told to produce a query from a user's question can be steered into writing a destructive one, and a model that returns markup can be steered into script content.
The fix is the same as in all injection problems: never concatenate untrusted or model-generated text into an interpreter. Use parameterised queries, a narrow set of allowed operations, output encoding and least-privilege database accounts. Treat model output exactly like user input.
Practice. Review a text-to-SQL feature and list the controls that would still protect the database if the model were fully compromised.
Remote Code Execution Core
Remote code execution means an outsider makes the system run code of their choice. LLM applications meet it when they offer a code interpreter, run generated scripts, load model files that contain executable content, or use frameworks with unsafe evaluation functions. Several real vulnerabilities in LLM tooling have been of this kind.
Defence is isolation. Run generated code in a sandbox with no network, a read-only filesystem, a small resource budget, and a short lifetime, and never hold credentials in the same environment. Keep libraries patched, and prefer weight formats that cannot carry code. When testing, prove impact with a harmless action inside the sandbox, such as reading a canary file, and stop there.
Practice. Inspect how a code-running tool of an agent is sandboxed and list what an escape would reach.
Insecure Deserialization Recommended
Deserialization turns stored bytes back into objects. Some formats can execute code while loading. In machine learning this matters because model checkpoints, pipelines and datasets have often been shared in formats built on Python's pickle, so loading an untrusted file can run an attacker's code on your machine.
Prefer formats that store only tensors, such as safetensors, verify hashes and provenance of downloaded models, scan files before loading, and load untrusted artefacts only in isolated environments. Review any code that reads caches, queues or user uploads for the same pattern.
Practice. Audit a project for places it loads model or data files and note the format and source of each.
Stage 5 Data, Model and Infrastructure Attacks
Goal. Understand attacks on training data, model confidentiality, access and the systems that surround the model.
Adversarial Examples Core
An adversarial example is an input changed slightly, often imperceptibly to a person, so that a model gives a wrong answer. The classic cases are images with small pixel changes that flip a classification, but the idea applies to audio, malware detection and text.
They exist because models rely on features that do not match human perception. Testing uses libraries that generate perturbations under a size budget and measure how accuracy falls. White box attacks use gradients, while black box attacks query repeatedly or transfer examples from a substitute model. Defences include adversarial training, input preprocessing and ensembles, though none is complete, so evaluate against adaptive attackers, not just the attack the defence was designed for.
Practice. Use a small image classifier and a perturbation library to plot accuracy against perturbation size.
Data Poisoning Core
Data poisoning corrupts what a model learns by inserting or altering training, fine-tuning or retrieval data. Goals include lowering overall quality, or planting a backdoor, where the model behaves normally except when a specific trigger appears.
Poisoning is realistic wherever data comes from scraping, user feedback, shared datasets or community contributions. Controls include provenance records, restricting who can write to training and retrieval stores, outlier and duplicate analysis, holding out trusted evaluation sets, and testing for trigger behaviour. For retrieval systems the same logic applies to the document store: whoever can edit it can influence answers.
Practice. List every write path into the data your model or retrieval index uses and mark who can use it.
Model Inversion Recommended
Model inversion and the related membership inference attacks try to recover information about the training data from a model's outputs: whether a particular record was used, or an approximation of a sensitive attribute or example. Models that overfit or memorise are more exposed, and so are models trained on personal or confidential text.
Testing measures how much a model reveals about its training set, for example by comparing confidence on known training records and held-out records. Mitigations include removing sensitive data before training, deduplication, regularisation, limiting output detail such as confidence scores, and differential privacy during training where the utility cost is acceptable.
Practice. Compare model confidence on training and unseen records for a small model and relate the gap to privacy risk.
Model Weight Stealing Recommended
Weights are the product of expensive training, so they are a target both as files and through behaviour. Direct theft happens through compromised storage, insider access or leaked backups. Extraction attacks query a model many times and train a copy that imitates it, and research shows some details of architecture can be inferred from outputs.
Protect weights as high-value secrets: strict access control and logging, encryption at rest, separate environments, and signed artefacts. Against extraction, apply rate limits and quotas per account, monitor for systematic query patterns, minimise the detail returned, and use terms of service as a legal layer. Be clear about which of these you can test within scope.
Practice. Write a threat model for a hosted model covering both file theft and query-based copying, with one control for each.
Unauthorized Access Core
Many incidents around AI products are not clever model attacks but ordinary access failures: exposed admin consoles, open vector databases, API keys in public repositories, shared accounts, or an assistant that retrieves documents the signed-in user should not see.
Check the entry points first. Is every endpoint authenticated? Can one tenant reach another's data through retrieval or cached prompts? Are logs and conversation histories protected? Does the model's context ever include data from a different user? Fixing these gives larger security gains than most prompt-level work.
Practice. Test a lab multi-user chat app for cross-user data exposure through history, retrieval and caching.
Authentication Core
Authentication proves who is calling. For AI services it covers the people who use the app and the machines that call the model, vector store and tools. Use established standards such as OAuth 2.0 and OpenID Connect for users, short-lived credentials for services, and per-user or per-tenant keys instead of one shared key.
Common faults include long-lived secrets in client code, tokens that are never rotated, tools that call back-end systems with an all-powerful service account, and weak controls on the admin interface for editing prompts. Store secrets in a manager, never in prompts or repositories, and make rotation routine.
Practice. Scan a sample repository for secrets and design how each would move to a secret manager with rotation.
Authorization Core
Authorization decides what an authenticated caller may do. In agent systems the key principle is that the agent acts with the rights of the user it serves, not with its own broad rights. When a model decides to call a tool, the tool must check the user's permissions on the server side, as it would for any API request.
Avoid the confused deputy problem, where a powerful component is tricked into using its authority for someone with less. Practical controls: scoped tokens per task, allow-lists of tools and arguments, read-only by default, human approval for destructive or irreversible actions, and authorisation checks in retrieval so that documents are filtered before they reach the model.
Practice. Review an agent tool list and for each tool write the permission check that must exist outside the model.
API Protection Core
Model endpoints are APIs and need the usual protections plus a few specific to AI. Apply authentication, rate limits, request size limits and quotas to defend availability and cost, since a single prompt can consume large compute. Validate structured inputs, set maximum output and tool-call counts, and set timeouts.
Monitor for scraping and extraction patterns, such as very high volume with systematic variation. Return errors that do not reveal internal prompts or stack traces. Keep an inventory of every endpoint, including internal and forgotten ones, because test and staging deployments of model services are often exposed without controls.
Practice. Add per-key rate limits and a maximum token budget to a small model-serving API and test that they hold.
Infrastructure Security Core
AI systems run on infrastructure that needs ordinary hardening: patched hosts, network segmentation, locked-down containers, secure CI/CD, and protected storage for datasets, embeddings and weights. GPU clusters and notebook servers are often set up quickly and left exposed.
Pay attention to the supply chain: model hubs, third-party plugins, Python packages and container images can all carry malicious or vulnerable code. Pin versions, verify checksums and signatures, scan dependencies, and separate training environments that touch sensitive data from those that fetch public code. The Docker Roadmap and Kubernetes Roadmap roadmaps cover the platform side.
Practice. Produce a bill of materials for a small model-serving project listing every model, package and image it depends on.
Agentic AI Security Core
Agents combine a model with tools and a loop, so a single manipulated step can lead to a chain of real actions. The risks multiply: injection through any tool output, excessive permissions, actions without review, memory that stores poisoned content for later, and several agents trusting each other's messages.
Design principles: grant the least authority needed for each task, separate the component that reads untrusted content from the one that holds privileges, require approval for high-impact actions, log every tool call with its arguments, and limit loops by steps, time and cost. Evaluate agents by running multi-step scenarios and checking what they did, not just what they said. The AI Agents Roadmap roadmap covers how agents are built.
Practice. Draw an agent that reads email and can send messages, then redesign it so reading and sending are separated.
Stage 6 Testing LLM Systems in Practice
Goal. Build repeatable evaluations, run them against a lab target and report results clearly.
LLM Security Testing Core
LLM security testing combines application testing with behavioural evaluation. A good plan has layers: the model alone, the model with its system prompt, the full application with tools and data, and the deployed service with its filters and monitoring. A weakness may appear at one layer and be hidden by another, so note which layer each result applies to.
For each test record the model version, configuration, date, input, output, and a verdict with reasons. Run each case several times because outputs vary, and report rates, not single successes. Keep a library of cases tagged by threat so coverage is visible.
Practice. Write ten test cases tagged by threat class for a lab assistant and run each five times, recording the verdicts.
Custom Testing Scripts Core
Off-the-shelf tools cover common cases, but real engagements need scripts tailored to the target: sending cases through its actual API, handling authentication, parsing structured replies and scoring them. Keep scripts simple. A loop over test cases, a call to the target, a scorer and a results file is enough to start.
The harness below scores whether a harmless marker planted in a document leaked into the reply, which is how you measure indirect injection susceptibility in a lab without any harmful content.
import csv, statistics
def run_suite(call_target, cases, runs=5):
rows = []
for case in cases:
hits = 0
for _ in range(runs):
reply = call_target(case["document"], case["question"])
if case["marker"] in reply:
hits += 1
rows.append((case["id"], hits / runs))
return rows
def report(rows, path):
with open(path, "w", newline="") as f:
csv.writer(f).writerows([("case", "rate")] + rows)
print("mean rate", round(statistics.mean(r for _, r in rows), 3))Because the marker is harmless, the same suite can run in continuous integration after every prompt change.
Practice. Adapt the harness to your own lab assistant and run it before and after adding a defensive instruction to the prompt.
Benchmark Datasets Recommended
Public benchmarks give a shared yardstick. There are collections of harmful-request categories for measuring refusal behaviour, sets for truthfulness and bias, and datasets of injection scenarios for agents. They let you compare models and track progress.
Use them with care. Benchmarks leak into training data, so scores may overstate real safety, and they rarely match your product's policy and users. Treat them as a baseline, then add your own cases drawn from your threat model and incident history. Hold back a private set that is never published.
Practice. Run a small public safety benchmark on a lab model, then add ten cases from your own threat model and compare results.
Testing Platforms Recommended
Several open-source and commercial tools help organise this work. Open-source scanners can run many probe families against a model endpoint and report failures, evaluation frameworks let you define test cases and graders, and Microsoft's PyRIT is a toolkit for orchestrating automated red teaming. Cloud providers and model vendors also offer built-in safety evaluations.
Choose by what you need: probe libraries for breadth, evaluation frameworks for regression and scoring, orchestration for multi-turn scenarios. Read the licence and data-handling terms before pointing a hosted tool at confidential systems, and remember that tools find known patterns, not novel ones.
Practice. Install one open-source probe scanner in your lab and compare its findings with your manual results.
Red Team Simulations Core
A simulation plays out a whole attack story with a goal, such as getting an assistant to disclose another customer's data, rather than testing single weaknesses. It chains steps: reconnaissance of the product's features, finding an input channel, steering the model, abusing a tool, and reaching the objective, all inside agreed rules.
Plan with a clear objective, a time box and success criteria, brief a trusted contact on the defender side, and record every step so the blue team can learn from it. A simulation answers whether defences work together under pressure, which isolated tests cannot.
Practice. Write a one-page scenario with objective, rules, steps and success criteria for a lab agent and run it.
Advanced Techniques Recommended
Beyond manual probing, mature teams use automation that searches for failures. An attacker model can generate and refine test inputs against a target, with a judge model scoring results. Multi-turn strategies test whether safety erodes over long conversations. Fuzzing varies inputs systematically, and gradient-based search applies to open-weight models.
For defenders these methods matter because they produce large, varied test sets and expose how brittle a defence is. Use them to stress a fix before shipping, and calibrate any judge model against human labels, since automated judges have their own error rate and bias.
Practice. Build a loop in which one lab model rewrites a harmless test case ten ways, and score how the target responds to each.
Continuous Testing Core
Models, prompts, tools and data change often, and any change can reopen a weakness. Put your evaluation suite in the delivery pipeline so it runs on every prompt edit, model upgrade, new tool and data refresh. Set thresholds, for example that the injection marker rate must not rise, and block releases that regress.
Keep fast smoke tests for every commit and larger nightly runs. Add each real incident or past finding to the suite as a permanent case. Review the suite regularly so it does not go stale.
Practice. Wire your lab harness into a CI job that fails when a measured rate rises above an agreed threshold.
Reporting Tools Core
A finding that cannot be reproduced or understood will not be fixed. Each entry should state the affected system and layer, the threat class and framework reference, steps or a script to reproduce, observed and expected behaviour, evidence with sensitive content redacted, impact in plain terms, a severity rating, and a recommended fix.
Use whatever the engineering team already tracks work in, and keep a shared template. Give an executive summary of key risks, and a technical appendix with raw results. Spreadsheets, notebooks, issue trackers and dedicated reporting platforms all work if they preserve evidence and let you track fixes to closure.
Practice. Write a complete finding for one lab weakness using a template with all the fields listed above.
Responsible Disclosure Core
When you find a weakness outside an engagement, for example in a public product, follow coordinated disclosure. Look for the vendor's security contact or programme, report privately with enough detail to reproduce, avoid accessing data beyond what is needed to prove the point, and allow reasonable time to fix before sharing details. Many model providers run reporting channels and bounty programmes that cover model safety issues.
Inside an engagement, the contract defines reporting and confidentiality. Either way, share lessons in general terms, protect affected people, and do not publish working abuse material.
Practice. Write a disclosure email for a hypothetical lab finding, including what you will and will not share publicly.
Stage 7 Defences and Evaluation
Goal. Choose and measure layered defences, and monitor systems after release.
Countermeasures Core
No single control stops these attacks, so use layers. A practical stack is: clear system instructions, input checks, privilege separation so the model cannot do more than the user can, output validation before anything is executed or displayed, human approval for high-impact steps, logging, and rate limits.
The strongest layers are those outside the model, because they do not depend on the model behaving. Ask of each risk: if the model were fully fooled, what would the surrounding code still prevent? Where the answer is nothing, add a control there.
Practice. For three risks from your threat model, write the outside-the-model control that limits damage if the model is fooled.
Input and Output Filtering Core
Filters reduce noise and catch obvious problems. On input they can limit length, strip hidden characters, flag instruction-like text in retrieved content and run a classifier. On output they can redact secrets, block unsafe categories and check that structured replies match a schema before being used.
The sketch below withholds replies that contain a canary string, placed deliberately in the system prompt so that any leak of it is detectable, or that look like credentials.
import re
CANARY = "canary-7f3a" # random per deployment
SECRET_LIKE = re.compile(r"(sk|key|token)[-_][A-Za-z0-9]{16,}")
def check_output(text):
if CANARY in text or SECRET_LIKE.search(text):
return False, "possible leak"
return True, "ok"
def respond(model_reply):
ok, reason = check_output(model_reply)
return model_reply if ok else "Sorry, I cannot share that."Filters will have false positives and misses, so measure both and never rely on them alone.
Practice. Add a canary and an output check to a lab bot and measure false positives on a set of normal replies.
Privilege Separation for Tools Core
Give the model a narrow, explicit set of tools and check every call in code. An allow-list with argument validation and per-user permissions means a manipulated model can only request things the policy permits.
ALLOWED = {
"search_docs": {"read_only": True},
"create_ticket": {"read_only": False, "needs_approval": False},
"send_email": {"read_only": False, "needs_approval": True},
}
def dispatch(user, call, approve):
spec = ALLOWED.get(call["name"])
if spec is None:
raise PermissionError("tool not allowed")
if not user.can(call["name"], call["args"]):
raise PermissionError("user lacks rights")
if spec.get("needs_approval") and not approve(call):
raise PermissionError("not approved")
return TOOLS[call["name"]](**call["args"])After a tool has returned untrusted content, consider narrowing the tool set for the rest of the turn.
Practice. Implement the dispatcher for a lab agent and write tests proving an unlisted tool and an unauthorised user are both refused.
Adversarial Training Recommended
Adversarial training adds hard examples, such as perturbed inputs or red-team transcripts with the desired safe response, to training or fine-tuning so the model learns to handle them. It improves resistance to the attacks it has seen and is used both for classic vision models and for safety tuning of language models.
Its limits are important. It costs accuracy and compute, it generalises only partly to new attack styles, and it can make a model over-refuse. Treat it as one layer, hold out attacks from evaluation so you measure generalisation, and refresh the data as new weaknesses appear.
Practice. Fine-tune or retrain a small classifier with adversarial examples and compare resistance on a held-out attack type.
Resilient Model Design Recommended
Resilient design starts before training. Choose the smallest capability that meets the need, keep sensitive data out of training sets, filter and deduplicate data, track provenance, and separate models by trust level rather than putting every task behind one all-powerful assistant.
At the system level, prefer designs where a compromised model has little to gain: structured outputs instead of free text for downstream code, read-only access by default, and isolation between users. Build in fail-safe behaviour, such as refusing or escalating to a human when confidence or checks fail.
Practice. Redesign a one-assistant-does-everything product as two components with different privileges and describe what each can reach.
Evaluating Defences Core
A defence is only as good as its measured effect. Define two numbers: how much attack success falls, and how much normal use is harmed. Evaluate against attacks that were not used to build the defence, and against an adaptive tester who knows the defence exists.
A minimal scoring routine compares attack success and benign refusal across configurations.
def evaluate(call_target, attacks, benign, judge):
asr = sum(judge(call_target(a)) for a in attacks) / len(attacks)
over_refusal = sum(is_refusal(call_target(b)) for b in benign) / len(benign)
return {"attack_success": asr, "over_refusal": over_refusal}Report both numbers with sample sizes. A defence that cuts attacks by half but blocks a fifth of legitimate requests may not be acceptable.
Practice. Evaluate two versions of a lab prompt using the function above and write a recommendation based on both numbers.
Monitoring Solutions Core
After release, you need visibility. Log prompts, retrieved sources, tool calls and outputs with user and session identifiers, while protecting privacy: redact personal data and limit retention and access. Add detectors for canary leaks, unusual tool use, long refusal streaks, spikes in volume and attempted extraction patterns.
Send alerts to the same channel as other security events, with runbooks for what to do: disable a tool, tighten limits, block an account, or roll back a prompt. Monitoring is also a source of new test cases.
Practice. Define five alert rules for a lab agent and the action each one triggers.
Continuous Monitoring Core
Monitoring is not a one-off project. Track trends over time: refusal rates, blocked outputs, user reports, cost per user, and results of scheduled evaluation runs against the live system. A sudden change often signals a model update by the provider, a new attack campaign or a regression in your own change.
Watch for emerging threats such as new input channels and protocol layers. Review incidents in blameless post-mortems, feed what you learn into the evaluation suite and threat model, and schedule regular red team exercises. Keep an inventory of models and versions in use, because providers deprecate and update models on their own timetable.
Practice. Create a dashboard sketch with the six metrics you would watch weekly, and say what each change would mean.
Stage 8 Community, Practice and Career
Goal. Find places to practise safely, keep learning and show your skills.
CTF Challenges Recommended
Capture-the-flag challenges give legal, scored practice. Several communities host LLM-focused challenges where a bot holds a hidden flag and the task is to understand how its defences work. They teach quickly because feedback is immediate and nobody is harmed.
Treat them as study, not as the whole skill. After each challenge write up what class of weakness it showed, what defence would have stopped it, and how you would test for it in a real engagement. Writing is what turns a game into a transferable skill.
Practice. Complete three beginner challenges and write a short note on the weakness class and the defence for each.
Conferences Optional
Security conferences such as DEF CON, Black Hat and the machine learning venues increasingly feature AI security tracks, and many publish talks as recordings. Academic venues in security and machine learning release papers on attacks and defences, often with code.
Forums and mailing lists run by security groups and tool projects are where practitioners trade tools and advice, so join with a purpose and never share client details. Attend or watch for new techniques, tool releases and case studies from practitioners. Write notes for yourself on each talk: the claim, the evidence, and whether it applies to systems you test.
Practice. Watch two recorded AI security talks and write a note on the evidence behind each main claim.
Research Groups Optional
University labs, company safety teams and independent research collectives publish most of the new work on model attacks and defences. Following their output shows where the field is heading, and many groups release datasets and tools.
Research opportunities exist for people without a doctorate, such as reproducing results and building open evaluation sets. Specialized courses with hands-on labs also help. Read papers with a tester's eyes: what threat model does it assume, how strong is the attacker, and would the result hold against a product with filters and monitoring? Keep a reading log with a one-paragraph summary for each paper.
Practice. Read one recent attack paper and one defence paper and summarise how their assumptions differ.
Industry Credentials Optional
Credentials can help with hiring filters, though a portfolio of write-ups, tools and talks is stronger evidence. General security certifications, penetration-testing certifications and a growing number of AI-security-specific programmes exist. Check what the syllabus actually covers and whether it is practical.
Be wary of overselling any credential. In interviews, explain what you can do, show a lab report, and be honest about what you have not yet tried.
Practice. Compare the syllabi of two credentials against this roadmap and list what each leaves out.
Projects to build
Build these in order. Each one should end up in a public repository with a short README that explains your decisions.
- Vulnerable-by-design chatbot labBeginner
Build a small assistant with a fake secret, a system prompt and a harmless tool, running a local model in a container. Document its trust boundaries and where each could be tested.
Lab setup · Threat modelling · Logging
- Injection susceptibility harnessIntermediate
Write a script that plants harmless markers in documents and measures how often a retrieval assistant obeys them across prompts and temperatures. Output rates as a table and chart.
Indirect injection · Evaluation · Python
- Defence comparison reportIntermediate
Add three defences in turn: a prompt change, an output filter and a tool allow-list. Measure attack success and over-refusal for each and recommend a combination with evidence.
Countermeasures · Metrics · Reporting
- Agent permission auditAdvanced
Take an open-source agent, list every tool and credential, build a privilege model, and add server-side authorisation and approval steps. Show with tests that a fooled model cannot exceed the user.
Authorization · Privilege separation · Testing
- Continuous red team pipelineAdvanced
Put your suite into CI so that every prompt or model change runs the cases and blocks regressions. Add a monitoring dashboard and a process for turning incidents into new cases.
Continuous testing · Monitoring · CI
Interview questions
Try to answer each question out loud before you open the answer.
What is the difference between prompt injection and a jailbreak?Junior
Prompt injection abuses the fact that a model cannot reliably separate instructions from data, so untrusted text steers what the application does. The victim is the application owner or the user. A jailbreak targets the model's safety training to produce content it should refuse, and the victim is the policy. I keep them separate because fixes differ: privilege limits and data handling for injection, safety tuning and classifiers for jailbreaks.
Why is indirect injection considered more dangerous than direct injection?Mid
In direct injection the attacker is the user, so the damage is bounded by what that user may do. In indirect injection the attacker plants text in a page, document or email that another person's assistant reads, so the victim never sees the attack and the assistant acts with the victim's rights. The danger grows with tool access and private data, which is why I focus on limiting authority after untrusted content is read.
How would you scope and start an AI red team engagement?Mid
I agree the written scope first: systems, data, allowed techniques, test windows, contacts, stop conditions and how evidence is stored. Then I get an architecture description and build a threat model, listing assets, input channels, trust boundaries and tools. From that I rank scenarios by impact, choose black, grey or white box access, and plan manual exploration plus automated suites. I finish with a report format agreed in advance.
Can a system prompt keep a secret?Junior
No. A system prompt is guidance that the model usually follows, not a protection. Assume a determined user can eventually extract it or infer it from behaviour. Anything sensitive, such as keys, other users' data or business rules that must hold, belongs in code outside the model. I treat the prompt as public and add canary strings so that leaks are at least detected.
How do you measure whether a jailbreak defence works?Senior
I define the policy and build a set of prohibited-request categories using sanitised stand-ins, with strategy families that were not used to build the defence. I run each case several times, judge outcomes with a calibrated grader checked against human labels, and report attack success rate with sample sizes and intervals. I also measure over-refusal on benign requests, because a defence that blocks normal use is a cost.
An agent can read email and send messages. How do you secure it?Senior
I separate reading from sending, treat email content as untrusted data, and require user approval for sends, especially to new recipients. Tools get least privilege and are checked server side against the user's rights. I log every tool call, rate-limit actions, and narrow the available tools after untrusted content is read. Then I test with harmless markers in test emails to confirm instructions in messages cannot trigger sends.
What is data poisoning and how would you test for it?Mid
Data poisoning corrupts training, fine-tuning or retrieval data so the model degrades or behaves badly under a trigger. I would map every write path into the data, check provenance and access control, and examine data for outliers and duplicates. For backdoors I test whether specific candidate triggers change behaviour compared with a clean baseline. For retrieval systems I check who can edit the index.
How can a model leak training data, and what reduces the risk?Mid
Models can memorise rare or repeated examples and reproduce them, and membership inference can reveal whether a record was in the training set. Overfitting increases the risk. I reduce it by removing sensitive data before training, deduplicating, limiting output detail, applying differential privacy when affordable, and testing by comparing behaviour on training and held-out records.
How do you protect model weights?Senior
I treat them as crown-jewel secrets: restricted access with logging, encryption at rest, separate environments, signed artefacts and no copies in ad hoc storage. Against extraction through the API I use per-account rate limits and quotas, minimal output detail and monitoring for systematic query patterns. I also include insider access in the threat model.
Why is loading a downloaded model file risky?Mid
Some formats, notably those built on Python pickle, can execute code on load, so an untrusted checkpoint can compromise the machine. I prefer tensor-only formats like safetensors, verify hashes and provenance, scan files, and load unknown artefacts only in an isolated environment with no credentials. The same applies to datasets and caches.
What would you include in a finding report?Junior
Affected system and layer, threat class and framework reference, reproducible steps or a script, observed versus expected behaviour, redacted evidence, impact explained in business terms, severity with reasons, and a recommended fix with an owner. I add the model version and configuration, the success rate over repeated runs, and a retest result once fixed.
Why run tests several times rather than once?Junior
Model outputs are probabilistic, so a case that fails once may pass nine times and vice versa. A single success proves a weakness exists, but a rate tells engineers how likely it is in use. I run each case multiple times, report rates and sample sizes, and re-run after any model, prompt or tool change.
Where should authorisation checks live in an LLM application?Mid
In ordinary server-side code, in the tools and data access layers, evaluated against the real user's identity and rights. Never in the prompt. The model should be able to request an action, and the backend decides whether it is allowed. Retrieval must also filter documents by permission before they reach the model.
How do you decide the severity of an AI finding?Mid
I look at impact and likelihood in the product's context. Impact covers which data, actions and people are affected and whether it is reversible. Likelihood covers how easy it is, whether it needs special access and how often it succeeds. A jailbreak on a closed internal tool with no private data ranks lower than a reliable injection that triggers payments.
What are the limits of automated judges in red teaming?Senior
A judge model has its own error rate and biases. It may miss subtle leaks, flag harmless text, or be steered by the output it grades. I calibrate judges against human labels on a sample, keep the rubric specific, use multiple judges or deterministic checks where possible, and keep humans reviewing borderline and high-severity results.
How do you keep red teaming ethical when handling harmful content?Mid
I work inside written scope, use sanitised stand-ins and synthetic data where possible, limit retention of harmful outputs, secure evidence, and give testers support and rotation. Reports describe the weakness class and fix, not reusable abuse material. Anything affecting real people outside scope goes through coordinated disclosure.
A vendor updates the model behind your product. What do you do?Senior
I treat it as a change that can reopen old weaknesses. I re-run the full regression suite and compare rates with the previous version, check new capabilities for new input channels, review logs for shifts in refusal or tool use, and decide whether to pin a version, adjust prompts, or roll back. Findings go into the model inventory.
Common mistakes
- Treating the prompt as a security boundary; put real controls in code outside the model.
- Collecting attack phrases instead of building measurable test suites; report rates across repeated runs.
- Testing only the bare model; test the whole application with its tools, data and filters.
- Ignoring ordinary security bugs around the model such as exposed endpoints and weak authorisation; check these first.
- Starting without written scope and stop conditions; agree them before any test.
- Relying on one filter or one safety layer; layer controls and assume each will sometimes fail.
- Measuring attack success but not over-refusal; track both so defences stay usable.
- Forgetting to record model version and settings; results are only valid for that configuration.
- Shipping a fix without adding the case to the regression suite; make every finding a permanent test.
Ready-for-interview checklist
- I can explain red teaming and how it differs from a penetration test and from quality testing.
- I can write rules of engagement with scope, data handling and stop conditions.
- I can build a threat model for an LLM app with tools and retrieval.
- I can explain direct injection, indirect injection and jailbreaks and their different fixes.
- I can describe data poisoning, model inversion and weight stealing and one control for each.
- I can explain why authorisation must live outside the model.
- I can write an evaluation harness that reports rates over repeated runs.
- I can design privilege separation and approval steps for an agent.
- I can write a clear finding with evidence, impact and a recommended fix.
- I can evaluate a defence using both attack success and over-refusal.
- I can design monitoring and incident feedback for a deployed AI system.
- I can discuss the main industry frameworks and map findings to them.
Related roadmaps
- AI Agents RoadmapTools, memory, planning, multi-agent systems and evaluation.
- Prompt Engineering RoadmapPrompt patterns, structured output, evaluation and safety.
- AI-Assisted Coding RoadmapWorking with coding assistants and agents safely and productively.
- API Design RoadmapREST, gRPC, GraphQL, versioning, auth, pagination and rate limits.
- Python RoadmapSyntax to packaging: data model, typing, async, testing.
FAQ
Do I need a machine learning degree to do AI red teaming?
No. Most application-level work needs solid security skills, programming ability and a working understanding of how language models behave. Deeper model attacks such as adversarial examples and extraction benefit from more mathematics, which you can add later.
Is it legal to practise?
Practise on systems you own or that explicitly invite testing, such as your own lab, open-weight models you run yourself and challenges built for the purpose. Testing other organisations' products without permission can break laws and terms of service, so get written authorisation.
How long does it take to become job-ready?
With programming and security basics, around three months of steady study and lab work covers this roadmap. A portfolio of a lab, an evaluation harness and a few sample reports matters more than the time spent.
Which should I learn first, attacks or defences?
Learn the threat model first, then each attack class alongside its defence. Understanding why an attack works makes the mitigation obvious, and testing a defence needs knowledge of the attack it addresses.
Will automated tools replace human red teamers?
Tools already cover broad, repetitive probing and regression testing. Humans still find novel, multi-step and context-dependent weaknesses, judge impact and talk to engineers. The common pattern is people designing the tests and tools running them at scale.