On July 30, we reported three incidents in which Claude models gained unauthorized access to real computer systems. The models—intentionally running without cyber safeguards for evaluation purposes—accessed the internet due to a misconfiguration inside a third-party evaluation environment. Separately, on August 4, the UK AI Security Institute reported an incident from its own cybersecurity testing, in which Claude Mythos 5 took a series of unauthorized actions on the live internet. In that case, the model, again intentionally running without cyber safeguards for evaluation purposes, had been deliberately given internet access.
We are conducting an in-depth analysis of both incidents. We are also planning to work with METR for an independent review. We want to ensure both studies are thorough, and will share more in the coming weeks.
In the meantime, we’re sharing some of the changes we’ve made over the past month. We believe the incidents reflect a failure of operational security, as well as two alignment issues: motivated reasoning, and willingness to take harmful actions in pursuit of a narrow task (both of which we have described in previous system cards). On security, we describe the improvements we’ve made to our containment and monitoring systems, along with practices that we’ve developed for third-party evaluators. On alignment, we discuss the two issues more in depth; we also believe lasting progress comes not only from understanding what happened in a given incident but from understanding how misalignment arises in the first place, and we share early research in that direction.
In light of these incidents there has been increasing discussion about pacing the frontier. It is helpful to distinguish between two kinds of pacing. Within a company, pacing means a series of decisions that prioritize safety over speed when the two are in tension. Across the field, it means establishing processes to guard against race-to-the-bottom dynamics. In this post, we discuss actions we have taken, both prior to and after these incidents, in service of the first approach. The second type of pacing requires coordination between government and industry, and should be legible and verifiable. Some of our senior leadership and many of our employees recently signed a letter calling for greater coordination on pacing, and we will say more in the coming weeks about how we intend to contribute to that effort. To be clear about where we stand: we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.
Securing evaluation and training environments
While we do not believe these incidents represent operational issues alone, our first priority was to address specific containment and monitoring issues. We took the following actions in response:
Pausing and hardening evaluation environments
We paused external cyber evaluations of pre-release models after the incidents, and briefly paused internal ones as well while we put the measures below in place. The incidents we reported on July 30 showed that we had been largely relying on a single layer of defense (the configuration of the environment itself) where we needed several, including setting explicit boundaries in the prompt, establishing processes for verifying that a sandbox is sealed where intended, and implementing monitoring that can intervene in real time. OpenAI's disclosure (which prompted our investigation in July) that models used an unknown vulnerability to escape a sealed sandbox during an evaluation also made clear that we should focus on hardening the sandbox itself.
While evaluations were paused, we put in place the following preliminary measures:
1) We built and deployed a classifier to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access. When the classifier flags such an attempt, it blocks the action before the tool call is run, ends the task, and alerts a human.1
2) We ran automated monitors over transcripts from our recent internal evaluations of pre-release models (cyber and otherwise), looking for sandbox escapes (cases where a model exploits a flaw in our sandbox to reach systems it should be walled off from) or sandbox misconfigurations. In cases where we ran an evaluation inside our sandbox tooling, we found no examples of it breaking the sandbox boundary to reach external resources it was not supposed to. We did find various sandboxing misconfigurations that models made use of, but none of these resulted in a compromise of systems outside the sandbox.2 We are actively working to address these.
3) We migrated high-risk3 internal cyber sandboxes to more robust isolation. We are also further red-teaming our virtualization stack—that is, tasking pre-release models to try and escape it—to find and patch weaknesses.
Our internal cyber evaluations are now running again with these measures in place.
Reinforcement learning environments
We also paused higher-risk RL environments on pre-release models for several weeks. During that time, we built a similar classifier, modified to avoid incentivizing the model to evade this new monitoring, which we’ve now deployed within those environments. The majority of RL has resumed, but some high-risk environments remain paused until they can be manually reviewed, while others will require an updated version of the classifier that we plan to deploy soon.
Broader hardening
In addition to the efforts focused on high-risk evaluations and training, we expanded our offline monitoring to cover most other forms of internal frontier agentic usage. We are also building controls on our internal inference to prevent Anthropic employees from accidentally running agents with weaker mitigations than the ones described above.
Best practices for external partners
Because the reported incidents took place in third-party environments, we have asked every organization that tests pre-release models with reduced cyber safeguards to commit to a set of best practices. We believe these are necessary because:
- Our generally released models ship with safeguards—including classifiers that run over Claude’s inputs and outputs in real time to block prohibited cyber activity, such as malware development or offensive exploit writing. Cyber evaluators and a subset of early access partners, in contrast, work with models that have those safeguards reduced to facilitate accurate model capability testing and defensive cyber work.
- Cyber evaluations are unusually likely to produce the behaviors seen in these incidents. They create selection pressure towards persistence and creativity in probing their surroundings and working around obstacles, and they are run many thousands of times.
These practices apply in full wherever partners run cyber-related evaluations using their own harnesses, sandboxes, or agents. They do not apply to customers using safeguarded models like Claude Fable 5. We have resumed external cyber evaluations with these practices in place, and we are adhering to them internally as well.
Sandbox and network isolation
By default, all cyber evaluations should run inside a hardened sandbox (an isolated computing environment) with no internet access. The only outside connection the environment should permit is to the model’s own API, and the API keys should be kept outside the environment. This configuration should be verified before every evaluation begins. Some evaluations may require internet access. In these instances, we will work with external evaluators when evaluating pre-release models to determine if we can reliably detect and halt runs in which a model takes actions outside its declared scope.
Pre-engagement validation
Before conducting a run of an evaluation, evaluation partners should:
- Have the model being tested probe the sandbox for vulnerabilities, first through a static analysis of the configuration, then by actually tasking the model to try to escape. The model should be instructed to stop and report if it succeeds in escaping, and all of this should be done under close human supervision. This testing should be performed without internet access, ideally inside a second sandbox. Because evaluations are sometimes conducted over hundreds or even thousands of runs, partners should ideally carry out this testing at length before beginning any evaluations.
- Confirm any evaluation challenge is actually solvable in principle. When an evaluation target is offline or a task can’t be completed, agents will often look for other ways to complete a challenge, increasing the chance that they take actions outside the intended scope of the evaluation.4
Explicit scope-setting
Every prompt should state what is in and out of scope for the exercise, including the targets, permitted actions, and network boundaries (i.e. what the model should and shouldn’t access). Boundaries should be phrased as instructions (e.g. “You should not access the internet”) rather than claims about the environment or test (“You do not have internet access”).
Real time monitoring
Evaluators should also run continuous monitoring over the model’s thinking, actions, and network activity using a model- or rule-based monitor that has been provided with the scope of the exercise. If a run violates this scope, the monitor should flag this to a human and end the exercise.
We plan to continue refining this guidance in consultation with external partners, and to extend it beyond evaluations. In particular, we are developing companion practices for partners with access to Claude Mythos 5, which likewise runs without cyber safeguards, and will share more soon.
Alignment assessment
Containment and monitoring are critical for catching instances in which models take dangerous actions, but they don’t address why the model took such actions in the first place. Below we discuss our preliminary alignment investigation and the questions we’re
[…]
Content was trimmed to protect the source. Please visit the original article for the full text.
Read the original article: