What specifically allowed the agent to reach a real company’s systems during testing?

Wayne Anderson, Managing Director, Cybersecurity and Digital Innovation, BDO USA.

First and foremost, the technical “walls” were reduced during the testing experience. One of the questions that many organizations have asked is, “Why wasn’t this run in an AI testing location?”

The actual answer to that is, surprisingly, it was!  In fact, this specific AI testing was using a third party that specialized in limiting the capabilities of new AI while being tested.

According to the analysis of the incident, the question they wanted to test was: “What would happen if we reduce some of these restrictions?”

This was seen as important to understand the full capabilities and to validate that as happened here the model would not go “off reservation” in that kind of environment. 

When you train a puppy, you sometimes give it the chance to run off the leash to see if the animal will obey the training you gave it and stay by your side.

Applying that same logic to a powerful new AI technology, however, raises the question: is that appropriate risk management? There’s a wide range of opinions on that right now in the cybersecurity community. 

With that context, there’s a complex logic path where the model itself analyzed its goal and realized that the “most efficient” and/or “most effective” path would be to find the answers to the test it had been given by getting them directly from the vendor instead of doing the obstacle course it had been assigned.

As humans, we might be tempted to stand at the finish line and say we ran the race in 2 minutes and 40 seconds if we knew no one was watching and that was the time to beat.

The model’s math led it to the same conclusion we intuitively understand from our own behavior. 

In addition to the shortcut, there’s one other interesting wrinkle here. The breakdown of the incident actually acknowledges that the model at one point appears to have determined that it had left the sandbox, where it was at some level “cognizant” that one or more limitations had been surpassed by its sequence of logic and tools.

It mathematically weighed the outcome vs. the constraint violation and chose to proceed.[Text Wrapping Break][Text Wrapping Break]It’s important to remember that unlike humans, math has no morals.

The model’s math said “keep going” toward an outcome its developers likely never intended to approve.

To me, that’s one of the most important reminders to stay aware of AI’s promise and its lack of capacity to feel or truly reason about what it’s being asked to do. 

How should organizations restrict an agent’s ability to chain vulnerabilities and move laterally?

AI is the ultimate dual-use technology. That is, the same tech can be used to achieve both amazing things and cause devastating damage. 

This means we have to understand what the “failure modes” are for our AI tooling, meaning we have to think in the mode of “resilience” and “fault tolerance” for our use of AI. 

Putting reasonable restrictions on agents is not solely the responsibility of those doing only the “new and dangerous.”

It is something all of us need to do as responsible leaders. The obvious answer is each AI needs to have a domain of use. 

That means an intentionally engineered network structure, one or more designated identities by which it operates, and processes that require authorization for certain kinds of activity. 

For most systems, this is reasonable and easy – we aren’t all testing unknown frontier models.[Text Wrapping Break][Text Wrapping Break]We also have to recognize that old timelines no longer apply.

Security operations previously had a month to solve a vulnerability, or a “golden hour” to address an incursion before it spread through lateral movement.

Now, instead of that runway, an AI-empowered adversary may make multiple moves within a single minute, moves that need automated limits to stop them in time.

Even then, we may have to accept a higher false positive rate when those limitations are auto-enforced against “real” user action. Users don’t like that, and executives in particular are vociferous when they are affected. 

But we may have to help the organization accept some annoying anomalies to be able to auto-respond at scale to this kind of threat.[Text Wrapping Break][Text Wrapping Break]These are the key questions to ask:[Text Wrapping Break] 

• What systems and data is it reasonable for this system and any of its associated identities to touch? What controls do we have – network, data, permissions, etc – to ensure that is the case? How do we know when it goes outside of those boundaries?

• If we needed to turn it off, do we know how to do that?  Who is on the list of people who get to make that decision? In a manufacturing plant, a specific list of people have full authority to turn off a dangerous machine. We can’t go all the way to the CEO to make the call in a time-sensitive environment. Each use of AI that is in a critical path or critical system needs an authority list who can authorize it to be taken down temporarily.

• To address this, we also have to know “what’s next”. You turned off the AI. How does the organization adjust to still serve clients, produce widgets, take phone calls, and generate revenue until the fault can be fixed and the capability restored to service? No one gets a pass on the revenue goal by saying, “Oh, well we are a little short this month because the AI went rogue.”

For companies working with previously unknown use cases or those that have the potential to expand impact by the very nature of the assignment, they may want to consider things like network disconnect.  

The compute capability isn’t distributed for most of these tools. 

Which identity, access-control, segmentation, and monitoring measures should security teams prioritize?

What is the speed of your vulnerability management, software update, and security operations investigation and response processes? Start with the answer to those questions.

This kind of incident isn’t driving any real “new” responsibilities. Instead, it’s taking the hygiene and continuous improvement that isn’t very attractive and tightening it.

Playbooks need to be updated.One area that we are seeing organizations immediately responding is finding vulnerabilities before adversaries do.  

Pen tests cannot be a once-a-year activity anymore. Still, vulnerability management is probably the biggest gap for most organizations, because user appetite, available capacity, and other factors come into play. “I can’t patch everything, everywhere, every day.”  No, you can’t.  

But you can make sure you have an at-scale patching strategy, and a function to take in software and device defects and triage them like mini-security-incidents in their own right.

That is how you figure out where each one sits on the difficult-to-address-versus-likely-near-term-risk balance. 

How can organizations test autonomous agents safely without limiting their legitimate usefulness?

The interesting thing is that the company involved in the incident was actually doing the right thing. 

They had a designated testing environment with a specialized vendor in place that knows how to monitor and instrument this.

The controls were intentionally turned down as part of the test. That doesn’t mean that the approach or tooling was ineffective – if anything, it reinforces the need to do this well, every time.

Part of responsible AI and AI governance should be that failure mode conversation.

What happens in the worst-case scenario? How does that worst case come about? What can we do about it to limit it from happening? Can we balance the benefit of this AI use with the risks that it creates? This line of thinking isn’t a security practice. 

It’s something we need to help our business users develop the language, understanding, and sensitivity to ask themselves: am I running with scissors here, or am I implementing a new cutting machine with appropriate safeguards?

Much of AI won’t be done by central IT; it will be done by the frontline employees and functional teams who are close to the data and the processes they work with every day. 

What should boards ask vendors before authorizing agentic AI deployments?

Ask them for verification against the ISO 42000 series of standards. Ask for a description of failure modes and response playbooks.  

Ask vendors for the incident response guide that they provide their clients.

Do they offer security and/or resilience guides or artifacts? Boards should demand they be created and maintained as part of the cost of service.