AI systems in the cloud combine everything security teams already worry about (data stores, compute clusters, CI/CD and APIs) with new assets such as model weights, vector databases and autonomous agents.
This article sets out a vendor-neutral AI workload security framework across five layers: identity, data, model supply chain, inference and runtime, and governance. Each layer lists concrete controls, detection ideas and the framework references they map to.
The guidance applies whether you train your own models, fine-tune open-weight models or call a managed model API from your own applications.
Why AI workload security needs its own control set
Breaches involving AI are already common, and access control is the recurring failure. IBM’s Cost of a Data Breach Report 2025 found that 13% of organizations reported breaches of AI models or applications, and 97% of those lacked proper AI access controls (2025).
The same report found that organizations with high levels of shadow AI paid about $670,000 more per breach, and 63% of breached organizations had no AI governance policy or were still developing one.
Three properties make AI workloads different from ordinary applications:
- Models are executable artifacts. Some model formats run code when loaded, so a downloaded model is closer to a binary than a data file.
- Inputs are instructions. Language models cannot reliably separate data from commands, which creates prompt injection.
- Machine identities do the work. Training jobs, pipelines and agents use service accounts and API keys with broad data access.
The OWASP Top 10 for LLM Applications 2025 and MITRE ATLAS catalog these risks; NIST’s AI 600-1 Generative AI Profile maps them to the AI Risk Management Framework.
OWASP has since published a 2026 revision, summarized in this overview of the OWASP GenAI LLM Top 10 2026.
Layer 1: Non-human identities and access
Identity is where most cloud AI incidents begin. Leaked cloud keys are tested by attackers within minutes; Entro Labs exposed working AWS keys publicly and saw first attacker API calls in 9 to 17 minutes, as reported in hackers hijacking cloud AI models with exposed keys.
The motive is often “LLMjacking”: using a victim’s account to run paid model inference.
Illustrative LLMjacking chain
This sequence follows a case documented by FortiGuard Labs and described in an LLMjacking attack using a leaked AWS IAM key:
- A long-lived IAM access key with administrator rights is exposed (T1552.001, Credentials in Files).
- The attacker authenticates and creates a new IAM user for persistence (T1136.003, Create Cloud Account).
- The attacker subscribes to paid models and invokes them, with charges billed to the victim (T1496, Resource Hijacking).
Identity controls
- Replace long-lived keys with workload identity federation and short-lived, role-assumed credentials for training jobs, pipelines and agents.
- Give each model service, pipeline stage and agent its own identity, scoped to the specific buckets, endpoints and model IDs it needs.
- Use service control policies or organization policies to restrict which model families and regions can be invoked.
- Inventory non-human identities and rotate or remove unused ones on a schedule (CIS Controls v8, Controls 5 and 6).
Detection example
An illustrative CloudTrail query (Amazon Athena) to spot unusual model invocation by identity:
SELECT useridentity.arn, sourceipaddress, eventname, count(*) AS calls
FROM cloudtrail_logs
WHERE eventsource = ‘bedrock.amazonaws.com’
AND eventname IN (‘InvokeModel’,’InvokeModelWithResponseStream’,’Converse’)
AND eventtime > date_format(current_timestamp – interval ‘1’ day, ‘%Y-%m-%dT%H:%i:%sZ’)
GROUP BY 1,2,3
HAVING count(*) > 500
ORDER BY calls DESC;
Pair this with alerts on new IAM users, new access keys, and model access or marketplace subscription changes. Enable model invocation logging where your provider supports it.
Layer 2: Training data and retrieval data protection
Training sets, fine-tuning corpora and retrieval-augmented generation (RAG) indexes often aggregate the most sensitive data a company holds.
They also introduce data and model poisoning (OWASP LLM04:2025) and sensitive information disclosure (LLM02:2025).
- Classify before you ingest. Run data discovery on sources feeding training or RAG, and strip or tokenize regulated data that the model does not need.
- Preserve access controls in retrieval. A RAG system should filter documents by the requesting user’s permissions at query time, not index everything into one shared store.
- Protect vector databases as production data stores: private networking, encryption with customer-managed keys, and no public endpoints.
- Track provenance. Record dataset versions, sources and hashes so you can trace and roll back a poisoned dataset.
- Restrict write paths. Few identities should be able to modify training buckets; alert on writes outside pipeline runs.
Layer 3: Model supply chain and MLOps security
Models, notebooks, Python packages and container images move through the same kind of pipeline as application code, and they are attacked the same way.
Pickle-based model files are a specific hazard because deserialization can execute code.
ReversingLabs found malicious models on Hugging Face that abused Pickle serialization to evade platform scanning (2025).
MLOps controls
- Prefer weight formats that do not execute code on load, such as safetensors, and block pickle loading in production where possible.
- Pull models only through an internal registry or proxy that scans artifacts and records hashes.
- Pin model versions by hash, not by mutable name or “latest”.
- Sign model artifacts and verify signatures before deployment, the same way you sign container images.
- Generate SBOMs for serving images and keep an AI bill of materials listing models, datasets and dependencies.
- Isolate notebooks and training clusters from production networks and from broad internet egress.
This aligns with OWASP LLM03:2025 (Supply Chain) and with ATT&CK T1195 (Supply Chain Compromise).
Layer 4: Inference, prompts and agent controls
Once a model is serving requests, the main risks are prompt injection (LLM01:2025), improper output handling (LLM05:2025), excessive agency (LLM06:2025) and unbounded consumption (LLM10:2025).
Indirect prompt injection is the hardest case: instructions hidden in a document, email or web page that the model reads.
See how prompt injection attacks bypass AI agents through user input for real examples.
| Risk | Control |
|---|---|
| Prompt injection | Treat all model output as untrusted; separate system instructions from retrieved content; use input and output classifiers as one layer, not the only one |
| Improper output handling | Encode or validate model output before it reaches browsers, shells, SQL or APIs |
| Excessive agency | Give agents minimal tools and scopes; require human approval for payments, deletions and external messages |
| Unbounded consumption | Per-identity rate limits, token quotas and budget alerts |
| System prompt leakage | Keep secrets and authorization logic out of prompts entirely |
Workflow automation increases exposure because agents act across systems. Review connectors and permissions in AI workflows with the same rigor as any privileged integration.
Layer 5: Cloud posture, runtime monitoring and governance
AI infrastructure still runs on ordinary cloud resources: GPU instances, Kubernetes clusters, storage buckets and managed notebooks. Misconfigured notebooks with public endpoints, over-permissive service accounts and unencrypted model buckets are frequent findings.
- Posture management. Extend CSPM or CNAPP coverage to AI services, including managed notebooks, model endpoints and vector stores. Agentless platforms such as Orca Security are one example of tools that inventory AI resources alongside other cloud assets; evaluate several against your own requirements.
- Runtime detection. Monitor training and serving containers for unexpected processes, outbound connections and crypto-mining behavior.
- Logging. Capture model invocation logs, prompt and response metadata (with privacy controls), and administrative API calls in your SIEM.
- Governance. Maintain an approved model and tool list, a process for reviewing new AI use cases, and regular audits for shadow AI. Map controls to NIST CSF 2.0 Govern and to the NIST AI RMF functions (Govern, Map, Measure, Manage).
Incident response steps for a suspected AI workload compromise
- Disable or rotate the implicated credentials and revoke active sessions.
- Snapshot affected compute and preserve invocation and audit logs.
- Check for new identities, model subscriptions, endpoints and changes to training data.
- Verify model and dataset hashes against known-good versions; redeploy from signed artifacts if anything differs.
- Review billing for anomalous inference or GPU spend to scope the abuse window.
Key takeaways
- AI workload security rests on identity first: short-lived credentials and per-workload scopes block the most common attacks.
- Treat model files as executable code; prefer safetensors, pin by hash and sign artifacts.
- RAG systems must enforce user permissions at retrieval time.
- Assume prompt injection will succeed and limit what model output and agents can do.
- Log model invocations and alert on unusual volume, new identities and new model subscriptions.
- Use OWASP LLM Top 10, MITRE ATLAS and the NIST AI RMF as shared reference points for controls.
FAQ
What is AI workload security?
It is the set of controls that protect AI systems running in the cloud, including the identities they use, training and retrieval data, model artifacts, inference endpoints and the infrastructure underneath.
What is LLMjacking?
LLMjacking is the abuse of stolen cloud credentials or API keys to run paid AI model inference at the victim’s expense. It usually starts with a leaked long-lived access key.
How do you secure machine learning models in the cloud?
Store models in a private registry, scan and sign them, load only safe formats, pin versions by hash, and restrict which identities can deploy or modify them.
What is the OWASP Top 10 for LLM applications?
It is an OWASP list of the most critical security risks for applications built on large language models, including prompt injection, sensitive information disclosure, supply chain and excessive agency.
How do you prevent prompt injection in AI applications?
There is no complete fix. Reduce impact by limiting model permissions and tools, validating outputs, separating instructions from untrusted content and requiring human approval for sensitive actions.
