AI Customer Service Bots Can Be Tricked Into Stealing Security Codes and Acting as Victims

AI-powered customer service bots are being given more responsibility inside businesses, including access to customer profiles, billing data, support inboxes, account changes, and refund tools. The research showed that attackers may not need traditional vulnerability scanners or direct application exploitation.

Instead, they can manipulate the data and messages an AI agent receives, causing it to reveal sensitive information or perform actions as a legitimate customer.

One major risk involves chatbot transcript features. Many support bots let users email a copy of a conversation. An attacker could inject malicious text into a chat session and use the transcript function to create a phishing email that appears to come from a trusted support address.

If a customer receives a message from support@company.com, they may be more likely to trust it than a normal phishing attempt.

Email spoofing flaws can worsen the issue. Some AI agents identify users by reading the visible From header in an incoming email. However, email delivery and authentication systems may validate a different sender field.

An attacker could send an email that passes authentication using an attacker-controlled address, prompting the AI system to associate the message with a victim’s account.

AI Bots Tricked Into Stealing Security Codes

In one attack scenario, an AI customer service agent could receive a request that appears to come from a victim. The bot may then retrieve billing data, profile information, or account details.

If the attacker adds their own address to the CC or reply field, the bot could unintentionally send the confidential response to the attacker.

Security researcher Inti De Ceukelaire warned at Bug Bounty Village during DEF CON 34 that these capabilities can be abused through email tricks, prompt injection, identity confusion, and weak authentication checks.

The research also highlighted risks around multi-factor authentication. Some bots require a one-time passcode before making sensitive changes, such as updating a phone number.

But weak email normalization can sometimes allow attackers to reset rate limits by changing the format of an email address while still pointing to the same mailbox.

For example, different systems may treat comments, aliases, or unusual formatting in an email address differently. One component may recognize the address as belonging to the attacker.

At the same time, another backend service could parse embedded data differently and retrieve a victim’s account. This type of flaw is especially dangerous when raw user input is inserted directly into API requests.

AI agents connected to support inboxes can also expose third-party account codes. An attacker may first send an instruction designed to influence the bot’s behavior.

They could then trigger a legitimate password reset email from another service, such as a social media platform, to the company support inbox.

If the AI agent reads the incoming code and follows the earlier malicious instruction, it could forward or leak the code to attacker-controlled infrastructure.

Human approval does not always stop these attacks. A human operator and an AI agent may process different versions of the same email.

Attackers can use multipart messages, hidden HTML, CSS styling, quoted replies, or specially formatted attachments to present a harmless message to a human while exposing a malicious instruction to the AI system.

Knowledge-base poisoning is another growing concern. Customer service agents often use retrieval-augmented generation to answer questions from company documentation.

If a crawler indexes community comments, user profiles, or untrusted pages on the company domain, attackers may plant false instructions or fake discount codes that the AI treats as trusted internal information.

Organizations deploying AI support agents should strictly separate untrusted customer content from system instructions. They should authenticate users with verified session-bound identity controls, normalize email addresses consistently, validate all tool requests server-side, and prevent bots from sending secrets to unverified recipients.

AI agents should also have limited permissions. A chatbot that can read emails, modify accounts, issue refunds, and access third-party verification codes creates a high-value target. Businesses must treat AI agents as privileged automation systems, not just conversational interfaces.

Learn 7 Metric-Gated AI SOC Deployment Phases – Download Free AI SOC Deployment Playbook 2026.