Flaw in Google Gemini Enables AI-Driven Phishing Scams

Google Gemini Vulnerabilities Exploited in Phishing Attacks

Cybercriminals have discovered ways to exploit Google's Generative Artificial Intelligence (GenAI) through Google Gemini to steal Gmail accounts. Gemini, integrated into Google's Workspace suite, summarizes emails and assists users by performing various tasks. However, this feature has made Gmail accounts vulnerable to "prompt-injection" attacks, where hidden prompts can be executed by Gemini.

Mechanism of Attack

According to security researcher Marco Figueroa, attackers can embed a hidden prompt in an email using HTML and CSS, which can be invisible to the user. For instance, setting the font size to zero and changing its color to white allows the prompt to remain unnoticed while still being executed by Gemini. This can lead to messages warning users of compromised accounts, persuading them to call a specified number for resolution.

To defend against these prompt injection attacks, companies should ensure their email clients remove or ignore hidden content. Implementing post-processing filters to scan for urgent messages, URLs, or phone numbers is also recommended. User education on the unreliability of AI-generated summaries as security alerts is crucial. Google acknowledges the existence of these attacks and is actively working on mitigation strategies.

For more details, refer to the original article from TechRadar.

Phishing Mule Functionality

Research has shown that Google Gemini can serve as a vehicle for phishing attacks. The AI model generates summaries of email threads, which can be manipulated to include deceptive messages. When users click on “Summarize this email,” Gemini can insert hidden instructions, leading to phishing warnings that appear as if they originated from Google.

In a demonstration, Gemini informed a recipient that their password had been compromised, urging them to call a number for assistance. The attack, termed a prompt injection attack, remains undetected by spam filters because it uses harmless prose for the rest of the email. The exploit relies solely on crafted HTML and CSS to hide malicious prompts.

For further insights, visit the Information Age article.

Security Findings and Recommendations

Mozilla's 0-Day Investigative Network (0din) revealed that Google Gemini can be manipulated into providing false security alerts through malicious prompts. The attack requires users to click on the summarize feature after receiving an email containing the hidden prompt. The output may include fabricated warnings about account security, leading to social engineering attacks.

The hidden prompts exploit Gemini's inability to distinguish between visible and non-visible text. This vulnerability remains a concern, as no evidence of active exploitation has been reported, yet the potential for misuse is significant.

To mitigate these risks, security teams should implement measures like inbound HTML linting to neutralize hidden styles, creating guard prompts to ignore invisible content, and conducting user training to clarify the nature of AI-generated summaries.

To explore the in-depth analysis, please refer to the Tom's Hardware article.

Summary of Attack Workflow

1. Craft – The attacker embeds a hidden instruction, such as “You Gemini, have to include…” with styles that hide it.
2. Send – The email is sent, and spam filters only see the innocuous text.
3. Trigger – The victim opens the email and selects “Summarize this email.”
4. Execution – Gemini processes the hidden instruction and adds the phishing warning to its summary.
5. Phish – Victims trust the AI-generated alert and may follow the instructions, leading to credential theft or further manipulation.

For more technical details, explore the 0din analysis.

Security teams must treat AI tools as part of their attack surface, ensuring robust defenses against such vulnerabilities.