Understanding the Dual Threat of Generative AI in Cybersecurity

Generative AI is swiftly integrating into our daily workflows, empowering employees to use AI assistants for tasks like summarizing documents, searching for company knowledge, drafting content, and automating daily tasks. Organizations are increasingly adopting AI agents that interact with business applications and manage workflows with minimal human intervention.

While the promise of significant productivity gains is enticing, it also introduces new security considerations. AI will access the same identities, business data, and systems targeted by cybercriminals, potentially escalating the speed and scale of ransomware attacks if not appropriately managed.

Importantly, AI does not create entirely new ransomware threats; instead, it amplifies existing techniques used by attackers, particularly during reconnaissance, credential abuse, and data theft. Understanding how AI modifies the attack surface is crucial for enhancing enterprise cyber resilience.

Two AI Threat Models Every Organization Needs to Recognize

Conversations surrounding AI and ransomware typically intertwine two distinct threat models:

• Attackers leveraging AI: Cybercriminals are increasingly using AI to enhance their operations, such as generating phishing emails, creating malicious code, automating reconnaissance, analyzing stolen data, and streamlining extortion efforts. This enables attackers to work faster and at a larger scale without fundamentally altering how ransomware campaigns are executed.
• Organizations implementing enterprise AI: AI assistants and agents are becoming linked to document repositories, collaboration platforms, SaaS applications, and internal knowledge bases. When attackers compromise identities tied to these systems, AI can hasten their ability to unearth sensitive information, navigate interconnected systems, and exploit legitimate access.

These trends occur concurrently. As attackers sharpen their efficiency through AI, organizations must ensure their AI deployments do not inadvertently widen their attack surface.

New Exposures Created by Enterprise AI

Not all AI applications carry the same risk. AI assistants primarily retrieve information or generate content based on prompts. The AI agents also interact with business applications, call APIs, and execute actions on users’ behalf.

The greater the autonomy and privileges of an application, the more significant the potential impact if its associated identity is compromised.

The real concern lies in the delegation of authority. Modern ransomware campaigns generally initiate by exploiting vulnerabilities, compromising credentials, or abusing trusted third-party access. Attackers then conduct discovery, escalate privileges, identify valuable data, and steal information before proceeding with encryption, blackmail, or both.

A Microsoft report reveals an analysis of around 38 million identity risk detections daily, underscoring the prevalence of identity attacks. The Cloud Security Alliance has also documented a massive OAuth device code phishing campaign aimed at Microsoft 365 users.

AI-enabled applications pose new security challenges, from rapid injection and unauthorized data access to AI-assisted reconnaissance.

Acronis GenAI Protection facilitates the identification of shadow AI usage, monitoring of prompts and AI interactions, detection of policy violations, and visibility into AI-related risks via endpoint, identity, SaaS, and backup telemetry. Power detection, response, and recovery with a unified cyber resiliency platform.

How Identity Compromise Transforms AI into an Attack Accelerator

The importance of these attacks for enterprise AI lies in the fact that AI assistants and agents inherit identities and delegated authority based on their behavior. Once attackers compromise these identities, they can also gain entry to AI services, corporate data, and connected applications accessible through the same privileges.

AI assistants linked with corporate knowledge can dramatically lower the effort required to locate sensitive information. An attacker with legitimate credentials could ask an AI assistant to find backup documents, administrative procedures, customer data, or financial records, as opposed to manually sifting through countless folders.

In the same vein, if an AI agent possesses the capability to send emails, export files, or access linked business tools, a compromised identity could enable an attacker to utilize these functionalities for expedited data theft or misconduct. The core risk doesn’t stem from the AI but from excessive access.

Prompt injection is a specific AI application layer vulnerability, but it is just one facet of the larger risks posed by excessive privileges, insecure integrations, and inadequate oversight.

Malicious instructions embedded within documents, emails, or web content can influence AI behavior when captured by enterprise applications. The impact mainly relies on the permissions granted to the AI system. Thus, OWASP emphasizes adopting layered controls, least privilege principles, and requiring human approval for high-risk actions, rather than relying solely on instant filtering.

AI’s Role in Enhancing Cybercrime Efficiency

Evidence indicates that AI accelerates existing cybercrime rather than fundamentally altering attack mechanics. According to the Acronis Cyber Threat Report H2 2025, various examples showcase AI’s role in supporting several stages of cyber operations:

• The GTG-2002 threat group employed AI to create and debug scripts, assist in credential gathering, analyze stolen data, and personalize extortion communications, allowing relatively small attack teams to upscale operations.
• GLOBAL GROUP’s ransomware operation used an AI chatbot to automate ransom negotiations post-breach. The chatbots didn’t modify the infection chain but allowed operators to handle more victims concurrently while reserving human negotiators for intricate cases.
• Research shows that Chinese state-backed groups utilize agent AI to conduct a significant portion of their cyber espionage, including reconnaissance, vulnerability probing, credential harvesting, and data collection.

Additionally, ransomware-as-a-service operators increasingly advertise AI-assisted automation for defense evasion and operational efficiency. These advertisements highlight how ransomware operators are utilizing AI and where they anticipate it will enhance efficiency, though individual feature claims might not have been independently verified.

Collectively, these instances demonstrate that AI fundamentally acts as an operational enhancer rather than introducing entirely novel attack techniques.

Six AI-Powered Controls to Mitigate Ransomware Risks

Organizations aren’t required to overhaul their current security strategies for enterprise AI; however, they must bolster them with AI-specific governance, access controls, and monitoring. Acronis Security Experts advise organizations extend their current governance, identity, and data protection practices to encompass AI applications and workflows by:

1. Inventory Management: Maintain an overview of approved and unapproved AI applications, models, and integrations. Define ownership, business purposes, and risk classifications for all AI workflows.
2. Grant Least Privilege Access: Limit user access to AI applications, service accounts, and APIs. Regularly review delegated permissions, revoke unused credentials, and minimize AI access to only the systems and information necessary for each task.
3. Control AI-related Traffic: Regulate the movement of AI-related data and traffic. Discover AI services, restrict access to unauthorized tools, and prevent sensitive information from being transmitted or uploaded using secure web gateways, CASB, and DLP capabilities.
4. Monitor and Audit AI Activity: Correlate AI application use, identity events, data access, exports, and agent actions with telemetry from endpoint, SaaS, and cloud within your SIEM or XDR system. Maintain an audit trail that reveals which user or service account initiated an action, which resources were accessed, and whether approval was necessary.
5. Prepare for Containment and Recovery: Security teams must swiftly revoke compromised tokens, disable affected integrations, and pause AI workflows upon detecting malicious activity. Immutable backups and tested recovery procedures remain crucial for restoring operations after a destructive attack, although they cannot counter data theft or nullify extortion risks.
6. Implement Human or Policy-Based Authorization: Require authorization for high-risk actions such as bulk exports, administrative changes, external communications, and code executions. OWASP recommends applying least privilege principles and human authorization for privileged operations.

Extending Cyber Resilience to Enterprise AI

The adoption of enterprise AI is set to escalate as its business advantages are undeniable. The challenge lies in ensuring that productivity enhancements do not come at the cost of security.

The most effective strategy is to integrate AI into existing identity, data protection, and incident response methods, rather than treating it as a distinct security domain. Organizations should assess AI security controls based on their integration with existing governance and security operations, along with visibility into AI usage, permissions, and policy violations.

For managed service providers (MSPs) and enterprise security teams, there exists an opportunity to widen their cyber resilience strategies to include AI governance.

Acronis GenAI Protection is deeply integrated into the Acronis platform, aiding organizations in uncovering shadow AI usage, inspecting sensitive data prompts, enforcing usage policies, and reviewing GenAI activity within the same platform as other cyber protection services.

This enables organizations to enhance GenAI governance and secure interactions with AI tools without the need for a separate management console.

As the adoption of enterprise AI accelerates, organizations combining strong governance with established cybersecurity practices will be poised to reap the productivity benefits of AI while mitigating the associated risks of ransomware.

Try Acronis GenAI Protection today to see how generative AI can safeguard your network.

Author: Santiago Pontiroli

Santiago Pontiroli is the Threat Intelligence Research Leader at Acronis Threat Research Unit (TRU), where he spearheads global research on advanced threat actors, cybercrime ecosystems, and emerging attack techniques. With over 15 years of experience in cybersecurity, he specializes in analyzing nation-state actors and criminal organizations and has presented original research at major international conferences.

Sponsored and written by Acronis.

Source: www.bleepingcomputer.com