Agentic AI for Malware Analysis: Automating Reverse Engineering with LLMs

Manual malware analysis is powerful but time-consuming. Reverse engineers spend hours on repetitive tasks—decrypting obfuscated strings, resolving hash-based API imports, unpacking multi-stage samples, and extracting indicators of compromise. Agentic AI, when paired with sandbox and reverse engineering tools, compresses this timeline dramatically while keeping analysts in control of the process.

In this webinar, independent reverse engineering expert Tim Blazytko explores how Large Language Models can operate as autonomous agents within a sandboxed environment, understand complex malware patterns, and automate entire analysis pipelines—from initial triage to IOC extraction. The key insight: agents work best not as replacements for human expertise, but as force multipliers that handle routine work and surface actionable intelligence.

What Is Agentic AI in Reverse Engineering?

Agentic AI differs fundamentally from chat-based prompting. Whereas traditional LLM interaction involves copy-paste workflows—sending code snippets to a chat interface for annotation—agents take a goal and iteratively solve it by calling tools, observing results, and deciding next steps.

An agent consists of four critical components:

Why this matters: Traditional reverse engineering is analyst-driven; agentic analysis is analyst-guided. The analyst provides strategic direction; the agent executes tactical tasks at superhuman speed, freeing expertise for novel problems.

Use Case 1: Automated String Decryption

One of the most common and time-consuming tasks in malware detection workflows is recovering encrypted configuration strings. Mirai, a botnet that targets IoT devices, encodes default credentials using a simple XOR cipher with a hardcoded key—a pattern analysts encounter daily.

The Manual Workflow

A human analyst would:

1. Locate the encryption routine: Find the function responsible for decryption (often via string cross-references or pattern matching in IDA/Ghidra).

2. Reverse the algorithm: Understand the XOR key, loop structure, and any additional obfuscation (e.g., rotations, increments).

3. Write a decryption script: Implement the algorithm in Python or similar, typically 5–15 lines of code.

4. Apply and annotate: Run the script against all encrypted blobs in the binary, then annotate the disassembly database with results.

Time: 30 minutes to 1 hour. Error rate: 5–10% (missed strings, misunderstood algorithm).

The Agentic Workflow

The analyst provides the address of the encryption function and tells the agent: decrypt({0x4a, 0x56, 0x56, 0x52, ...}) returns "admin". The agent:

1. Opens the binary in Ghidra Headless via MCP

2. Extracts and decompiles the function at the provided address

3. Analyzes the pseudo-C code to identify the XOR key (0x22 in Mirai's case)

4. Enumerates all encrypted string blobs across the binary using pattern matching (identifying call sites, cross-references)

5. Generates a Python script that decrypts each blob

6. Executes the script and writes results back to the Ghidra database as annotations

Time: 1–2 minutes. Accuracy: Near 100% (consistent algorithm application).

Use Case 2: Resolving API Hash-Based Imports

Modern malware often avoids direct imports by computing hashes of Windows API function names at runtime and doing a lookup to resolve them. This obfuscation defeats static analysis, forcing analysts to either reverse the hash function manually or use dynamic analysis to observe actual calls.

The Hash Resolution Challenge

CQing, a recent ransomware variant, uses this technique extensively. The malware contains code like:

if hash(api) == 0x6F0A52E1  // NtCreateFile
if hash(api) == 0xB45C8907  // NtAllocateVirtualMemory
if hash(api) == 0x29E3D104  // NtProtectVirtualMemory
NtClose, ...

Manual resolution process:

1. Reverse the hash function: Understand the algorithm—is it CRC32, DJB2, custom? Typically 10–50 lines of assembly.

2. Enumerate exports: Extract all exported functions from ntdll.dll, kernel32.dll, etc. using public headers or system snapshots.

3. Brute-force: Hash each export and compare against hardcoded constants in the binary. With 1000+ exports and 100+ hash checks, this becomes computationally intensive.

4. Map results: For each match, annotate the binary with the function name, requiring manual cross-referencing.

Time: Half a day or more. Error rate: 10–20% (hash collisions, misidentified algorithm).

The Agentic Approach

Analyst provides the hash function address and target DLL. Agent:

1. Decompiles the hash function using Ghidra MCP, understanding the algorithm in seconds

2. Fetches DLL exports programmatically (can embed a snapshot or query WinAPI in a test environment)

3. Generates optimized brute-force code:

for func_name in ntdll_exports:
    h = hash(func_name)
    if h == 0x6F0A52E1:
        resolved[0x6F0A52E1] = "NtCreateFile"
    elif h == 0xB45C8907:
        resolved[0xB45C8907] = "NtAllocateVirtualMemory"

1. Executes brute-force in parallel, identifying all matches in minutes

2. Annotates the database with resolved names and writes output to case directory

Time: 3–5 minutes. Accuracy: 100% (deterministic hash matching). Result:

0x6F0A52E1 = NtCreateFile
0xB45C8907 = NtAllocateVirtualMemory
0x29E3D104 = NtProtectVirtualMemory

Use Case 3: Multi-Stage Unpacking and Reconstruction

Many malware families are distributed as multi-stage droppers, with the real payload encrypted or obfuscated in later stages. FinFisher, for example, has 4–5 stages. Unpacking them manually requires dynamic analysis, static disassembly, and careful extraction of decrypted payloads—a process that can take days.

Agents can speed this significantly. By combining emulation-based analysis in a sandbox with static reverse engineering, an agent can:

1. Identify unpacking routines

2. Extract intermediate payloads as they're decrypted

3. Recursively analyze each stage

4. Build a dependency graph of the entire infection chain

The agent reports each stage's functionality, extracted indicators, and file hashes—providing a complete picture faster than manual analysis.

The Repetitive Side of Malware Analysis: What Agents Handle

Tim's research identifies six categories of repetitive malware analysis work that agents automate reliably:

Key insight: Agents excel at these focused, deterministic tasks. A single agent running for 1–2 hours can automate work that would take a human 1–2 weeks. The analyst's role shifts from manual execution to validation and strategic decision-making.

Automated Initial Triage: The Malware Analysis Pipeline

The most powerful agentic workflow is automated initial triage. When an unknown binary arrives, you need a rapid, structured answer: What is it? What does it do? What are the IOCs?

The Triage Pipeline: Four Stages

Stage 1: Artifact Collection — Run standard tools in parallel: capa (behavior analysis), Yara (signature matches), FLOSS (string extraction), basic imports enumeration. Dump all output to disk files for the agent to consume.

Stage 2: Inspection & Hypothesis Generation — Agent reads artifact files, filters by relevance (what's truly suspicious vs. noise), ranks findings by confidence, and generates ranked hypotheses. Examples: "likely EDR killer," "probable C2 communication," "potential cryptominer."

Stage 3: Structure Mapping & Deep Analysis — Agent spawns the disassembler (Ghidra Headless MCP or Binary Ninja MCP), cross-references discovered strings and API calls against the code, maps structural components (vtables, protocol handlers, encryption routines), and validates or refines hypotheses.

Stage 4: Reporting & IOC Extraction — Agent writes findings to a structured report, extracts IOCs (hardcoded IPs, domains, C2 endpoints, registry keys), generates Yara rules for detection, and writes everything to a case directory on disk.

The Agent Roles: Planner, Worker, Reporter

Tim's architecture uses specialized agent roles, each with a specific responsibility:

This multi-agent pattern enables complex workflows: the Planner can reason about incomplete information and steer the analysis, the Worker handles technical execution reliably, and the Reporter ensures output quality. All agents share a case directory that persists across restarts and context window resets.

Handoff to the Analyst

Once the pipeline completes (1–2 hours autonomously), the analyst receives:

1. Ranked hypotheses with confidence scores

2. Functional component map (code structure, vtables, protocols)

3. Decoded strings and resolved API imports

4. Extracted IOCs (C2 endpoints, dropped files, crypto keys)

5. Analysis report with recommendations for deeper investigation

The analyst validates findings, decides whether to pivot (e.g., "attribution"), iterate deeper on suspected functionality, or move forward with detection rule development. The agentic pipeline has already covered 70–80% of the analysis surface.

Combining Static and Dynamic Analysis

Agentic analysis shines when static and dynamic data are combined. A malware sample detonated in VMRay's sandbox produces detailed execution traces, network communications, dropped files, and behavior logs. An agent that processes both the binary and the sandbox dump can:

1. Use sandbox data to guide static analysis (e.g., "I see this file was created; now find where in the code it's written")

2. Validate static findings against observed behavior

3. Extract a richer IOC set: not just hashes, but C2 domains, drop locations, registry keys, and file operations in context

The integration works best through a standardized interface like the Model Context Protocol (MCP), which allows the agent to query a malware sandbox via API, retrieve analysis artifacts, and reason over both static and dynamic evidence.

Practical Setup: Docker, MCP, and Skills

The Agentic Analysis Environment

Tim's architecture isolates agents and tools in a sandboxed Docker container. The container includes Ghidra (headless mode), Binary Ninja, Yara, capa, shell scripts, and Python. Malware samples are mounted read-only; analysis results write to a case directory on disk.

The agent runtime sits atop this: Claude Code Interpreter, OpenAI Codex, or open-source runtimes like Ollama. The runtime executes code and commands inside the container, interfacing with tools via direct shell calls or the Model Context Protocol (MCP).

MCP: The Standardized Tool Interface

MCP is a client-server protocol that allows agents to discover, invoke, and process tool results in a structured way. Instead of agents generating shell commands or Python scripts (high context overhead, error-prone), they interact with MCP servers using a standard interface.

MCP advantages:

1. Tool discovery: Agent asks "what operations are available?" and receives a structured list (disassembly, decompilation, xref, types, symbols, annotations).

2. Reduced context: Agent calls mcp.call("ghidra.disassemble", {"address": "0x401000"}) instead of writing disassembly logic from scratch.

3. Reliable parsing: Tool output is guaranteed to be structured (JSON, protobuf), not fragile text parsing.

4. Stateful operations: MCP servers maintain state across calls (e.g., an open Ghidra project), unlike shell commands that reset on each invocation.

Ghidra Headless MCP: Six Core Capabilities

Ghidra Headless MCP exposes core operations:

Binary Ninja MCP provides similar coverage with faster analysis, preferred for live demonstrations and time-sensitive work.

Skills: Reusable Workflows

Skills are task-specific guidance encoded as markdown documents. They tell agents how to approach a problem, what sequences to follow, and what outputs are expected. Instead of asking the agent to "analyze this malware," you give it a skill that says:

1. Stage 1 - Artifact Collection: Run capa, Yara, FLOSS, strings, and imports enumeration; dump results to disk in structured formats (JSON, CSV).

2. Stage 2 - Filtering & Hypothesis: Read artifacts; filter by relevance (prioritize unknown imports, suspicious API calls); generate ranked hypotheses on functionality (e.g., "probable C2 communication", "likely encryption", "EDR killer suspected").

3. Stage 3 - Structure Mapping: Use Ghidra MCP to disassemble/decompile suspected functions; map control flow; identify vtables and protocol handlers.

4. Stage 4 - Report & IOC Extraction: Write findings to structured report; extract IOCs (C2 addresses, crypto keys, file drops); generate Yara rules; output to case directory.

Skills encode domain expertise once, reused across samples and teams. They limit exploration through structured guidance yet remain composable—combine multiple skills for multi-stage analysis.

Choosing the Right Model: Cloud vs. Local

Frontier Models (Cloud API)

GPT-4o, Claude 3.5 Sonnet, and Code Execution excel at agentic analysis—strong reasoning, fewer hallucinations, fast iteration. Trade-off: sample metadata and intermediate analysis artifacts may transit cloud APIs. Cost is manageable: ~$0.10–$1 USD per full analysis task. For public malware and shared threat intelligence, this is cost-effective.

On-Premises Setup with Local LLMs

Organizations handling classified, sensitive, or air-gapped samples need on-premises LLMs. The architecture remains identical to cloud-based agentic analysis, with one change: the agent runtime connects to a local LLM server instead of a cloud API.

Local setup (on-premises):

1. Local LLM Server: Ollama, llama.cpp, or LM Studio running on a dedicated machine. Loads the open-weight model (Qwen, GLM, Kimi, DeepSeek) from VRAM or quantized disk storage.

2. Agent Runtime: Connects to the local LLM server via HTTP API instead of cloud endpoints. Same interface, no cloud transit.

3. Docker Container: Contains Ghidra Headless MCP, Yara, capa, analysis tools. Accesses local LLM via network bridge.

4. Data Flow: Malware samples → Docker container → local LLM server (within same facility) → analysis results to disk. No external network required.

Open-Weight Models: Capabilities and Hardware

Available models as of July 2026:

Trade-Offs: Cloud vs. Local

Real-World Case Study: Bundestrojaner (mfc42ul.dll)

Tim Blazytko's agentic analysis pipeline successfully reversed a sophisticated surveillance RAT called Bundestrojaner. The binary (mfc42ul.dll) is heavily obfuscated with complex control flow and encrypted data structures. This case demonstrates how agents reduce analysis time from weeks of manual work to hours of agent-assisted investigation.

Initial Hypothesis: Full-Featured Surveillance RAT

The agent identified and validated:

1. Complete command dispatch vtable at 0x1003d7d0 containing 10 remotely-callable commands

2. Orchestrator vtable at 0x1003d7f8 implementing C2 data loop, host fingerprinting, and task scheduling

3. Persistence mechanisms via registry RunKeys and scheduled tasks

4. Information gathering: file enumeration, process listing, screen capture

Without agentic assistance, these vtables would require days of manual cross-reference resolution and pattern matching.

Secondary Hypothesis: C2 with AES-128 Encryption

The agent mapped the protocol layer:

1. Protocol vtable at 0x1003d330: encrypt+send, recv+decrypt, raw I/O via WinSock

2. Encryption: function sub_10012130 (AES-128-ECB)

3. Decryption: function sub_10013260

4. Key expansion: function sub_10014410

5. Wire format: C3PO-r2d2-POE magic → 4-byte length field → AES-128-ECB ciphertext payload

The agent discovered and validated the encryption protocol in one focused task, then generated a decryption PoC, extracting actual C2 traffic patterns and hardcoded server addresses.

This case exemplifies how agentic analysis handles sophisticated malware: the agent automates pattern discovery and structural analysis, while the analyst provides intuition, validation, and tactical decisions about where to dig deeper.

Limitations: Non-Determinism, Context Windows, and Control

Agentic malware analysis is powerful but faces real constraints. The analyst must remain in the loop.

Non-Deterministic Reruns

The same analysis executed twice may yield different results due to model temperature and sampling. This is especially problematic for reproducibility in incident response or legal discovery. Mitigation: use low temperature (0.0–0.3) for consistency, set a random seed when possible, and require the agent to validate critical findings with a second tool pass.

Context Window Limits

Modern LLMs have large context windows (100K+ tokens), but large binaries, multi-stage analysis, and verbose tool output consume context quickly. When an agent reaches the window limit, it loses prior reasoning and must restart. Mitigation: use a case directory on disk as persistent state. Have the agent checkpoint hypotheses, artifacts, and intermediate results regularly, then summarize before pushing forward. This transforms the analysis into resumable, compartmentalized steps.

Validation Still Required

LLMs can confidently produce incorrect analyses, especially with heavily obfuscated code. Agents hallucinate less when given dedicated tools and structured workflows (skills), but they are not infallible. The analyst must review findings, cross-check IOCs, and validate conclusions before acting.

Analytical Depth and Human Expertise

Agents excel at automating routine tasks—string decryption, API hashing, initial triage. But novel obfuscation techniques, sophisticated evasion, and attribution require human intuition and domain knowledge. Agents are force multipliers, not replacements. The best workflow has agents handle the initial 70–80% of analysis, surfacing patterns and hypotheses that human experts then validate, refine, and extend.

Key Takeaways

Agentic AI transforms malware analysis economics by automating repetitive tasks and compressing analysis timelines:

Meet the Expert

Tim Blazytko

PhD in Binary Program Analysis | Independent reverse engineering expert specializing in malware analysis, deobfuscation, and agentic automation. Founder of synthesis.to.

Open-source projects: Ghidra Headless MCP, Binary Ninja MCP, Agentic Malware Analysis Pipeline

Contact: tim@blazytko.to | Twitter: @mr_phrazer