7 steps in a repeatable reverse engineering workflow
7 static indicators to check before a sample runs
6 behavioral signals to capture during detonation
Malware, malicious software built to evade detection and hide its purpose, can turn into an open-ended reverse engineering slog when it resists a quick sandbox or antivirus verdict.
This guide breaks malware reverse engineering into seven steps, from defining the research goal through unpacking and converting findings into detections. Work through the steps in order for a repeatable workflow covering ransomware, trojans, and rootkits.
Figure 1: The reverse engineering loop
Step 1: Define the Reverse Engineering Goal
Before opening a disassembler or debugger, pin down exactly what the investigation needs answered. Chasing a full understanding of an entire binary burns time incident response can’t spare.
The objective might be recovering a malware family’s configuration, mapping a persistence mechanism, or decoding the vulnerabilities and C2 logic behind an unexplained execution chain.
Threat intelligence, hash reputation, and prior sandbox output frequently answer the question outright. Escalate to manual reverse engineering only when triage leaves a genuine gap, since the wrong assumption here carries real risk.
Step 2: Build a Safe, Isolated Analysis Environment
A lab comes before any sample gets touched. Run it inside virtual machines on VirtualBox, VMware, or Hyper-V, using host-only networking so the guest has no path to the real internet or your production system. Take a clean snapshot right after setup so every run starts fresh.
Fake the internet rather than exposing a sample to it directly. Tools like INetSim or a Remnux gateway VM, a Linux distribution built for this purpose, generate DNS and HTTP responses so malware can “phone home” without reaching a live server. Some samples still need limited connectivity to detonate fully; treat that as a case-by-case call.
Figure 2: Isolated analysis lab
Step 3: Triage the Sample With Static Analysis
Examine the file before it ever runs. A quick pass through these indicators shows where to focus deeper analysis:
- File hash and reputation. Check whether the sample is already known.
- PE headers. Review Windows executable metadata and entry points.
- Imports. Flag API calls tied to networking, persistence, or injection.
- Strings. Pull readable URLs, commands, and registry paths.
- Entropy. Watch for random sections suggesting packing or encryption.
- Embedded resources. Inspect bundled files or configuration data.
- Obfuscation indicators. Note anything suggesting the malicious code is deliberately hidden.
Packed samples show few readable strings alongside high entropy, an early sign that unpacking (Step 6) will be necessary.
Static analysis narrows down where to look next, not where to stop. Treat every triage finding as a lead for the dynamic and code-level steps that follow.
Step 4: Detonate the Sample and Observe Its Behavior
Running the sample and watching what it does, sometimes called behavioral analysis, fills in what static analysis can’t. Capture network traffic alongside execution with Wireshark or the sandbox’s built-in capture, since C2 communication is easiest to catch as it happens. Watch for:
- Process activity. Track processes the malware creates or injects into.
- Registry changes. Identify registry modifications tied to persistence.
- File-system activity. Monitor files the sample creates or drops.
- Network connections. Capture communication with domains or C2 infrastructure.
- Dropped payloads. Identify executables or scripts written to disk.
- Memory activity. Look for injected code or decrypted data in memory.
Modern malware checks for VM artifacts or debugger presence, then triggers the attacker’s evasion logic. VMRay DeepResponse, an evasion-resistant sandbox, still captures genuine behavior, as VMRay’s “Deleting the Defenders” research shows.
Step 5: Disassemble and Debug the Code
Translating machine code into something readable, through disassembly and debugging, is where reverse engineering earns its name. A disassembler and debugger do most of the heavy lifting from here on.
Read the Assembly Code
IDA Free and Ghidra let you jump to strings and lean on a decompiler to read pseudocode instead of raw assembly code when time is short. Start from a suspicious string or import flagged during triage instead of reading the binary top down.
Watch the Logic Run
Set breakpoints on flagged API calls, like InternetConnectA, to inspect live parameters as the sample runs. Patch out anti-debugging checks such as IsDebuggerPresent so the sample completes instead of terminating itself mid-analysis.
Step 6: Unpack and Deobfuscate the Sample
Packing and obfuscation hide a sample’s real code or configuration, regardless of the programming language behind it. Confirming a packer is present comes first; stripping it away comes second.
Confirm and Strip the Packer
Entropy signals from Step 3 point toward a packer; Detect It Easy or PeStudio confirm it. Once the sample is running, dump the unpacked code from memory and rebuild the Import Address Table with a tool like Scylla.
Handle Script-Based Obfuscation
Obfuscated scripts in JavaScript, PowerShell, or AutoIt are usually easier to deobfuscate than to reverse blind; loading the file in its native interpreter and breaking on execution beats manual decoding.
Figure 3: Unpacking sequence
Once unpacked, a sample goes back through Step 5, since the workflow loops rather than runs straight through. DarkGate is a good illustration of why that loop matters: VMRay’s research on the family found it shipping as an obfuscated AutoIt script that only reveals its shellcode after deobfuscation.
Step 7: Turn Findings Into Detection Rules and Documentation
Malware analysis that stays in one analyst’s head doesn’t help the next investigation. Converting it into something reusable makes the work worth the time it took.
Figure 4: Config decrypted in memory
- YARA rules. Turn distinctive strings or byte sequences into reusable detections.
- Indicators of Compromise. Record hashes, domains, and registry keys.
- C2 infrastructure. Document servers and communication patterns.
- Malware configurations. Extract campaign IDs, encryption keys, and mutexes.
- MITRE ATT&CK mappings. Map confirmed behavior to known tactics.
- Behavioral detections. Turn execution patterns into detections that outlast indicators.
- Threat intelligence. Share validated findings through platforms such as MISP.
Build a Repeatable Reverse Engineering Workflow
These seven steps work as a loop more than a checklist. A scoped goal from Step 1 shapes what triage in Step 3 looks for, unpacking in Step 6 sends the sample back through disassembly in Step 5, and a newly decoded configuration can raise a fresh question that restarts the loop at Step 1. That looping structure is what makes the workflow repeatable across ransomware, trojans, and rootkits.
VMRay DeepResponse handles the detonation and evasion-resistant capture that loop depends on, so a sample’s anti-VM and anti-debugger checks don’t just produce a clean, useless report.
That leaves analysts free to spend their time on the one step no tool can automate: deciding what the findings actually mean.
Start Automating Your Malware Detonation
Let evasion-resistant detonation handle Step 4, so anti-VM and anti-debugger checks reveal behavior instead of hiding it. Your analysts keep their time for disassembly, unpacking, and deciding what the findings mean. Explore VMRay DeepResponse to see how it fits your workflow.
Try VMRay Explore VMRay DeepResponse
Frequently Asked Questions
Is it legal to reverse engineer malware?
It’s standard practice in defensive security research and incident response, though legality varies by jurisdiction and how the sample was obtained. Organizations should also make sure their analysis follows applicable licensing, privacy, and evidence-handling requirements.
How long does malware reverse engineering take?
Malware reverse engineering for basic triage can take minutes, while heavily packed malware can take days. The timeline depends on factors such as obfuscation, packing, anti-analysis techniques, and how deeply the malware needs to be investigated.
When is malware reverse engineering necessary?
Malware reverse engineering is useful when basic analysis does not reveal enough about how a sample works, what it targets, or how it avoids detection. It can help analysts uncover hidden capabilities, persistence methods, command-and-control behavior, and other details needed to improve detection techniques.