Malware Analyzer Performance - VMRay

Malware Analyzer Performance

Jul 31st 2014

Measuring performance of a malware analysis system in “samples per hour” is misleading and does not in any way give an accurate representation of it effectiveness. However, this measure is often used to conceal the significant overhead of such solutions. In practice, the throughput of “samples per hour” almost completely depends on the configured timeout value and the utilized hardware, rather than the analyzer:
“If you set the same timeout for different analysis systems, then the throughput of all systems will (almost) be the same although the result quality varies greatly.”
What really matters in the end is how much (and of course, what) information can be gathered until the timeout is reached. Therefore, one needs to measure the computation overhead of an analyzer compared to native execution. In order to do this, we have created a benchmarking test that simulates typical malware behavior and executed it with identical timeout values inside the following different analyzers:
a)       VMRay Analyzer
b)       One famous malware analyzer that utilizes the hooking technology
c)       QEMU, the fastest emulator on the market that is the basis of many malware analyzer products
For each environment, we measured how many iterations of the benchmarking algorithm were executed until the system timed out. Figure 1 shows that our technology clearly outperforms all other approaches. We are about twice as fast as the hooking solution and several magnitudes faster than the Emulator, even without any additional analyzer on top of it that would clearly impose additional performance degradation.

 

 

In a second test, we additionally took into account the necessary computation time for post-processing of the generated analysis data, because this obviously also influences the performance and throughput of a system. Since we were testing the pure emulator, QEMU, and not a real emulation-based analyzer, we had to assume a post-processing time for this approach. To be fair, we had chosen a very optimistic time, one that was much shorter than those of VMRay and hooking approach. In fact, this is rather unrealistic and even varnishes the emulation results. Figure 2 shows the benchmarking index that combines the results from the first test with the time penalty of the post-processing. Again, our solution is far better than the others.

 

 

The performance advantage of VMRay Analyzer directly stems from the fact that everything is executed on bare metal hardware and without the use of any emulation. Furthermore, our unique transition monitoring technology focuses solely on the relevant malware operations and ignores most irrelevant events that occur during hooking-based analysis.
Another important aspect for an analysis system’s performance is its parallelizability. As VMRay Analyzer is based on the well-established hypervisor KVM, it can very easily be parallelized. For the time being, we recommend that you not run more VMs than CPU cores in the system. However, we are currently implementing a feature that reduces the CPU usage to zero during system startup and, hence, enables you to run up to three times more VMs than cores.

Calculate how much malware false positives are costing your organization:
Malware False Positive Cost Calculator