Stepping into the Breach: Improving Security Researchers’ Ability to Dynamically Analyze macOS Malware at Scale

Aug 01st 2019

This content covered in the blog is based on my Objective By the Sea talk “Hypervisor-Based Analysis of macOS Malware”. You can access the slides from my presentation here.


A Growing Threat and a Dearth of Tools

Compared to Windows, macOS accounts for only a small percentage of all malware, but the threat is growing. Malwarebytes reported that the volume of Mac-specific malware grew 62% in Q4 2018, and new attack methods continue to emerge. Although threats against consumers are declining, attacks against infrastructure and businesses are on the rise.

Analysts looking for tools that can analyze and detect macOS malware have few options compared to their counterparts in the Windows world, which has been fighting malware for decades. Disparate tools exist to monitor aspects of macOS—process creation and persistence, file creation, system calls, etc.—but there are big gaps:

  • There is no tool to generically trace programs at the function call level.
  • Because existing tools need to run inside the analysis environment, they’re easily detected by malware.
  • The current fragmented approach thwarts automation and needs an unacceptably high level of manual intervention.

Furthermore, the few dynamic malware analysis tools for macOS can’t meet escalating requirements for scalability and performance. Faced with these constraints, even some tech giants rely (unhappily) on home-brewed solutions for internally analyzing and detecting macOS malware.


OBTS Slides - Limitations of Current macOS Malware Analysis Tools

Figure 1: Old Limitations, New Imperatives


VMRay Analyzer Adds macOS Analysis and Detection

VMRay is stepping into the breach. With VMRay Analyzer Version 3.0 and recent enhancements in Version 3.1, we added macOS dynamic analysis and detection to our well-established platform. Leveraging deep expertise and years of experience protecting Windows environments, Version 3.1 addresses the unique challenges of macOS, enabling researchers and DFIR teams to dynamically analyze and detect malware at scale.

Key features include:

  • Automated submission of macOS executables, app bundles and Java files for analysis via the UI or a REST API
  • Detailed function logs, with full visibility into the behavior of macOS malware
  • A severity verdict (with a score of 0 to 100), based on a malware sample’s behavior
  • Detection of macOS-specific sandbox evasion techniques and persistence mechanisms
  • Detailed network analysis Information
  • YARA rule matching


Now, Near, Deep

The VMRay sandbox is the foundation of our Now, Near Deep architecture (Figure 2), which submits suspected macOS malware files to multiple levels of scrutiny:

  • A rapid reputation engine determines if a file is benign, malicious, or unknown.
  • Static analysis extracts potentially harmful active elements (macros, embedded URLs)
  • Dynamic behavioral analysis fully examines malware activity to determine the severity of a threat.


VMRay's Now, Near, Deep Architecture
Figure 2: VMRay’s Now, Near, Deep Architecture


Immediate Visibility and Insight into Malware Behavior

The sandbox provides an isolated environment where macOS malware is allowed to execute, its behavior is monitored and it is assigned a severity score which enables other tools in the security ecosystem make block/allow decisions about suspect behavior. The sandbox — which uses an agentless, hypervisor-based approach — remains undetectable throughout the analysis, a trait that prevents malware from evading discovery.

As Figures 3 to 6 illustrate below, VMRay’s automated, analyst-friendly reporting tools provide immediate insight into malware behavior, with easy drill-down to gain further understanding. The visuals below are excerpted from a real-world analysis of ColdRoot, a remote access Trojan that targets macOS.

The process graph in Figure 3 presents a high-level view of the commands ColdRoot executed to create files, run scripts, escalate privileges, and so on. As you can see, the malware disguised itself as an Apple service to gain persistence and thereafter enabled keylogging and remote desktop access.


Process Graph ColdRoot
Figure 3: Process graphs provide a high-level view of malware behavior


Digging deeper, the function log generated by the VMRay Analyzer (Figure 4) reveals which API calls were responsible for some of the behavior depicted in the process graph. AuthorizationExecuteWithPrivileges() was used to enable enhanced system access via the Accessibility API and install a persistent Launch Daemon with root privileges.

// enable Accessibility API
[0050.862] AuthorizationExecuteWithPrivileges (authorization=0x539eb0, pathToTool="/usr/bin/touch", options=0x0, arguments=([0]="/private/var/db/.AccessibilityAPIEnabled"),
communicationsPipe=0x0) returned 0

// run script which will add ColdRoot to the Accessibility DB
[0051.460] AuthorizationExecuteWithPrivileges (authorization=0x539eb0, pathToTool="/bin/sh", options=0x0, arguments=([0]="/private/var/tmp/"), communicationsPipe=0x0) returned 0

// copy LaunchDaemon definition [0052.844] AuthorizationExecuteWithPrivileges (authorization=0x539eb0, pathToTool="/bin/cp", options=0x0, arguments=([0]="/Users/jdoe/Downloads/", [1]="/Library/LaunchDaemons"), communicationsPipe=0x0) returned 0


// load and start LaunchDaemon
[0053.218] AuthorizationExecuteWithPrivileges (authorization=0x539eb0, pathToTool="/bin/launchctl", options=0x0,
arguments=([0]="load", [1]="/Library/LaunchDaemons/"), communicationsPipe=0x0) returned 0
[0053.683] AuthorizationExecuteWithPrivileges (authorization=0x539eb0, pathToTool="/bin/launchctl", options=0x0,
arguments=([0]="start", [1]="/Library/LaunchDaemons/"), communicationsPipe=0x0) returned 0

Figure 4: Function logs add detail to the picture (comments added)

The next two analysis logs show the function calls ColdRoot made: to activate keylogging by installing an SLEventTap (Figure 5) and enabling remote desktop access using SLDisplayCreateImage() (Figure 6).

// install event tap (SL == SkyLight == CoreGraphics)
[0034.621] SLEventTapCreate (tap=0x1, place=0x0, options=0x0, eventsOfInterest=0x1c00,
                              callback=0x6a3d0, userInfo=0x0) returned 0x509d50
[0034.805] CFMachPortCreateRunLoopSource (allocator=0x0, port=0x509d50, order=0) returned 0x50ff20
[0034.805] CFRunLoopGetCurrent () returned 0x5123c0
[0034.806] CFRunLoopAddSource (rl=0x5123c0, source=0x50ff20, mode="kCFRunLoopCommonModes") 
[0034.807] SLEventTapEnable (tap=0x509d50, enable=1) 
[0034.807] CFRunLoopRun () 

// on keypress: get keycode
[0088.346] SLEventGetIntegerValueField (event=0x53a580, field=0x9) returned 36
[0088.346] SLEventKeyboardGetUnicodeString (event=0x53a580, maxStringLength=0xa, 
                                            actualStringLength=0xb0579d48, unicodeString=0xb0579d4e) 

// write to log
[0088.349] open (path="/private/var/tmp/adobe_logs.log", oflag=9) returned 3
[0088.350] __ioctl (fildes=3, request=0x402c7413) returned -1
[0088.350] bcopy (src=0x31b704c, dst=0xb0579bc0, len=0xa) 
[0088.350] __write_nocancel (fildes=3, buf=0xb0579bc0*, nbyte=0xa) returned 10
[0088.350] __close_nocancel (fildes=3) returned 0

Figure 5: Function calls for keylogging (comments added)
In the case of remote access, below, time stamps show that ColdRoot first opened a secondary TCP connection to the C2 server, then took a screenshot every half-second and sent those to the C2 server via the newly created socket. Note that a rather large buffer is passed to the second send() call (“length=0x3beec”), suggesting that it might be an image–which is confirmed by looking at the sent data (below).

// take screenshot using SkyLight (aka CoreGraphics)
[0038.037] SLMainDisplayID () returned 0x5b81c5c0
[0038.042] SLDisplayCreateImage (displayID=0x5b81c5c0) returned 0x53c800
[0038.155] CGImageGetHeight (image=0x53c800) returned 0x360
[0038.155] CGImageGetWidth (image=0x53c800) returned 0x480

// send to C2
[0037.851] socket (domain=2, type=1, protocol=0) returned 4
[0037.857] connect (sockfd=4, addr=0xb1189df0*(sin_len=0x10, sin_family=0x2,
sin_port=0x3419, sin_addr=””), addrlen=0x10) returned 0
[0040.638] send (socket=4, buffer=0x320f028*, length=0x4, flags=0) returned 4
[0040.640] send (socket=4, buffer=0x35a2d18*, length=0x3beec, flags=0) returned 245484

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 00 00 01  |......JFIF......|
00000010  00 01 00 00 ff db 00 43  00 01 01 01 01 01 01 01  |.......C........|
00000020  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|

Figure 6: Function calls for remote desktop access (comments added)


Making Dynamic Analysis Comprehensive Yet Efficient

Dynamic malware analysis is challenging in many ways, and VMRay is one of the rare providers who has mastered those challenges. Underlying the macOS-specific capabilities of VMRay Analyzer are three core traits that are OS-independent:

  • Full visibility of function calls at every level of the OS, to eliminate gaps in
  • Isolation and transparency to ensure the analysis process cannot be detected (and evaded) by malware
  • Efficiency and automation to handle macOS malware threats at scale while maximizing the effectiveness of DFIR teams.

Full visibility of function calls

With the different levels of abstraction provided by system APIs, malware can interact with the OS at any level (see Figure 7). A dynamic analysis sandbox needs to monitor all potential sources of misbehavior, from direct syscalls to high-level API calls. Otherwise, it will miss detecting malware that may cause significant harm.


Full Visibility of Functional Calls

Figure 7: Monitoring malware behavior on every level of the OS


VMRay stands apart from other most other dynamic analysis platforms in providing that full visibility. Function calls are automatically intercepted at the highest abstraction level possible. By preserving high-level semantics, VMRay simplifies behavior analysis. Furthermore, minimizing the number of calls being intercepted improves efficiency.

Isolation and transparency

For dynamic analysis to be bulletproof, malware samples need to run in an isolated environment, allowing potentially malicious behavior to be monitored and executed, without impacting production systems or corrupting the sandbox itself. In addition, the analysis process should be undetectable to the malware, which might otherwise hide its behavior to thwart scrutiny and insight.

To meet both goals and still maintain full visibility, the analysis process needs to run on a higher privilege level than any of the malware samples being analyzed. VMRay achieves this by placing the sandbox in the hypervisor layer (see Figure 8). This means that no agents or other telltale modifications to the operating system are necessary, and malware cannot detect that it is being analyzed, even from the kernel.


Analysis System Runs in the Hypervisor

Figure 8: The analysis system runs in the hypervisor layer


Efficiency and automation

Efficiency is achieved by supplanting several disparate tools with a single sandbox that delivers more comprehensive results in much less time. Because VMRay Analyzer can be controlled using a REST API, it can easily be integrated into existing environments to automate workflows. The fact that you don’t need a specialist to analyze the vast majority of files — they’re just thrown into the sandbox, which returns a verdict — is a major improvement in macOS analysis and detection.


Unique Contributions by VMRay

In developing and evolving VMRay Analyzer, our team has made significant contributions to achieving isolation and transparency. VMRay co-founders Carsten Willems and Ralf Hund developed a breakthrough approach, based on two-dimensional paging, that allows VMRay to intercept the execution of malware at the function call level while maintaining efficiency and without the need of altering the operating system or revealing the existence of the sandbox.

In addition, by using Virtual Machine Introspection (VMI), VMRay can monitor the activity of the target machine at every level, entirely from the outside (see Figure 9). VMRay runs as part of the hypervisor on top of the host OS, which in turn is running on bare metal. Each component is independent, so VMRay can run on different hypervisor and hardware combinations.


Virtual Machine Introspection

Figure 9: VMI enables transition monitoring


VMRay’s monitoring approach can be applied to every OS and every architecture. However, each operating system has some unique features that require special handling to capture each aspect of program execution. Among the macOS-specific challenges that VMRay successfully addresses is the monitoring of Objective-C calls, Inter-Process Communication (IPC) using XPC and low-level Mach messages.

As Figure 10 illustrates, we implemented support for the Objective-C programming language so that the function log closely resembles the source code, making it more easily readable by analysts.



NSLog(@"Hello, World!");

NSProcessInfo *processInfo = [NSProcessInfo processInfo];
NSLog(@”Process ID is: %d”, [processInfo processIdentifier]);

NSString *username = [processInfo userName];

NSFileManager *filemgr = [NSFileManager defaultManager];
NSString *filename = [[filemgr currentDirectoryPath]

[username writeToFile:filename

NSLog(@”Content written to path: %@\n”, filename);

Analysis Log

[0045.565] NSLog (format="Hello, World!")

[0045.706] +[NSProcessInfo processInfo] returned 0x7f9a3740d080
[0045.706] -[NSProcessInfo<0x7f9a3740d080> processIdentifier] returned 488
[0045.706] NSLog (format=“Process ID is: %d”)

[0045.706] -[NSProcessInfo<0x7f9a3740d080> userName] returned=“xsbgsz”

[0045.824] +[NSFileManager defaultManager] returned 0x7f9a37402850
[0045.824] -[NSFileManager<0x7f9a37402850> currentDirectoryPath]

[0045.916] -[NSString<0x7f9a3740d150> stringByAppendingPathComponent:“user.txt”]


[0045.916] -[NSString<0x7a736762737865> writeToFile:“/Users/xsbgsz/user.txt”
atomically:1 encoding:0x1 error:0x0]

returned 1

[0045.923] NSLog (format=“Content written to path: %@\n”)

Figure 10: Function logs that simplify the analyst’s job

As shown in Figure 11 below, XPC defines a powerful way for processes to exchange data which is heavily used across the operating system for various tasks. However, XPC is also used by malicious software, for example, to establish persistence or spawn processes outside the current context. VMRay is unique in that it intercepts and monitors every low-level message and thereby detects all malicious behavior. In contrast, sandboxes that do an incomplete monitoring job will invariably miss critical behavior.

IPC Messages
Figure 11: Monitoring IPC messages to catch evasive malware


Summing it All Up

In VMRay Analyzer v3.0, we introduced comprehensive analysis of macOS malware, enabling security teams to better secure heterogeneous IT operating environments. With Version 3.1’s recent addition of severity scores and support for JAVA files, VMRay now offers the same out-of-the-box capabilities for macOS as for Windows.

For the first time, security teams benefit from full visibility into macOS malware behavior and the means to defeat evasion-resistant macOS malware strains. Furthermore, efficiency and automation features empower security teams to handle larger analysis volumes, speed up detection and improve the productivity and efficiency of security personnel and infrastructure.