The scenarios for the DFRWS 2011 Forensics Challenge were two seemingly unrelated crimes that turned out to be tightly linked with each other. The first scenario was a suspicious death and the goal of the investigation was to determine whether the victim killed himself or was murdered. The second scenario was an intellectual property theft case and the goal of the investigation was to document any evidence that intellectual property was stolen and to support termination of the suspected insider.

An interesting outcome of the challenge was that using dd to acquire data from the Android device in Scenario 1 did not copy the important information in out-of-band (OOB) areas of the YAFFS2 file system. As a result, it was not possible to reconstruct the file system. However, contestants were still able to carve out usable content from this data.

The winning submission provides a technical analysis of data structures found in memory dump from Android mobile devices and provides an Android analysis toolkit that extracts specific items and formats them in a report. Using this toolkit to perform a forensic examination of a full NAND dump of a YAFFS2 file system (such as in Scenario 2 of the DFRWS 2011 Forensics Challenge) first requires the file system to be mounted under Linux as an emulated Flash device (using nandsim).

A sample of the information extracted by the winners from the SQLite database located on the Android device in Scenario 2 (mtd8\data\com.android.providers.telephony\databases\mmssms.db) is provided here:

Much more information was extracted from both Android devices as detailed in the reports, which include an impressive graphical reconstruction of events.

A health care provider for Alaska would monitor its network connections to ensure that network connections are limited to its main source of users, i.e. those in Alaska. An insurance company in St. Louis will see mostly traffic from IP addresses in Missouri, but Illinois as well, due to the city  being on the state line.   Occasionally, administrators may notice connections being made from Hawaii, Bermuda, or Italy, signifying users who are on vacation but are still wired in to their work. However, a long-term series of connections from a Eircom subscriber, Ireland’s largest ISP, should spark interest to the network administrator of a Seattle tax firm.

While anonymous web connections from global addresses are common, specific attention should be paid to such addresses being used to access password-protected areas of a corporation. This could include remote file access, VPN and web-based corporate email.

In such cases the logs from these applications, usually supplied in plain text or W3C format, contain details about transactions to include the remote IP address and the account name being authorized. In reviewing logs from various incident responses cmdLabs has found details to show that a short log review made on a daily basis could help smaller corporations determine quickly if a user account was compromised and accessed from a remote location.

For example, the log sample below from a Cisco ASA tracks VPN connections. The user “cmdLabs\bbaskin” was accessed via the IP address of 159.134.100.100 on 2 April, 2011, an IP that was traced back to Ireland. A few hours later the same account was accessed from an IP address in Austria.

Apr  2 21:53:37 192.168.1.1 Apr 02 2011 21: 53:08: %ASA-6-302013: Built outbound TCP connection 7823 for inside:10.10.10.50/389 (10.10.10.50/389) to NP Identity Ifc:192.168.1.1/1047 (192.168.1.1/1047)
Apr  2 21:53:37 192.168.1.1 Apr 02 2011 21: 53:08: %ASA-6-1
04: AAA user authentication Successful : server =  10.10.10.50 : user = cmdLabs\bbaskin
Apr  2 21:53:37 192.168.1.1 Apr 02 2011 21: 53:08: %ASA-6-113009: AAA retrieved default group policy (DfltGrpPolicy) for user = cmdLabs\bbaskin
Apr  2 21:53:37 192.168.1.1 Apr 02 2011 21: 53:08: %ASA-6-113008: AAA transaction status ACCEPT : user = cmdLabs\bbaskin
Apr  2 21:53:37 192.168.1.1 Apr 02 2011 21: 53:08: %ASA-6-734001: DAP: User cmdLabs\bbaskin, Addr 159.134.100.100, Connection Clientless: The following DAP records were selected for this connection: DfltAccessPolicy

For this small set of data it is trivial to query each IP address to determine its country of origin, netblock owner, and other details that would highlight unauthorized access. The problem arises when you have hundreds of thousands of such transactions in your daily log files.

One service that cmdLabs uses regularly is the IP to ASN WHOIS server run by Team Cymru. This server provides quick and easy access to country codes for a given IP address. However, it has two limitations: it requires Internet-access which is not readily available from a forensic workstation and to process a large bulk of IPs you have to use their Netcat process which only returns ASNs and not country codes.  To overcome these limitations I’ve developed a simple solution that could process hundreds of thousands of IP addresses to determine country codes.

This solution is a small Python script called IP2CC that takes an IP address as input and outputs the originating country code for that IP.  This solution requires three components:

1. The free country code database located at http://www.maxmind.com/app/geolitecountry (updated monthly)
2. Python API module to access this database located at http://code.google.com/p/pygeoip/
3. The IP2CC.py script. Downloadable at the end of this blog post.

The script allows for input to be given via the command line, stdin, or an input file. In normal use it will simply output the country code. With the –c or -t option the output will contain both the IP and country code in either a comma-separated version (CSV) or tab-separated (TSV) output, respectively.

Python ip2cc.py –i <ip> -f <input file> [-c] [-t]

> python ip2cc.py -i 11.11.11.11
US

> python ip2cc.py -i 22.22.22.22 -c
22.22.22.22,US

> echo 33.33.33.33 | python ip2cc.py
US

> python ip2cc.py -f IP.txt -c
14.48.7.101,AU
12.51.21.19,US
10.61.14.9,Internal


In one use, we’ll eliminate known intranet/extranet IP addresses and run the resulting list through IP2CC to produce a master list of foreign accesses. This script will run in Linux and OSX in conjunction with the native OS command line tools. For a Windows environment you will find additional capabilities by installing the necessary GnuWin32 components. For example, when reviewing a NCSA-formatted log with the IP address in the first field:

D:\> type in051611.log | egrep –v “^192” | gawk “{print $1}” | python ip2cc.py -t | egrep –v “US|Internal” | gawk -F\t "{print $1}" | sort | uniq > DailyForeignIPs.txt
D:\> for /F %i in (DailyForeignIPs.txt) do grep “%i” in051611.log >> DailyForeignConnections.txt

The first command above will save a simple text listing of all unique foreign IP addresses into a file for processing. The second line takes each IP address from that resulting file and compares it back against the logs to extract all lines that include its presence. The resulting DailyForeignConnections.txt can then be quickly reviewed to determine if any accounts were accessed from a foreign IP address.

Dealing with the VPN logs shown earlier, we’ll change our command line a bit.  Using the standard Cisco log file index as a source we can see that the log id of 734001 will show us the remote IP address of a user login. We’ll search the log for that id and then parse out the IP address in the 15th field. An additional hindrance is that the IP address is appended with a comma, which we’ll remove with the ‘tr’ command.

D:\> type asavpn-051611.log | findstr "734001" | gawk “$15 !~ /^192/ {print $15}” | tr -d "," | python ip2cc.py –t | egrep –v “US|Internal” | sort | uniq > DailyVPNForeignIPs.txt

This is ultimately just a very simple Python script. In-house, we use it as a mere function within larger processes, but its simplicity allows for it to be used in a variety of result-tuning processes. Customization is easy. At times I’ll make an offshoot of the script to process input from `uniq` command with the `-c` count option occasionally. The `uniq –c` adds a new column that specifies the total number of instances of that IP address which is useful when evaluating the persistence of a single IP amongst thousands. A few small changes to the Python will allow you to read this count and add it to the CSV output for easy integration into a spreadsheet.

Usage of a tool like IP2CC is a first step to opening an administrators eyes to traffic beyond their network. A good administrator or security engineer should monitor not only the traffic that flows across their network but also the perceived traffic that flows from a network’s outer nodes to the Internet. Monitoring for your company’s existence in spam black-lists, a malware rating on services like Web of Trust, and other indicators can give clues that an infection or intrusion may be underway within your network. We’ll discuss these points, and others, in a future blog post.

Downloads:

IP2CC Python Source Code v1.0

Eoghan Casey - Digital Evidence & Computer Crime, 3rd Edition

This book is divided into five parts, beginning with the fundamental concepts and legal issues relating to digital evidence and computer crime in Part 1 (Digital Forensics: Chapters 1 – 5). Part 2 of this text (Digital Investigations: Chapters 6 – 9) covers investigative aspects of digital evidence and computer crime. Part 3 of this text (Apprehending Offenders: Chapters 10 – 14) deals with specific types of investigations with a focus on apprehending offenders, including Violent Crime in Chapter 10, Sex Offenders on the Internet in Chapter 12 and Investigating Computer Intrusions in Chapter 13. Part 4 of this book (Computer Forensics: Chapters 15 – 20) begins by introducing basic Forensic Science concepts in the context of a single computer, and goes on to apply these concepts in updated chapters dedicated to networked Windows, Unix, and Macintosh computers and mobile devices. Part 5 (Network Forensics: Chapters 21 – 25) covers computer networks from an investigative perspective, focusing specifically on the Internet and performing forensic analysis on network logs and traffic.

This material provides the foundation for the more advanced companion text, the Handbook of Digital Forensics and Investigation.

Many thanks to Susan Brenner, Christopher Daywalt, Monique Mattei Ferraro, Bert-Jaap Koops, Terrance Maguire, Mike McGrath, Tessa Robinson, Bradley Schatz, Ben Turnbull and Brent Turvey for their excellent contributions to this textbook.

File Carving Limitations

Most file carving tools require a known file header in order to salvage deleted data. For instance, to recover a deleted 3gp file, most carving tools look for the file headers such as the following.

Hex view of 3gp header in the Motorola V3 Flash memory dump

If the file is fragmented or the header is missing, the file carving approach will not salvage the deleted video successfully. In this example, a file carving tool that searched the Motorola V3 memory dump for several 3gp header signatures found two files in as shown in the audit log:

05/24/2011, 11:26:35
QuickTime 3GP (3gp), header: ftypisom
QuickTime 3GP (3gp), header: ftyp3gp
QuickTime 3GP (3gp), header: ftypmmp4
Default file size: 1024 KB
Maximum file size: 100 times (individual file type definition defaults sizes respected)

E:\Physical GSM Motorola V3 RAZR\Flex Partition 1140000-1fe0000.bin
Scope: 000000 - E9FFFF
Extensive byte-level search

9D0E80 - AD0E7F: 00001.3gp
B888F0 - C888EF: 00002.3gp

05/24/2011, 11:26:35
2 file headers were found. 2 files were retrieved.

However, the salvaged files were invalid because the original files were fragmented. Furthermore, the names and directory paths of these files were not obtained using this method, demonstrating a further limitation of file carving.

Salvaging Video Fragments

When video files are fragmented, it is necessary to consider the video file format in more detail. Fortunately, many digital video formats have a structure that can be used to find and salvage individual frames. A frame is a discrete section of the video that can have a timecode or sequence number and other characteristics that can be useful for salvaging digital video clips.

The defraser tool can be used to identify frames for several video formats in a forensic duplicate of any piece of storage media, including a removable storage card, computer hard drive and Flash dump from a mobile device. The following screenshot shows defraser used to detect video related data in the Motorola V3 memory dump.

Although the defraser tool does not automatically piece together the frames into a video that can be played, it does make the frames available for manual reconstruction. With some effort, defraser may be used to combine fragmented frames into a valid video file that can be played.

As with file carving methods that rely on header signatures, the carving methods employed by defraser do not provide the filenames and directory path of salvaged video data in the context of the original file system.

File System Reconstruction

Ultimately, the most effective approach to extracting digital video files from acquired digital evidence such as a Flash memory dump from mobile device is to reconstruct the logical arrangement of data. On mobile devices, this logical structure involves the flash abstraction layer and file system. Using mobile device forensic tools such as Cellebrite Physical and XRY, it is possible to reconstruct and review logical file structure of a Flash memory dump as shown below with a 3gp video stored in an MMS related file in the Motorola V3 memory dump. Note that different tools may interpret the logical structure differently and show more files and folders, clearly demonstrating the importance of validating the results of forensic examination tools.

Extracting the MMS file using such a mobile device forensic tool and extracting the video content as discussed in the “Delving into Mobile Device File Systems” blog post results in a 3gp file that can be played using VLC media player.

Examination of Salvaged Video

After salvaging digital video files it is important to review the resulting data closely for potential anomalies. For instance, using MediaInfo to extract metadata from video files shows details related to its creation and format. The following screenshot shows metadata from a 3gp video extracted from the Motorola V3 memory dump, revealing that the embedded date-time stamp was set to an incorrect date.

In addition, reviewing individual frames within a salvaged video file can reveal anomalies such as portions of two unrelated videos being combined into one salvage file. The following screenshot shows frames extracted from a 3gp file using DCCI Video Validator revealing footage from two unrelated video files.

Conclusions

When a video file is fragmented or the header of a video file is overwritten, carving methods that rely on header signatures and contiguous files will not salvage video files successfully and may even incorrectly combine unrelated video fragments into a single file or fail to detect the presence of video content altogether. However, using specialized tools such as defraser, a digital investigator may be able to salvage fragments of video files and piece them together into a valid video file. This process of reconstructing video fragments is time consuming and error prone, particularly when dealing with numerous video files on a single piece of storage media or mobile device. Therefore, whenever feasible, it is preferable to reconstruct the logical arrangement of data to extract the complete content of video files. Whichever method is most effective for salvaging digital video, it is important to examine the results closely to ensure the accuracy and completeness of the resulting videos. Such a review includes inspecting embedded metadata for anomalies and reviewing keyframes for possible fragments of unrelated video footage.

When only a portion of the disk space that was reserved for a new file is used to store data associated with that file, this leaves a discrepancy between the logical file size and the actual amount of data stored in the file. As a result, you can have a file that appears to have a logical size larger than the actual amount of data stored for that file. The space between the end of valid data and the end of file is called uninitialized space.

“In NTFS, there are two important concepts of file length: the End of File (EOF) marker and the Valid Data Length (VDL). The EOF indicates the actual length of the file. The VDL identifies the length of valid data on disk. Any reads between VDL and EOF automatically return 0 in order to preserve the C2 object reuse requirement.” (Microsoft fsutil documentation)

Uninitialized space is similar in concept to file slack except that it is contained within the logical file size. Unlike file slack which is no longer associated with a file, data in uninitialized space is in a kind of limbo, trapped at the end of an allocated file but not actually part of that file.

The effect of file initialization behaviors are most easily demonstrated on Windows XP with fsutil as shown here. First, we create a new file that can contain 1024 bytes:


C:\Test>fsutil file createnew cmdLabs-setvaliddata 1024
File C:\Test\cmdLabs-setvaliddata is created


Then we set the valid data length of the new file to 1000 bytes, which leaves 24 bytes unused at the end of the file.

C:\Test>fsutil file setvaliddata cmdLabs-setvaliddata 1000

Valid data length is changed

NTFS captures the difference between logical file size and valid data length in two MFT fields as shown here:

Salvaging Data from File System Limbo
The significance of this from a forensic analysis standpoint is that a file with a valid data length smaller than the logical file size can contain data associated with two files: data associated with the new file (VDL bytes), and data from the old file in uninitialized space (logical file size – VDL bytes).

From a forensic analysis perspective, this uninitialized space can be beneficial. While various disk cleaning utilities can be configured to wipe file slack, they generally do not touch data in uninitialized space. As a result, deleted data can remain in uninitialized space indefinitely, even despite data destruction efforts, and can be salvaged by forensic analysts.

However, this arrangement of data can create complications for forensic analysts, particularly when dealing with larger files that have substantial amounts of uninitialized space. For instance, when carving for certain file types, it is common to export unallocated space. However, any data in uninitialized space will not be included in unallocated space. Similarly, when performing keyword searches, a forensic analyst could incorrectly attribute a hit in the uninitialized space with the new file.

In one case, several approaches were employed in an effort to salvage video fragments:

• examined deleted video files still referenced by file system
• performed file carving on unallocated space only
• processed file slack only for fragments of video files

None of these approaches recovered videos from a time period of interest. It was not until we conducted a forensic analysis of uninitialized space that additional video fragment were found.

Misinterpreting Normal File System Behavior as Backdating

Another complication from a forensic analysis standpoint arises when the file creation process is interrupted before the contents of the file is written to disk, because the new file system entry will point to a cluster that still contains data associated with an older file. When this occurs and a date can be associated with the older file, forensic analysts might think that a newer file was overwritten by an older one. This phenomenon can be misinterpreted as evidence of backdating.

As an example, consider a newly created file that has not been initialized and has not had any associated data saved to disk as shown here using fsutil:

C:\Test>fsutil file createnew cmdLabs-creatnew 1024
File C:\Test\cmdLabs-creatnew is created



When a file is initialized but the associated contents was not written to disk, the initialized file system entry may point to a cluster that contains old data as shown below using EnCase. By default, EnCase shows uninitialized space in blue text. The cluster that was allocated to the new file “cmdLabs-createnew” contains older data (folder entries of files from earlier in January).




This situation can be misinterpreted as backdating if the forensic analyst assumes that the clock had to be set back to the old date when the file contents was saved to disk.

My deepest thanks to the contributors: Cory Altheide (Mandiant) – Christopher Daywalt (cmdLabs) – Andrea de Donno (Lepta) – Dario Forte (DFLabs) – James Holley (Ernst & Young) – Andy Johnson (University of Maryland, Baltimore County) – Ronald van der Knijff (Netherlands Forensic Institute) – Anthony Kokocinski (CSC) – Paul Luehr (Stroz Friedberg) – Terrance Maguire (cmdLabs) – Ryan Pittman (US Army) – Curtis Rose (Curtis W. Rose & Associates) – Joseph Schwerha (TraceEvidence) – Dave Shaver (US Army) – Jessica Reust Smith (Stroz Friedberg).

Backup files from an iPhone or iPod Touch provide an excellent example of SQLite databases that digital forensic examiners can exploit with relative ease, provided they are not encrypted. Data backed up from an iPhone using iTunes such as call logs, contacts, multimedia, and other files are, by default, stored in SQLite database files under “~/Library/Application/Support/MobileSync/Backup” Mac. On Windows XP these backup files are stored in the user’s profile under “C:\Documents and Settings\[userprofile]\Application Data\Apple Computer\MobileSync\Backup” and Windows Vista has a “Roaming” subfolder in this path.

SQLite databases can be examined using a command line tool like sqlite3.exe (http://www.sqlite.org/) or with a GUI tool like SQLite Database Browser (http://sqlitebrowser.sourceforge.net/) shown here with the call log backed up from an iPhone.

The dates are in Unix string format and can be converted using Perl as shown here:

$ perl -e "print scalar(gmtime(1247848584))"
Fri Jul 17 16:36:24 2009

The use of SQLite databases gives forensic practitioners the ability to query the available data directly using the SQL database language. Although a full treatment of SQL is beyond the scope of this discussion, simple examples are provided here to get you started.

C:\>sqlite3.exe E:\iPhoneBackup\call_history.db
SQLite version 3.6.16
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
_SqliteDatabaseProperties call
sqlite> select * from call WHERE address like '%868%';
2|+186835xxxxx|1247848584|60|4|-1
3|+186835xxxxx|1247853361|0|5|-1
4|+186835xxxxx|1247854453|0|5|-1
9|+186831xxxxx|1247895923|60|4|-1
10|+186835xxxxx|1247936960|60|5|-1
11|+186835xxxxx|1247941792|0|4|-1
12|+186835xxxxx|1247941827|0|4|-1
13|+186835xxxxx|1247941920|0|4|-1
14|+186835xxxxx|1247942844|0|4|-1
16|+186835xxxxx|1248015352|60|4|-1
17|+186835xxxxx|1248015674|0|4|-1
18|+186835xxxxx|1248016092|0|5|-1
26|+186835xxxxx|1248177103|0|5|3

The Symbian operating system for mobile devices also makes use of SQLite databases, and other computer applications store investigatively useful information in SQLite databases, including Firefox 3 and Skype. For instance, the moz_places table in the places.sqlite file from Firefox 3 is shown below.

This file can also be queried using SQL, as shown here being queried for all URLs containing the cmdLabs web site.

C:\tools>sqlite3 E:\firefox\places.sqlite
SQLite version 3.6.16
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> .tables
moz_anno_attributes  moz_favicons         moz_keywords
moz_annos            moz_historyvisits    moz_places
moz_bookmarks        moz_inputhistory
moz_bookmarks_roots  moz_items_annos

sqlite> select * from moz_places WHERE url like '%cmdlabs%';
621|http://www.cmdlabs.com/|Home|moc.sbaldmc.www.|1|0|1||2000
622|http://www.cmdlabs.com/page11/page11.html|Blog|moc.sbaldmc.www.|1|0|0||100
623|http://www.cmdlabs.com/services/services.html|Services|moc.sbaldmc.www.|1|0|0||100
624|http://www.cmdlabs.com/services/services/services-4.html|Training and Education|moc.sbaldmc.www.|1|0|0||100

Programs like Firefox that maintain usage records in these databases may leave remnants of deleted items that may be recoverable from unallocated disk space as detailed in Murilo Tito Pereira’s article “Forensic analysis of the Firefox 3 internet history and recovery of deleted SQLite records” (www.digitalinvestigation.net).

Files created using Microsoft Office applications have more metadata than many forensic practitioners realize. Word documents, Excel spreadsheets, Powerpoint presentations, and Outlook e-mail messages are essentially a file system within a file. They are structured storage files that use OLE to create the equivalent of folders (called storages) and files (called streams).

For example, consider metadata embedded within Word 2003 documents. The Summary Information metadata extracted from a Word document using Harlan Carvey’s wmd.pl Perl script is shown here:

--------------------
Summary Information
--------------------
Title : cmdLabs
Subject :
Authress : LastName FirstName
LastAuth : LastName FirstName
RevNum : 39
AppName : Microsoft Word 11.4.2
Created : 01.28.2009, 12:12:00
Last Saved : 02.05.2009, 00:36:00
Last Printed : 02.03.2009, 15:08:00

Beyond the Summary Information metadata that most forensic practitioners are familiar with and many tools can extract, Word documents also have a FILETIME value in the ROOT ENTRY header that records the last time a document was altered. This value can provide the last modified time of a document even if the timestamps in the file system or Summary Information metadata have been maliciously altered (utilities are available that make such tampering simple).

An example of this date-time stamp in the ROOT ENTRY header is provided here (2/5/2009 12:36:04 AM):

Forensic examiners should also be aware that Microsoft Office documents have embedded metadata associated with individual objects within the file, as shown here using SSView (http://www.mitec.cz/).

Excel also contains an abundance of metadata stored within its Binary Interchange File Format (BIFF5 – 8). For instance, the cells that were selected the last time a spreadsheet was saved, and the registered name that most recently opened the document with write access. Much of this metadata is accessible using BIFFView (http://b2xtranslator.sourceforge.net). A portion of the BIFFView output with the WRITEACCESS field is show here:

Reading the documented file formats of Microsoft Office files (http://msdn.microsoft.com/en-us/library/cc313118.aspx) can help forensic practitioners delve deeper into metadata, but can also be misleading and inaccurate. Therefore, it is crucial to perform controlled experiments to locate and understand the meaning of specific metadata.