Security/Mentorships/MWoS/2014/Cross-platform memory scanning in Go

Introduction

We are a team of Computer Science students from Argentina, highly motivated by coding, security and free software.

Members

• Agustin Martinez Suñé
• Marco Vanotti
• Nahuel Lascano
• Patricio Palladino
• Professor: Alejandro Furfaro
• Mozilla Advisor: Julien Vehent

Description

The Mozilla InvestiGator (MIG) project needs a way to inspect the content of the memory of a system, and detect threats. The typical approach in memory forensic is to dump the memory of a system, and perform analysis on another system, using tools like Volatility. We are looking for an approach that is less invasive, where an agent running on a target system can inspect its own memory without disrupting operations. Existing libraries, such as Volatility, are hard to ship to remote systems because of their size and dependencies. The goal of this project is to design and code a lean, cross-platform, memory inspection library in the Go language that can be integrated into MIG.

Scope

Success Criteria

2014-11-28

current state

• process: abstraction for handling processes in each operating system
  • macos not yet supported
• memaccess: purely internal to masche, read memory bits from a process
  • 19: Test Fails for Linux 32 bit binaries
  • 20: on Linux: Permission Denied is being reported as a softerror instead of harderror
  • 21: memaccess.WalkMemory doesn't respect the starting address parameter
• listlibs: functional for windows and linux, but buggy on linux
  • no macos support right now
• memsearch: used to search inside the memory of a process

• to search memory:

1. call Process to obtain a process handler
2. call memsearch using the process handler

• to search libraries:

1. call Process to obtain a process handler
2. call listlist using the process handler

TODO

1. finish refactoring of Process package
   • include modifying listlib and memsearch to take a Process as input
2. fix bugs in linux implementation (19-22)
3. OSX: need listlibs implementation
4. Documentation

2014-11-07

Agenda:

• Testing on windows 32 bits and with 32 bit processes.. It works!
• Also tested with 32bits binaries in os x. Works!
• We should Always use 64 bit app if possible.
• Linux fixed parsing of mapped regions with whitespaces.
• Pato implemented the encoding detection algorithm, but it's ***SLOW***
• We have to talk about the scope of the project and deadlines.
  • report deadline Dec 12th
  • mozilla presentation tuesday Jan.5th

todo

• add warning in library init() to check if the lib is compiled for 32 bits and running on 64 bits OS

2014-10-17

Summary of the week and Meeting:

• Update on Linux status: almost done, needs testing and better error handling.
• Update on memory interface decisions: Sometimes a memory region will cease to exist before we finish traversing it. Now if we fail, we get the next available memory region starting from the failed address, and retry to read the memory 5 times before skipping the region.
• We decided to make a WalkMemory function that traverses the memory and applies a given function to each memory blocks (memory block size is defined as a parameter to WalkMemory, and each block overlap by half it size).
• We talked about the test case with python socket.recvfrom_into: It is possible to do, but we have to know how to find the function names table in memory, to know if a program is calling that function.
• Go forbidding passing Go pointers to C: (Go Issue). We decided to wait for the proposal from the Go team to see how much does it affect us.
• There's a working example of the memsearch functionality in the "examples" folder. It is working in Windows.
• We have to define our deadlines for presenting the project in our university and doing a presentation to the Mozilla Team.

2014-10-10

We worked on a Go interface for memory access (Process interface), and polished the C interface.

• Mac code for memory access was working this week.
• We found out that Go has a bug using Windows HANDLE's. So Windows code was not working.
• Started refactor of Linux memory access to use the Process interface.
• Three weeks to finish memsearch.

2014-10-03

No meeting this week. We continued working in the Mac and Windows memory access interface. We decided to do more things in Go than in C because the performance penalty of using CGO is not that big.

2014-09-26

Summary of the week and Meeting:

• We started defining a memory access interface in C for both Mac OS and Windows.
• Testing: Created a go test for searching known byte sequences in memory, and discussed how to organize tests in the future, probably the best would be to use a script for compiling and running the binaries. Right now our test spawns a child process and searches in its memory, we want to test on a non-related process.
• Discussed a bit more about in-memory encoding issues. Pato will write a document with his ideas. We decided to use a normal Go regexp and ignore encoding issues for now.

2014-09-19

Summary of the week and Meeting:

• Pato showed working code for reading memory on OS X.
  • It needs to be integrated with CGO like Marco's.
  • Report soft errors (protection failures too for now).
  • Lacks the actual search, and inter-region search.
  • Can be improved to use less memory allocations (mach_vm_copy).
• Marco showed code for pgrep in windows
  • It's safe to pre-allocate 32k for filenames, do that.

For Next Week:

• Discuss charset issues
• Finish OS X memory grep
• Document the code in a platform independent files so godoc always shows it. And the specific bits in the platform dependent ones.
• Start writing doc about the operating systems and how they work.
• Linux and Windows: investigate if inter-region searches are needed.

About memory protection on OS X

We were getting some errors while trying to read memory with mach_vm_read because of memory protection in very simple test processes.

The errors were because we tried to read regions of the memory that were used as stack guards and preallocated ones for malloc bookkeeping, which don't have read, write nor execute access. Other processes have even more regions like this (webkit, for example, has a second version of malloc regions, and something similar for it's JS JIT'ed code).

Debuggers change protections (if needed) before reading a region and then restore the original one. This is safe while the process is paused, but if not, a race condition may make you restore an old protection and the process may crash.

Possible solutions:

• Don't read non-readable regions. CONS: It'd be trivial to avoid MIG's memory scanner if we do this.
• Pause while accessing non-readable regions. In general this regions are just a small percentage of the entire address-space.
• Don't pause the process and pray it doesn't crash.

2014-09-12

Summary of the week:

• We have working versions for memory grepper on linux and windows.
  • discussion about sliding the memory buffer to apply a regex against it.
• Moved everything to different folders (package listlibs and memsearch)
  • try to keep consistent naming across files, avoid uppercase in filenames
• Refactor listlibs so it can return the soft errors (files that it couldn't access)
• Tested reading invalid memory sections in Windows: ReadProcessMemory with an invalid address won't do any harm.

For next week(s):

• pgrep functionality for linux/windows/mac (given the regex of an executable, return a list of pids)
• Optimize the way we are searching for a string in memory in linux to minimize the memory reads.
• Investigate how to implement a grep like functionality to search for a string in memory.
• Also match on binary strings (hex strings)

2014-09-05

Summary of the week and Meeting:

• Basic linux memory searcher and windows memory regions listing
  • We talked about the source code and different approaches in using C and handling errors.

• Double licensing: GCore uses GPL license so we would have to use both licenses if we want to use Gcore's source code.
• Document the need for root access whenever is present: In Mac and windows there are many things that can be done without root access, but we will use root for ease of development and document those areas.
• University's essay: start writing down notes
• Development in the open on github? Yes. Also discussions should be held in the open.
• Memory maps can change, so the addresses may be invalid: For memory searching we won't stop the processes because it may take too long, just for core dumps if we need to.
• Grep something more than the heap and the stack? Not for now.
• grep on go or C: figure out what's better, performance may or may not be an issue. go'd probably result in more robust code.
• #mwos has been created in mozilla's irc. That's the preferred channel to contact Julien through irc.

For Next Week:

• Pato has working code to read memory on MacOS
• looking at finishing the memory grepper next week
• Julien to ask Mozilla's lawiers about double licensing
• test if reading invalid memory sections crashes the process in each os

2014-08-29

2014-08-22

2014-08-15

Summary of the week

• We couldn't meet because we started classes this week and had a complicated schedule.
• Finished windows version of library grepping.
• Shellcode injection example under the "experimental" branch.
• Made progress reading the book, decided to focus on acquiring memory first.
• Changed project name to MASCHE: Memory Analysis Suite for Controlling Harmony of Endpoints.

Plan for next week:

• Manually create a memory dump for each os that is readable by volatility.

2014-08-08

Summary of the week

• Linux program to check libs for regexp done.
• Port windows EnumProcessModules to go: 50% (only ported EnumProcesses and started with EnumProcessModules)
• Mac: Program to list memory regions of a pid. This enables us to do vm_read to read arbitrary memory addresses.
• We have now a repository on github: https://github.com/mozilla/masche

Plan for next week:

• Researh more about Mac, read The art of memory forensics.
• Read about: Hollow process detection
  • influence of ASLR ? /proc/sys/kernel/randomize_va_space
• Finish porting windows EnumProcessModules to go.
• Write a symbol table attack in C and write the module that detects it.

2014-08-01

Summary of the week:

• Linux: /proc/<pid>/maps for .so mapped files, and /proc/<pid>/mem for raw memory.
• Windows has a couple of functions via the Windows API, that can be called with LoadLibrary or CGO. We decided to use CGO because it was easier.
• Mac: Not so easy, there's a way to get a coredump without killing the process, we can do that to analyze what we want. osxbook has an example.
• If we use CGO we won't be able to cross-compile. Not a problem.
• Go has packages for accessing ELF/Mach-O/PE files in the debug package, we can analyze binaries and get more info.
• We got working examples on Linux and Windows to get a process loaded libraries.

Plan for next week:

• Investigate how to get more information about libraries (version number, build, revision, etc)
• Read more about Mac and how to get the data that we need.
• Decide on a name for the project
• Sign committer agreement

2014-07-14

• Team presentation
• Project introduction: MIG and the memory scanner module.
• Set deadline: mid/end of December
• Agreed on meeting once a week. Meetings scheduled by the team.
• Plan for the next week: Investigate how to get the information that the memory scanner will provide in linux, windows and OS X.

<date>

• current work
• blocking points
• discussion points
• upcoming work