Security/Mentorships/MWoS/2014/Cross-platform memory scanning in Go: Difference between revisions

← Older edit

Security/Mentorships/MWoS/2014/Cross-platform memory scanning in Go (view source)

Revision as of 16:55, 28 November 2014

8,407 bytes added , 28 November 2014

→‎Updates

Ulfr

Confirmed users

529

edits

@@ Line 18: / Line 18: @@
 === Success Criteria ===
 == Updates ==
+=== 2014-11-28 ===
+current state
+* process: abstraction for handling processes in each operating system
+** macos not yet supported
+* memaccess: purely internal to masche, read memory bits from a process
+** 19: Test Fails for Linux 32 bit binaries
+** 20: on Linux: Permission Denied is being reported as a softerror instead of harderror
+** 21: memaccess.WalkMemory doesn't respect the starting address parameter
+* listlibs: functional for windows and linux, but buggy on linux
+** no macos support right now
+* memsearch: used to search inside the memory of a process
+* to search memory:
+# call Process to obtain a process handler
+# call memsearch using the process handler
+* to search libraries:
+# call Process to obtain a process handler
+# call listlist using the process handler
+==== TODO ====
+# finish refactoring of Process package
+#* include modifying listlib and memsearch to take a Process as input
+# fix bugs in linux implementation (19-22)
+# OSX: need listlibs implementation
+# Documentation
+=== 2014-11-07 ===
+Agenda:
+* Testing on windows 32 bits and with 32 bit processes.. It works!
+* Also tested with 32bits binaries in os x. Works!
+* We should Always use 64 bit app if possible.
+* Linux fixed parsing of mapped regions with whitespaces.
+* Pato implemented the encoding detection algorithm, but it's ***SLOW***
+* We have to talk about the scope of the project and deadlines.
+** report deadline Dec 12th
+** mozilla presentation tuesday Jan.5th
+==== todo ====
+* add warning in library init() to check if the lib is compiled for 32 bits and running on 64 bits OS
+=== 2014-10-17 ===
+Summary of the week and Meeting:
+* Update on Linux status: almost done, needs testing and better error handling.
+* Update on memory interface decisions: Sometimes a memory region will cease to exist before we finish traversing it. Now if we fail, we get the next available memory region starting from the failed address, and retry to read the memory 5 times before skipping the region.
+* We decided to make a WalkMemory function that traverses the memory and applies a given function to each memory blocks (memory block size is defined as a parameter to WalkMemory, and each block overlap by half it size).
+* We talked about the test case with python socket.recvfrom_into: It is possible to do, but we have to know how to find the function names table in memory, to know if a program is calling that function.
+* Go forbidding passing Go pointers to C: ([https://code.google.com/p/go/issues/detail?id=8310 Go Issue]). We decided to wait for the proposal from the Go team to see how much does it affect us.
+* There's a working example of the memsearch functionality in the "examples" folder. It is working in Windows.
+* We have to define our deadlines for presenting the project in our university and doing a presentation to the Mozilla Team.
+=== 2014-10-10 ===
+We worked on a Go interface for memory access (Process interface), and polished the C interface.
+* Mac code for memory access was working this week.
+* We found out that Go has a [https://code.google.com/p/go/issues/detail?id=8921 bug using Windows HANDLE's]. So Windows code was not working.
+* Started refactor of Linux memory access to use the Process interface.
+* Three weeks to finish memsearch.
+=== 2014-10-03 ===
+No meeting this week. We continued working in the Mac and Windows memory access interface. We decided to do more things in Go than in C because the performance penalty of using CGO is not that big.
+=== 2014-09-26 ===
+Summary of the week and Meeting:
+* We started defining a memory access interface in C for both Mac OS and Windows.
+* Testing: Created a go test for searching known byte sequences in memory, and discussed how to organize tests in the future, probably the best would be to use a script for compiling and running the binaries. Right now our test spawns a child process and searches in its memory, we want to test on a non-related process.
+* Discussed a bit more about in-memory encoding issues. Pato will write a document with his ideas. We decided to use a normal Go regexp and ignore encoding issues for now.
+=== 2014-09-19 ===
+Summary of the week and Meeting:
+* Pato showed working code for reading memory on OS X.
+** It needs to be integrated with CGO like Marco's.
+** Report soft errors (protection failures too for now).
+** Lacks the actual search, and inter-region search.
+** Can be improved to use less memory allocations (mach_vm_copy).
+* Marco showed code for pgrep in windows
+** It's safe to pre-allocate 32k for filenames, do that.
+For Next Week:
+* Discuss charset issues
+* Finish OS X memory grep
+* Document the code in a platform independent files so godoc always shows it. And the specific bits in the platform dependent ones.
+* Start writing doc about the operating systems and how they work.
+* Linux and Windows: investigate if inter-region searches are needed.
+===== About memory protection on OS X =====
+We were getting some errors while trying to read memory with mach_vm_read because of memory protection in very simple test processes.
+The errors were because we tried to read regions of the memory that were used as stack guards and preallocated ones for malloc bookkeeping, which don't have read, write nor execute access. Other processes have even more regions like this (webkit, for example, has a second version of malloc regions, and something similar for it's JS JIT'ed code).
+Debuggers change protections (if needed) before reading a region and then restore the original one. This is safe while the process is paused, but if not, a race condition may make you restore an old protection and the process may crash.
+Possible solutions:
+* Don't read non-readable regions. CONS: It'd be trivial to avoid MIG's memory scanner if we do this.
+* Pause while accessing non-readable regions. In general this regions are just a small percentage of the entire address-space.
+* Don't pause the process and pray it doesn't crash.
+=== 2014-09-12 ===
+Summary of the week:
+* We have working versions for memory grepper on linux and windows.
+** discussion about sliding the memory buffer to apply a regex against it.
+* Moved everything to different folders (package listlibs and memsearch)
+** try to keep consistent naming across files, avoid uppercase in filenames
+* Refactor listlibs so it can return the soft errors (files that it couldn't access)
+* Tested reading invalid memory sections in Windows: ReadProcessMemory with an invalid address won't do any harm.
+For next week(s):
+*pgrep functionality for linux/windows/mac (given the regex of an executable, return a list of pids)
+*Optimize the way we are searching for a string in memory in linux to minimize the memory reads.
+*Investigate how to implement a grep like functionality to search for a string in memory.
+*Also match on binary strings (hex strings)
+=== 2014-09-05 ===
+Summary of the week and Meeting:
+* Basic linux memory searcher and windows memory regions listing
+** We talked about the source code and different approaches in using C and handling errors.
+* Double licensing: GCore uses GPL license so we would have to use both licenses if we want to use Gcore's source code.
+* Document the need for root access whenever is present: In Mac and windows there are many things that can be done without root access, but we will use root for ease of development and document those areas.
+* University's essay: start writing down notes
+* Development in the open on github? Yes. Also discussions should be held in the open.
+* Memory maps can change, so the addresses may be invalid: For memory searching we won't stop the processes because it may take too long, just for core dumps if we need to.
+* Grep something more than the heap and the stack? Not for now.
+* grep on go or C: figure out what's better, performance may or may not be an issue. go'd probably result in more robust code.
+* #mwos has been created in mozilla's irc. That's the preferred channel to contact Julien through irc.
+For Next Week:
+* Pato has working code to read memory on MacOS
+* looking at finishing the memory grepper next week
+* Julien to ask Mozilla's lawiers about double licensing
+* test if reading invalid memory sections crashes the process in each os
+=== 2014-08-29 ===
+=== 2014-08-22 ===
 === 2014-08-15 ===
 Summary of the week
 * We couldn't meet because we started classes this week and had a complicated schedule.
+* Finished windows version of library grepping.
 * Shellcode injection example under the "experimental" branch.
 * Made progress reading the book, decided to focus on acquiring memory first.
+* Changed project name to MASCHE: Memory Analysis Suite for Controlling Harmony of Endpoints.
 Plan for next week:
@@ Line 32: / Line 176: @@
 * Port windows EnumProcessModules to go: 50% (only ported EnumProcesses and started with EnumProcessModules)
 * Mac: Program to list memory regions of a pid. This enables us to do vm_read to read arbitrary memory addresses.
-* We have now a repository on github: https://github.com/mozilla/migmem
+* We have now a repository on github: https://github.com/mozilla/masche
 Plan for next week: