Auto-tools/Projects/OrangeFactor

Meeting: September 22, 2010

Parsing: - logparser lives here: http://hg.mozilla.org/automation/logparser

 - THIS DOESN'T WORK! its a straight port from topfails (which is to say it works but is inadequate and there are also bugs on it)

- should be done to jython standards for hadoop

 Storage:
 - files in filesystem mirror that from the ftp site
 - (raw) log -> parser -> flume (sp?) -> hdfs
 - block size: 128M
   - does this make looking through files slow?

What do we want?

- we have a (proposed) schema
- we have a (proposed) REST interface
- (we should put this on a wiki page and move towards finalization)

Process:

- we give python script (e.g. logparser)
- invoked on every log file
- output == what we want