Elmo/Retention Policy

Elmo generates a host of data in various forms. This document describes how the amount of data stored will be limited, backed up, or lost.

Right now, there's no retention policy. We store everything.

Data types

The data generated by elmo is stored in two forms right now, SQL and plain files.

SQL

Most of the process data and statistics is stored in SQL, through django apps.

Files

Buildbot stores build logs in individual text files, next to status pickles.

SQL

Right now, there's no immediate need to create a retention policy for our databases.

TODO: Ask IT when that comes.

Files

There are different file types, with different importance. Ordered by how badly they break the buildbot install when lost:

master-ball/l10n-master/changes.pck stores the changes. This file wants to be pruned every now and then, but needs to make sure that the changenumber is kept. Right now, Axel occasionally does:

cd site/master-ball/l10n-master
../scripts/buildbot stop .
python
import pickle
cm = pickle.load(open('changes.pck'))
len(cm.changes)
del cm.changes[:]
cm.parent = None
cm.saveYourself()
^D
# in the shell again
../scripts/buildbot start .