ReleaseEngineering/How To/Cleanup Old Candidates Directories

From MozillaWiki
Jump to: navigation, search

We accumulate release candidates as time goes by. Since we have a finite amount of space in that location (500G at the time of writing) we must tidy these up as we periodically. It can be useful to have old candidates around for later debugging so we move them to other storage first, and later remove all the binaries while retaining logs. The sequence of locations is

/pub/mozilla.org/firefox/candidates
/data/cltbld/old-candidates-dirs/firefox
/data/cltbld/old-candidates-dirs/firefox/stripped

We also have old xulrunner and mobile candidates in /data/cltbld/old-candidates-dirs/. We'll talk about these in reverse order, since that's the way you need to maintain space.

This is all subject to change with the move to SCL3 and a new staging server.

Cleaning up /data/cltbld/old-candidates-dirs/firefox/

Releases end up in /data/cltbld/old-candidates-dirs/firefox/stripped/ as a final resting place, with only the buildbot and signing logs, SUMS files + sigs, plus the <platform>_info.txt files kept for buildIDs. This is a few 10s of MB instead of 10+ GB.

We move files from /data/cltbld/old-candidates-dirs/firefox/ into stripped/ like this

# ffxbld@stage
cd /data/cltbld/old-candidates-dirs/firefox/
ls -ltr | head

to get a list of the oldest candidates. You can run du on those directories if you want to know how much you're going to free. Then start moving them, eg

./move-stripped 6.0

returning

Saving important objects from 6.0-candidates
Deleting 6.0-candidates
Free space is now:
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p1     1.9T  1.6T  289G  85% /data

Cleaning up /pub/mozilla.org/firefox/candidates/

NB: Leave the last build from each branch present for update generation. This includes old branches (eg 3.0.19, 3.5.19, 3.6.28). Otherwise move the oldest first

Assuming you have space in /data/cltbld/old-candidates-dirs/firefox/ it's as simple as

# ffxbld@stage
cd /data/cltbld/old-candidates-dirs/firefox/
ls -1 /pub/mozilla.org/firefox/candidates/

You can run du on those directories to see how much space you need. Then start moving them,

./move-from-candidates 10.0esr 11.0b2 10.0.1

This will take take quite a while, as each is multiple GB and ionice is used to run at low-priority. It'll show you progress along the way though.

TODO - figure out a work around for contrib/ files that aren't owned by ffxbld. In the meantime this will stop deleting after it fails on the first version. Run as root to avoid this.