Build:OutageReports:20070410-02
From MozillaWiki
Outage Template
On Apr 10th from ~ 4 am, moz180-linux-tbox experienced a service outage for ~ 10 hours.
What was affected:
Firefox linux builds on Mozilla1.8.0 branch (en-US and locales)
What was the cause of the outage:
The /builds partition was full. It looks like symbol generation has been busted for the last few days (at least), with firefox-bin.elf ending up a random, truncated, size and zero byte files for the other executables.
What will be done to prevent this in the future:
Some ideas
- Monitor tinderbox disk space (eg Nagios)
- Make tinderbox go red on symbol problems (creation and push)
