Sheriffing/TBPL/FuturePlans

Goals

• get rid of Tinderbox (bug 586539)
• be super-fast, even when going back in time

Necessary prereqs

• for getting rid of Tinderbox:
  • move all the useful stuff on Tinderbox over to TBPL, specifically:
    • current sheriff (bug 540464)
    • tree state (bug 525464)
    • other notes at the top
  • put logs somewhere outside Tinderbox
    • where? (FTP is bug 530318)
  • move starring system outside Tinderbox
    • where? TBPL server?
• for being fast:
  • either have fast access to Tinderbox / Buildbot data in the first place *or*
  • cache Tinderbox history on TBPL server in a database
    • for how long? a week?
    • do the same for pushlog and buildbot data?

Transition plan for moving away from Tinderbox

• use data from both systems
• give TBPL a server component with a database
• put everything into TBPL's database
• correlate buildbot data with Tinderbox data if possible
  • finished Tinderbox jobs have a unique ID, buildbot jobs don't
    • create something (hash starttime + slave name?) - see bug 586539 comment 2
  • put the correlation into TBPL database
• everything using PHP and MySQL for now

Decisions we need to make

• How much should happen on a server and how much on the client?
• If it doesn't happen on the client, what server should do it?

These things need to go on a server, no matter which one

• Generating buildbot JSON data in the right format
  • In dbaron's experiment this is a script on dbaron's server that transforms data from build.mozilla.org
  • But we should probably generate the data in the right format on build.mozilla.org in the first place
• Starring
  • Resides in Tinderbox at the moment
• Storage and processing of logs and summaries
  • Is done by Tinderbox at the moment.
  • Plan is to upload logs to FTP (bug 530318) and to do the processing (error line linking, summary generation) on a server
• Commenting on random-orange bugs ("Tinderboxpushlog Robot")
  • Currently done by a PHP script on TBPL's server
• Searching bugzilla for random-orange bugs
  • Also happens with PHP script on TBPL server at the moment
  • Too slow to do every time on the client, needs to be done and cached on a server
• (General caching so stuff is fast)

Caching plan

We need to decide what we cache where. At the moment it looks like this:

Caches already in place

• Pushlog data is cached on hg.mozilla.org by the pushlog itself.
  • (Side issue: at the moment cache invalidation is broken there, see bug 587559)
  • People have been asking for another cache on the TBPL side, see bug 526159
• Tinderbox data is partly cached by Tinderbox itself: the last 24 hours are in json.js, further back in time nothing is cached.
• Full Tinderbox logs are stored as .gz files on the Tinderbox server, but linkified logs and summaries are regenerated on every request, as far as I know.
• The summaries of these logs are cached by TBPL.

Caches during transition period

where by transition period I mean the time before we've replaced Tinderbox completely

This is what I'm proposing. Please discuss.

• Pushlog data still only cached by Pushlog itself
  • Load will be reduced significantly when the new Pushlog JSON is in place (bug 584836, waiting to be deployed: bug 585882)
• Tinderbox data cached in TBPL database
• Buildbot data cached in TBPL database
• Full Tinderbox logs still only stored on Tinderbox server
• Summaries still cached by TBPL.

Caches after Tinderbox is gone

At this point we don't need to corellate Tinderbox data with Buildbot data anymore, so most of TBPL's database can go away. The Buildbot server is assumed to be fast.

• Pushlog data still only cached by Pushlog itself
• Buildbot data cached on Buildbot server in the right format
• Full build logs somewhere on FTP
• Processed build logs and summaries cached on some other server (probably on the server that's processing the logs)
• Summaries cached on that same server

Next actions

1. Create a database structure
2. Create a PHP script that periodically loads Buildbot data + Tinderbox data into the database
3. Create a spec for the data that TBPL server should send to TBPL client
4. Create a script that reads from the database and sends this kind of data
5. ... more to come

Arpad’s ideas

• bitrotted server and client implementation in bug 553549
  • based on php, mysql; parses pushlog html and tinderbox json
  • revived the client part in bug 576489, bitrotted a bit again :(, would like to wait for pl json and tb json2 before continuing work
• how fast is the new pushlog json? does it beat a custom database?
  • the json includes the list of changed files which is irrelevant to tbpl
• how to properly handle builds that pull from multiple repos (preliminaty ui in bug 587761)
  • builds.m.o json has confusing attribute names for multi-repo builds
  • want to display multiple pushlogs side-by-side, or easy switching without the need to reload the data
  • what happens for "no pushes between X and Y" cases (see next point)
• "how many pushlog entries I can see" vs. time-based queries; former needs dynamic database access as opposed to pregenerated json files
  • history-as-you-scroll
• playing around with node.js and MongoDB, however mongodb driver for node is slow.
  • could do log preprocessing and bug suggestion in node when the tb json is loaded and save result to database