User:Zeller

Ship It

Bug 826753 - release automation should update ship it at certain points

Notes

• /status - Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
• /statuses.html – Pretty GUI view of the info shown in /status. Analogous to /releases.html
• /status/<release-name>
  • GET: Lists release status info in JSON. Analogous to /releases/<release-name>
  • POST: covered in Bug 1032985
• /status/<release-name>.html – Pretty GUI view of the info shown in /status/ GET, possibly including a visual timeline of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease. (status steps taken from Bug 826753 comment 0)

Questions

• Why does /release 404?
• Should /statuses be available through a 3rd tab (ie Reviewed/Completed/Statuses), or through a new "view statuses" link?
• Does a ViewStatuses() need to be added to kickoff.js? In addition to the ViewReleases() function?

To Do

• Fix line 331 in model.py to sort the product properly
• When RelMan hits 'go', add proper row to database
• Make system diagram for interactions in shipit additions
• Make /status GET queriable (other than var1 and var2). Other release tables use the column variables ready and complete.
• Run a JOIN via the name column between release_status and *_releases, and print that JSON info upon /status/<release-name> GET request.
• Take progress bars from bootstrap rather than change the entire css

Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database

https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2

Add REST API entry point for updating update.db at /status/<release-name> when POST

Notes

• Use name field in release_status as an unofficial foreign key to the 3 *_releases tables (Can't use JOIN for 3 primary keys (all name) in the *_releases tables)

Questions

• How exactly should I divide the release pulse messages into the 6 statuses: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease.

To Do

• Have StatusAPI update the database tables *-release, and check that name is in release_data
• Check to see if name already exists in tables *-releases, and check that name is in release_data before adding to release_status... use as switch between creating a new row and updating rows.
• DATETIME stamps need to not have T or +01:00
• Sort data into the 6 statuses

Bug 1032978 - Add a standalone process that listens to pulse for release related buildbot messages

Long-running standalone script (shipit-agent.py) that listens to pulse and uses / status/<release-name> with POST to update update.db for <release-name> status

Notes

• pulse.m.o gives buildbot events
  • Look for build.release-*.finished (Found in ['_meta']['routing_key']?)
  • Use FF candidate buildlogs to help determine the names of the builder I am interested in (ie the first part before "bmNN" is the name of the builder)
• If collecting, release_data, then the table will gain ~0.5mb/release, which I estimate makes it ~117Mb/year.
• If version = "None", then routing_key doesn't have build.release-*
• Some pulse fields are unnecessary, such as 'exchange', and 'serializer'
• Nothing says *.finished
• Ignore the following messages:
  • Any routing_key that doesn't match build.release-* (including unittest.release-*, talos.release-*, etc... they are tests we don't usually look at.
  • Any version or build_number that is None... there was a bug for that, they are test related. These also seem to only show up on release messages that don't match build.release-*, like unittest.release-*, talos.release-*, etc.
  • Any xulrunner jobs, they are implicitly run by Firefox
• [Examples https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2] given by rail on how to sort pulse messages into the 6 statuses

Questions

• What do do with release messages that don't fit the expected:
  • Have a product other than "firefox" when supposedly it's a firefox build (ie "mobile" or "xulrunner"
• Did I scrap all of Firefox-31.0b5-build1 or am I missing something?
• Should I look at routing_key, or key, or?
• Do we want to keep all buildbot release pulse data, to be able to use it later to deduce new things, or forgoe all that entirely? (feature creep)
• Should protocol be configurable via puppet or with command line option?
• Is it possible to use the topic option with pulse to listen to only build.release-* messages?
• Should unique_label have a set name?
• Should I be using a different consumer like BuildConsumer? NormalizedBuildConsumer will miss some builds, pulsetranslator has a hardcoded list of recognized platforms, and will drop others. http://hg.mozilla.org/automation/mozillapulse/file/default/mozillapulse/consumers.py
  • Dump out the full message stream for a while until you see one of the ones you care about.
• Should we send the concat'd data instead of payload, and have the common_keys listed in the REST API and hold all the logic for the None's and such there?
  • No because the API should only be hit with messages we care about, and so should be filtered before ever propagating a POST request to the REST API

To Do

• Give rail an example of the product "mobile" showing up when I expect "firefox"
• Run with supervisord if supposed to be run in the foreground
• Read credentials from a file deploted by puppet, rather than relying on .netrc for username and password
• Make protocol configurable via puppet or with command line option
• Use selfserve-agent.py as the template for this
• Mark it as durable

Bug 1032975 - Add new table(s) to shipit database

New table(s) in update.db

Questions

• Should we have 2 tables (release_status and release_data), or just 1 (release_status)?
• Is release_status schema adequete?

To Do

• Give rail the estimate for how much data will be added to the release_data table
• Show sheeri the schema for help in optimizing columns, and to get other feedback once ready

Questions

• Can only store events with results=0 because otherwise (name, event_name) has duplicates when results!=0
• status/<release_name> GET could send back a dict like {'Firefox-31.0b5-build1': [...]} rather then {'name': 'Firefox-31.0b5', 'data': [...]}
• Do we want to go ahead and change all 'status' names to 'events'? ie status.html would be events.html
  • For some, but colloquially 'status' makes more sense then 'events' for user facing things
• Make sure that repack_complete event messages are being counted as True even after all chunks are counted.
  • Simply counting chunks is not enough, this should still pass a True even without chunks
• When submitting a build, when choosing ready, the page reloads to show the new build in the Reviewed tab, but none of the Running tab populates. It requires refreshing the entire page to see.
• Include static copy of bootstrap?
  • Perhaps copy just the snippets I would like
• Top level keys complete and progress could be 1 float key
• When using a nonexistent releaseName for status/<releaseName> the GET does not fail, but instead sends a normal JSON response with an empty events value
• Having to split the '+01:00' from sent on pulse messages in order to parse into datetime object for entry into the database. Better way? dateutil is not installed, so that would change dependencies, but from dateutil.parser import parse works great!
  • Going to just dump the +01:00 for now, but in the future it could be added by running sent = datetime.strptime('2014-08-08T16:12:59', "%Y-%m-%dT%H:%M:%S") + timedelta(+1)