User:Zeller
From MozillaWiki
Contents
- 1 Ship It
- 1.1 Bug 826753 - release automation should update ship it at certain points
- 1.2 Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database
- 1.3 Bug 1032978 - Add a standalone process that listens to pulse for release related buildbot messages
- 1.4 Bug 1032975 - Add new table(s) to shipit database
- 2 Problems with Bug 1032978
- 3 Testing Staging-Release
- 4 Questions
Ship It
Bug 826753 - release automation should update ship it at certain points
Notes
- /status - Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
- /statuses.html – Pretty GUI view of the info shown in /status. Analogous to /releases.html
- /status/<release-name>
- GET: Lists release status info in JSON. Analogous to /releases/<release-name>
- POST: covered in Bug 1032985
- /status/<release-name>.html – Pretty GUI view of the info shown in /status/ GET, possibly including a visual timeline of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease. (status steps taken from Bug 826753 comment 0)
Questions
- Why does /release 404?
- Should /statuses be available through a 3rd tab (ie Reviewed/Completed/Statuses), or through a new "view statuses" link?
- Does a ViewStatuses() need to be added to kickoff.js? In addition to the ViewReleases() function?
To Do
- Fix line 331 in model.py to sort the product properly
- When RelMan hits 'go', add proper row to database
- Make system diagram for interactions in shipit additions
- Make /status GET queriable (other than var1 and var2). Other release tables use the column variables ready and complete.
- Run a JOIN via the name column between release_status and *_releases, and print that JSON info upon /status/<release-name> GET request.
- Take progress bars from bootstrap rather than change the entire css
Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database
https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2
Add REST API entry point for updating update.db at /status/<release-name> when POST
Notes
- Use name field in release_status as an unofficial foreign key to the 3 *_releases tables (Can't use JOIN for 3 primary keys (all name) in the *_releases tables)
Questions
- How exactly should I divide the release pulse messages into the 6 statuses: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease.
To Do
- Have StatusAPI update the database tables *-release, and check that name is in release_data
- Check to see if name already exists in tables *-releases, and check that name is in release_data before adding to release_status... use as switch between creating a new row and updating rows.
- DATETIME stamps need to not have T or +01:00
- Sort data into the 6 statuses
Long-running standalone script (shipit-agent.py) that listens to pulse and uses / status/<release-name> with POST to update update.db for <release-name> status
Notes
- pulse.m.o gives buildbot events
- Look for build.release-*.finished (Found in ['_meta']['routing_key']?)
- Use FF candidate buildlogs to help determine the names of the builder I am interested in (ie the first part before "bmNN" is the name of the builder)
- If collecting, release_data, then the table will gain ~0.5mb/release, which I estimate makes it ~117Mb/year.
- If version = "None", then routing_key doesn't have build.release-*
- Some pulse fields are unnecessary, such as 'exchange', and 'serializer'
- Nothing says *.finished
- Ignore the following messages:
- Any routing_key that doesn't match build.release-* (including unittest.release-*, talos.release-*, etc... they are tests we don't usually look at.
- Any version or build_number that is None... there was a bug for that, they are test related. These also seem to only show up on release messages that don't match build.release-*, like unittest.release-*, talos.release-*, etc.
- Any xulrunner jobs, they are implicitly run by Firefox
- [Examples https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2] given by rail on how to sort pulse messages into the 6 statuses
Questions
- What do do with release messages that don't fit the expected:
- Have a product other than "firefox" when supposedly it's a firefox build (ie "mobile" or "xulrunner"
- Did I scrap all of Firefox-31.0b5-build1 or am I missing something?
- Should I look at routing_key, or key, or?
- Do we want to keep all buildbot release pulse data, to be able to use it later to deduce new things, or forgoe all that entirely? (feature creep)
- Should protocol be configurable via puppet or with command line option?
- Is it possible to use the topic option with pulse to listen to only build.release-* messages?
- Should unique_label have a set name?
- Should I be using a different consumer like BuildConsumer? NormalizedBuildConsumer will miss some builds, pulsetranslator has a hardcoded list of recognized platforms, and will drop others. http://hg.mozilla.org/automation/mozillapulse/file/default/mozillapulse/consumers.py
- Dump out the full message stream for a while until you see one of the ones you care about.
- Should we send the concat'd data instead of payload, and have the common_keys listed in the REST API and hold all the logic for the None's and such there?
- No because the API should only be hit with messages we care about, and so should be filtered before ever propagating a POST request to the REST API
To Do
- Give rail an example of the product "mobile" showing up when I expect "firefox"
- Run with supervisord if supposed to be run in the foreground
- Read credentials from a file deploted by puppet, rather than relying on .netrc for username and password
- Make protocol configurable via puppet or with command line option
- Use selfserve-agent.py as the template for this
- Mark it as durable
Bug 1032975 - Add new table(s) to shipit database
New table(s) in update.db
Notes
Questions
- Should we have 2 tables (release_status and release_data), or just 1 (release_status)?
- Is release_status schema adequete?
To Do
- Give rail the estimate for how much data will be added to the release_data table
- Show sheeri the schema for help in optimizing columns, and to get other feedback once ready
Problems with Bug 1032978
- When running test producer and consumer, when api is called, getting the following error:
build-tools)localhost:build-tools jzeller$ python buildfarm/release/shipit-notifier.py -c buildfarm/release/shipit-notifier.ini.example 2014-08-11 14:37:37,400 listening for pulse messages 2014-08-11 14:37:37,427 Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'2.6.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] 2014-08-11 14:37:37,440 Open OK! 2014-08-11 14:37:37,440 using channel_id: 1 2014-08-11 14:37:37,446 Channel open 2014-08-11 14:37:39,794 msg received - release-mosilla-beta-firefox_tag 2014-08-11 14:37:39,794 adding new release event for Firefox-31.0b1-build1 with event_name release-mosilla-beta-firefox_tag Traceback (most recent call last): File "buildfarm/release/shipit-notifier.py", line 115, in <module> main() File "buildfarm/release/shipit-notifier.py", line 112, in main pulse.listen() File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/mozillapulse/consumers.py", line 148, in listen self.connection.drain_events() File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events return self.transport.drain_events(self.connection, **kwargs) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events return connection.drain_events(**kwargs) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/amqp/connection.py", line 325, in drain_events return amqp_method(channel, args, content) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/amqp/channel.py", line 1908, in _basic_deliver fun(msg) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/messaging.py", line 592, in _receive_callback return on_m(message) if on_m else self.receive(decoded, message) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/messaging.py", line 559, in receive [callback(body, message) for callback in callbacks] File "buildfarm/release/shipit-notifier.py", line 91, in got_message receive_message(config, *args, **kwargs) File "buildfarm/release/shipit-notifier.py", line 74, in receive_message status_api.update(name, data=payload) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/buildtools-1.0.4-py2.7.egg/kickoff/api.py", line 117, in update url_template_vars={'name': name}).content File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/buildtools-1.0.4-py2.7.egg/kickoff/api.py", line 47, in request auth=self.auth) TypeError: request() got an unexpected keyword argument 'config'
- After deleting config=self.config in previous issue, got this error:
(build-tools)localhost:build-tools jzeller$ python buildfarm/release/shipit-notifier.py -c buildfarm/release/shipit-notifier.ini.example 2014-08-11 14:54:01,065 listening for pulse messages 2014-08-11 14:54:01,095 Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'2.6.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] 2014-08-11 14:54:01,106 Open OK! 2014-08-11 14:54:01,107 using channel_id: 1 2014-08-11 14:54:01,112 Channel open 2014-08-11 14:54:05,748 msg received - release-mosilla-beta-firefox_tag 2014-08-11 14:54:05,748 adding new release event for Firefox-31.0b1-build1 with event_name release-mosilla-beta-firefox_tag 2014-08-11 14:54:05,775 Starting new HTTP connection (1): 127.0.0.1 2014-08-11 14:54:05,778 "HEAD /csrf_token HTTP/1.1" 200 0 2014-08-11 14:54:05,779 Request to http://127.0.0.1:5000/status/Firefox-31.0b1-build1 2014-08-11 14:54:05,779 Data sent: {'csrf_token': '20140811222405##82b09f99876eead53a695ae607c656733e2d4202', 'data': {u'event_name': u'release-mosilla-beta-firefox_tag', u'group': u'tag', u'results': 0, u'sent': u'2014-08-11T14:54:05+01:00'}} 2014-08-11 14:54:05,779 retry: Calling <bound method Session.request of <requests.sessions.Session object at 0x1077a85d0>> with args: (), kwargs: {'params': None, 'timeout': 60, 'url': u'http://127.0.0.1:5000/status/Firefox-31.0b1-build1', 'config': {'danger_mode': True}, 'data': {'csrf_token': '20140811222405##82b09f99876eead53a695ae607c656733e2d4202', 'data': {u'event_name': u'release-mosilla-beta-firefox_tag', u'group': u'tag', u'results': 0, u'sent': u'2014-08-11T14:54:05+01:00'}}, 'method': 'POST', 'auth': ('admin', 'password')}, attempt #1 Traceback (most recent call last): File "buildfarm/release/shipit-notifier.py", line 115, in <module> main() File "buildfarm/release/shipit-notifier.py", line 112, in main pulse.listen() File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/mozillapulse/consumers.py", line 148, in listen self.connection.drain_events() File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events return self.transport.drain_events(self.connection, **kwargs) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events return connection.drain_events(**kwargs) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/amqp/connection.py", line 325, in drain_events return amqp_method(channel, args, content) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/amqp/channel.py", line 1908, in _basic_deliver fun(msg) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/messaging.py", line 592, in _receive_callback return on_m(message) if on_m else self.receive(decoded, message) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/kombu/messaging.py", line 559, in receive [callback(body, message) for callback in callbacks] File "buildfarm/release/shipit-notifier.py", line 91, in got_message receive_message(config, *args, **kwargs) File "buildfarm/release/shipit-notifier.py", line 74, in receive_message status_api.update(name, data=payload) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/buildtools-1.0.4-py2.7.egg/kickoff/api.py", line 117, in update url_template_vars={'name': name}).content File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/buildtools-1.0.4-py2.7.egg/kickoff/api.py", line 59, in request auth=self.auth, params=params) File "/Users/jzeller/Mozilla/build-tools/lib/python2.7/site-packages/buildtools-1.0.4-py2.7.egg/util/retry.py", line 32, in retry return action(*args, **kwargs) TypeError: request() got an unexpected keyword argument 'config'
- May need to add default=None to the platform Column in model.py
- Any reason to have a default set on results? How about for anything nullable?
- Should we add enUSPlatforms to the ReleaseForm object?
Testing Staging-Release
- Edited config/mgerva.ini and saved as config/config.ini
- python repos_setup.py -c config/config.ini -b 1032978 -v 100 -r fennec -u jozeller
- scp -r staging-release/ jozeller@dev-master1.srv.releng.scl3.mozilla.com:
- python staging_setup.py -c config/config.ini -b 1032978 -v 100 -r fennec -u jozeller
- screen
- ctrl+a A "bash/SHIPIT"
- cd /tmp/staging/release-kickoff
- ./shipit.sh
- ctrl+a A "bash/BUILDBOT"
- cd /tmp/staging/staging/
- make start
- ctrl+a A "bash/RELEASERUNNER"
- cd /tmp/staging/
- ./release-runner.sh
- Ran into some trouble so had to make the following changes
- cp ~/config ~/.ssh/config; chmod 400 ~/.ssh/config
- Ran into some trouble so had to make the following changes
Questions
- Can only store events with results=0 because otherwise (name, event_name) has duplicates when results!=0
- status/<release_name> GET could send back a dict like {'Firefox-31.0b5-build1': [...]} rather then {'name': 'Firefox-31.0b5', 'data': [...]}
- Do we want to go ahead and change all 'status' names to 'events'? ie status.html would be events.html
- For some, but colloquially 'status' makes more sense then 'events' for user facing things
- Make sure that repack_complete event messages are being counted as True even after all chunks are counted.
- Simply counting chunks is not enough, this should still pass a True even without chunks
- When submitting a build, when choosing ready, the page reloads to show the new build in the Reviewed tab, but none of the Running tab populates. It requires refreshing the entire page to see.
- Include static copy of bootstrap?
- Perhaps copy just the snippets I would like
- Top level keys complete and progress could be 1 float key
- When using a nonexistent releaseName for status/<releaseName> the GET does not fail, but instead sends a normal JSON response with an empty events value
- Having to split the '+01:00' from sent on pulse messages in order to parse into datetime object for entry into the database. Better way? dateutil is not installed, so that would change dependencies, but from dateutil.parser import parse works great!
- Going to just dump the +01:00 for now, but in the future it could be added by running sent = datetime.strptime('2014-08-08T16:12:59', "%Y-%m-%dT%H:%M:%S") + timedelta(+1)