WebDevITQAMtg09102010

• Ways to get Webdev/IT/WebQA on the same page, concerning versions of libraries and binaries (PHP/Python/MySQL, etc.)
  • Desired/necessary level of formality: pre-release checklist? What determines "sign off"?
    • Day before, pre-release meetings
    • Webdev lead makes the call and does go/no-go decision
• Where the responsibility of application/site performance and stress-testing lies, and when it's needed
  • Still lies with IT and WebDev
• How we can get better at scheduling releases (https://bugzilla.mozilla.org/show_bug.cgi?id=587824 comes to mind)? What's the process? Do we need rules about how many combined large (AMO/SUMO, etc.) and small (Personas, Input) releases we do in a week?
  • Leave it up to WebDev/WebQA to gauge
  • OK with 4pm releases, can't do earlier than 4pm
• We launched our last AMO release with a brand-new Zeus cluster, but didn't test it prior to launch; maybe Wil knew, but could QA have tested, earlier? (There were no problems, but there could've been.)
• How do we ensure we have user-data backed up, pre-push?
• If something goes wrong during a push, what are the criteria (I realize this is hard to talk to, in terms of specifics) by which we decide to roll back? What's the allowed user-impacting downtime? Is it the outage window? If we think we can solve a problem, is it fine to extend past the outage window? Do we need to message our users further, with an ETA, etc.?
  • we don't truly "panic" until the end of the outage window

• Outcomes/followups:
  • try to ensure that push bugs have owners earlier (mrz)
  • chris is rolling out a security-inventory system (mrz)
    • put the app behind VPN and have WebDev learn how to use it, and do inventories (mrz/morgamic)
  • case-by-case basis for sites that need to graduate from VM -> prod (webqa/webdev)
  • set up cron jobs to copy prod DBs -> staging (webqa/webdev to raise issue to IT when it arises)
  • lead developer should know their schemas/run schematic -- to make sure that the schemas are the same between prod/staging (can diff the schemas) (webdev)
  • come up with more useful or app-specific outage pages, and a way/person to update that content with a helpful ETA, etc. (owner needed)
  • try switching the docroot and the DB on a virtual IP for a small site (mrz/webdev)
  • can do pushes Tuesday, Wednesday, Thursday (Wed. only if needed)
  • small content changes can go whenever
  • enable read-only mode on more sites (webdev)