AMO:Releases/5.11.9

Post Mortem

What didn't go well

• L10n didn't freeze
• We committed changes right up until the last second...and then some (.4, anyone?)
  • poor scheduling
  • Need more time for QAing and/or feedback
  • QA should not explode 4 days before release
  • initial code quality was (apparently) low
• felt like everyone was super-stressed
• Implementations didn't match mocks, resulting in lots of post-freeze bugs
• Not all developers were present for the release

What went well

• We launched without rolling back!
• Many improvements suggested in 5.11's post-mortem implemented.
• Email script worked well and was effective
• No rush of critical bugs after launch

How we can improve

• Be more flexible in allowing QA to delay a release.
• Formally look at mocks together
• QA and product will re-open bugs if things don't match mocks

Per-person notes

fligtar

• launched without rolling back
• didn't have a bunch of follow-up bugs
• email script went well
• implementations didn't match mocks
• glad that we hit deadlines and didn't move back
• not all developers present for release
• preview was down a lot

potch

• got it out the door and launching was important and glad we did
• unfortunately that we were qa'ing until the hour before and making changes until the hour before
• liked having gratification of a lifehacker blog post
• turn on/off releases are difficult
• better procedures for large changesets
• lots of last minute bugs
• that affects next release
• level of stress/pressure at end of release
• 3 way tension between engineering, product, qa for launch -- decision process taking into account wasn't perfect
• disagreement on importance of launching vs. full qa cycle

jbalogh

• schedule was right with djangocon the next week
• needed to get things out there
• everything exploded 4 days before the release
• explosions should happen after 5.11.8 because a lot of stuff went out already
• agrees with krupa about the bloat of site. doesn't enjoy looking at some parts of the site because there's so much going on. doesn't feel supportive of some of the things that we do and doesn't feel like they're helping people download add-ons. features like collection contributors, email sharing, collection voting. makes it difficult to work on the site and doesn't think it's good for our users. (potch: resource strapped team should consider not doing these things) shares: jeff, dave, potch, krupa, chowse, jorge to some extent

krupa

• new collections looks really good
• likes that devs were very responsive and fixed almost all major bugs filed
• had lots of help from fligtar qa'ing
• admits that in the end it was a good idea to push
• 5.11.8 was 140 bugs a week before
• so many traceback errors, and could only find bugs when other traceback bugs were fixed
• taking in regression-prone fixes at the last minute, like search bugs
• 5.11.9 had 100+ bugs. 2 huge releases week by week is not really smart
• concerned that site is getting content heavy
• we're very good in terms of release process compared to last year. we should give everyone credit for that. thinks we're being a little too critical about 5.11.9 because of our chain of successful releases.

davedash

• fun working with potch. could make something shitty that worked and then potch would make it pretty
• collections looks a lot better and is simpler.
• whole qa thing is always going to be an issue. it sucks to have changes up to last minute but doesn't know of a better way to do it.
• waiting a long time to release after doing weekly releases loses momentum

clouserw

• large push that was delayed once or twice
• can't flag bugs that are going to change

stephend

• deviance from mocks made it difficult to know what to do to make stakeholders happy
• mocks don't take into account interactions of features.
• everyone was stressed and didn't think we were ready to push but had turning point when talking to us
• only wil knew ahead of time that we were switching to new zeus cluster