- 1 QA Staff Meeting Notes Oct 12, 2011
- 1.1 Meeting Details
- 1.2 QA Org Items
- 1.3 Discussion Items
- 1.4 QA Team Updates
- 1.5 Notes, Takeaways and Action Items
QA Staff Meeting Notes Oct 12, 2011
# Every other Weds @ 1:30p Pacific Time - Mozilla HQ, Warp Core # # Vidyo: https://v.mozilla.com/flex.html?roomdirect.html&key=UK1zyrd7Vhym # Phone: +1 650 903 0800 or 650-215-1282 x92 Conf# 95312 (US/INTL) # +1 800 707 2533 (pin 369) Conf# 95312 (US Toll Free) # +1 416 848 3114 x92 Conf# 95312 (Canada) # IRC: irc://irc.mozilla.org:6697/#qa
QA Org Items
- New Hires
- John H, Mobile QA automatoin - Already here 4 weeks (say hi!)
- John M, Sevices QA team - ETA 11/7
- Upcoming Events
- Mobile summit Mon 10/17/11. Attendees: Aaron, Kevin, Naoki, Martijn
- Ioana Chiorean and Vlad Maniac of Softvision will visit Mozilla Office between OCT 19-27
- GTAC meet & greet mixer https://mozqa.sync.in/mozwebqa-GTAC-party-planning
- [unconfirmed] Naoki and Tony to represent QA at MozCampAsia. Proposals still in flux. Nov 19-20
- [juanb]: Oct 13, 14; wedding in Seattle (my brother's).
- tchung: oct 16-28, pto, hawaii.
- Dave Hunt: Any day now, for 2 weeks.
- Henrik Skupin: Oct 13, 14
- Tracy: afternoon of 10/13 and all day 10/14
- Cameron: Oct 20, 21
- overview of 7.0.1 postmortem - Anthony
- general take-away:
- QA needs to be more aggressive in helping to reproduce bugs, diagnose severity and prevalence
- PM needs to be more aggressive in making sure issues are visible to QA
- Update testing strategy cannot rely solely on automation
- Updates strategy going forward -- see QA project page
- will be two tiered: automation and manual testing with various rich profiles
- time-to-market should be roughly the same given that builds now come out Wednesday morning and some of this testing can be done in parallel (manual on betatest, automation on releasetest)
- general take-away:
- q3 goals overview https://intranet.mozilla.org/QA/Q32011_QAgoals
- team leads any take aways from Q3?
- q4 goals review https://intranet.mozilla.org/QA/Q42011_QAgoals
- matt added the top level org goals
- add bhags to team goals (etherpad for each team)
- scrumpads discussion
- move to new system
- overview of usage
- pass down from directors meeting
- native fennec
- Progress on Team Slides?
- Would like these in advance of some of the upcoming events.
- 2-3 Slides that show easy paths to contribute to your particular team area
QA Team Updates
Desktop Firefox (Juan, Anthony)
- Firefox 8 (juanb): Testing in progress, to be released Friday morning.
- Firefox 9 (ashughes): No status update
- Firefox 10 (juanb): Heads up on probable compatibility issues due to two digit version. Web Compatibility test run in planning phase.
- Crash Status and Investigation
- Maintenance and Security ESR (Al)
- No new updates at this time.
- New Lab Operating Systems Available
- Windows 8 Developer Preview
- Windows 2000
Browser Technologies (Tony)
- Native Fennec for Android project announced this week as a top Priority, starting Firefox 10. QA brainstorm here
- The "503 meltdown of 2011" post mortem is scheduled 10/13
- Fennec (Aaron/Kevin)
- Performance Benchmark baseline to be established
- Crash reporting for Fennec working with Socorro team
- biweekly newsletter being sent out
- Toronto Mobile Workweek next week
- Sync (Tracy)
- Sync fell over last week due to servers not being prepared for load increase of Instant Sync released in Fx7. Client was also not correctly respecting back-off requests from the server which made it difficult for OPs to repair the servers on the fly. Post-mortem scheduled for 10/13
- Setup process has been somewhat simplified and Mobile devices can now initiate device pairing process before a sync account is even setup.
- Mihaela from Softvision is getting up to speed with weekly client train cycles and is starting to take on some test case writing.
- Test pilot (tracy)
- Test Pilot for Mobile was released last week as was updated version for desktop
- Sync Server (jbonacci)
- Deployments have been delayed for a couple of weeks due to the sync/503 meltdown and other issues
- Upcoming (planned) deployments:
- Bug 693385 - deploy updated server-core (2.6.x) to syncstorage
- server-node-assignment rpm-1.6-1
- Something for Account Portal
- Bug 686485 - roll out membase
- BrowserID (jbonacci)
- BrowserID has taken a giant step forward with the current Beta train: 10/06/2011
- New functionality, certs support, IE9 support, and, of course, the new UI
- BrowserID Test Day 1 is coming up this Friday, 10/14/2011
- ==> QA NEEDS MODERATORS! ... https://etherpad.mozilla.org/testday-20111014 <==
- BrowserID has taken a giant step forward with the current Beta train: 10/06/2011
- Pancake (nhirata)
- .17 Release
- backend algorithm is being revamped
- Places is being used
- AMO 6.2.6 went live 10/06
- goals are set
- working on paypal testing for marketplace
- refactoring page objects for themes
- Affiliates - on track to dark launch on 10.13.11
- Input - Q4 bug triaging in-progress
- working on 4.1 release
- 1.3 milestone shipped 10.11.2011
- 2.3 milestone shipped on 10.10.11
- Softvision is very active on the project
- Automation complete, can be seen in Jenkins: http://qa-selenium.mv.mozilla.com:8080/view/Mozillians/
- Pushed Mozillians 1.0.1 last Thursday
- Community invites submitted for both MozCamp Europe and MozCamp Asia.
- Second Draft of QA Sig - https://docs.google.com/document/d/1DZBKZl0mSXZjwcwFbnRmx1VfSZ8vcK2GMJbi9W-fwKA/edit?hl=en_US&pli=1
- Mozilla Festival - November 4-6, 2011, London.
- marcia and henrik attending
- Mozilla Stewards Questions
- If interested, please attend the Contribute Group meeting tomorrow at 10 AM Pacific as I have asked David Bowsell to clarify if they want to have stewards across various teams.
- Video Work
- Revised nightly video has been completed and a bug has been filed to get it upon the Nightly First Run page.
- QMO.next (Al):
- Should be updating Buddypress to version 1.5 with an updated theme on Tuesday or Thursday next week, depending on Craig Cook's time.
- We're pulling from Github to our new Dev server every 10 minutes automatically now, making testing changes easier.
QA Automation Service (Henrik, David)
- Team was successful last quarter by accomplishing all listed team goals
- Q4 goals have been finalized: https://intranet.mozilla.org/QA/Q42011_QAgoals#QA_Automation_Services_Goals
- Henrik will participate in Mozilla Festival and MozCamp Europe in Berlin, proposals have been submitted
- Selenium 2.8 has been released
- Mozmill 1.5.5 has been released
- Mozmill 2.0rc1 has been released
- A-team has started work on Mozmill 2.1 (e10s support)
- Project pages for new projects have to be created and leads have to be assigned:
- Fully automated Mozmill tests for releases
- Prototype of Fennec driver
- Community Documentation
- CI system for daily and localized Firefox builds
- Selenium Grid for Android and iOS
- Long-term Goals for 2012
- Marching forward to get our staging instance up and running with IT. Keep working through platform and logistical issues as they come up.
- Working toward the 0.6 milestone for the uTest Demo
- Demo'd the product to the uTest designer they just hired.
- Carl Meyer will visit Mt. View and be in-da-house Oct 17th & 18th.
- Q4 goals done, but haven't yet gone over them with Matt.
Notes, Takeaways and Action Items
- please sign up to help with BrowserID Testday this friday, Oct 14
- mevans to pool together ideas for next meeting about this quarter and future
- marcia to create a template for team slides
- bmoss to chase down who is going to mozcamp (w/ mary colvig)
- (everyone) Please sign up for test day on friday BrowserID
- Action Items
- draft of Q4 goals for teams by tuesday
- draft of Q4 goals for individuals by end of week
- add feedback to workweek for those that haven't commented yet
- mevans: schedule of post mort
- how to respond on emergency situation; checklist to be followed/consulted
- marlenac to post as QMO event about E. Hendrickson's brown bag
- all: overall presentation; each group does it's talk
- Marlena/David : event planning for GTAC
- Mevans : ask James Whittiker for brown bag
- David C: Blog post about his unit test development framework
- Juan : update tests for chemspill issue
- Anthony taking ownership -- see tracking page
This Meeting Raw Notes
need to change the QA meeting to vidyo - bullet points : Gtac mixer, email david burns, marlena, mattb, or mattevans mbrandt|mtg: nhirata|meeting: here's the url to the mixer -> http://www.gtac.biz/home/announcements/meetgreetmixer - PTO: look at wiki - 7.0.1 Post morteum - general take away : QA need to be more aggressive on triage - PM needs to help to make this more visual, and make QA more aware of these issues - our own part, update has been 100 % on automation - need to adopt a similar model to functional testing - automation + manual testing - ashughes to take testing / strat for update testing - rich profiles testing - shoe horning for manual testing? - shouldn't effect that much - build gets 24 hrs earlier - can do manual testing in parallel to automation - effects should be negligible - why no automation? - under some testing such as addon testing? - testing with different plugin version - richer profile testing, is necessary - they don't have to be linked to the hip - release testing can be automated - should investigate - updater code changes - some things matter on the updater, some don't - real time makes sense - should check, Rob Strong/Rel Eng - update testing has always been a concern - renewed - also important with silent update - test plan concerns ? - Ashughes has taken over as a lead - automation strat - update testing requirements need to be nailed down - much richer profile testing - Q3 goals wrapped up - action item from the last staff meeting for team leads to give update - from the team leads, wrap up/take aways from Q3 goals: - Stephen : web apps, etc utilized dave hunt's plugins, helped out quite a bit - LDAP to jenkins hook up - open up that even further with IT - automation specific - Juanb : Q3 goals met, relied to a large extent on Softvision - going forward, we need to change things to scale for the future - keeping up but will end up getting tougher - Tony : mobile side, following Firefox lead - did miss a goal, for crowd source tool - did get the test working, but not logging - need to get logging working and deploy to crowd - downtime for features on sync - plumbing on operation - instant sync landed - weekly train model for client/server for sync - porting some of the functional load test - not integrated to production - support web apps team on the mobile side - David/Henrik : we expect to get more things done - follow up : that we set goals that can be accomplished - new team, setup of the new team on wiki - get more community members - prototype for the fennec driver - endurance test - useful things to get in the test run - endurance test for memshrink - Case Conductor: - wrap up: dev project - .4/.5 release shipped - great progress - 2 different items weren't hit - making some features as goals is ill advised - reprioritize features - security issues - internal server deployment - user research done from rebecca that was great - some community use it, and are getting feedback - Q4 goal = CC 1.0 - Q3 was very productive - we responded, we have good highlights, lots to learn, lots to carry over - top level QA goals are what we publish - team goals down to personal goals - altering the processes, a bit - top level QA goals - been a little tough because Eng focus has changed - what are the proper goals that align with Dev has been tough - Bob : direction from all hands/work week that has been completed - that gives a good guideline - need to put goals together with what QA thinks is important - and then check alignment - we should have enough information to say what's important and what's not: https://intranet.mozilla.org/QA/Q32011_QAgoals - we did publish draft goals: : https://intranet.mozilla.org/QA/Q42011_QAgoals top level goals - 1st on Tony's group : - mobile test harness for crowd sourcing : - take reftest/crashtest => remotely put that on server, and use an addon and fire off and run the test on the device - report failures back to server - get a baseline on real failures on real devices - who on mobile team will be working on this when we go native - mw22 working with jmaher - if they change the framework w/native then we're going to have to go a different route - should we stop? 10 and on is the earliest - we're talking 12 weeks, and new front end: how much effort, and how much benefit, let's stop here. let's get 8/9 out and then get ready from 10 - 2 weeks away for addon working; talk offline; giving this goal last week, we didn't change it early enough - need to take a look at this for change in direction - Marcia : - remote sig : draft link to draft doc (formerly embassador ) https://docs.google.com/document/d/1DZBKZl0mSXZjwcwFbnRmx1VfSZ8vcK2GMJbi9W-fwKA/edit?hl=en_US&pli=1 - develop community in different countries. sig helps grow the community. - overall one or one for each team? - second draft circulated - update testing as a high priority - automate what we can do - automated - functional testing - been a phased in approach of what we do for update testing - one button initiative - controlled by jenkins - David Burns : - this could change for native fennec - will go to toronto for the tooling will work for fennec front end - case conductor : 1.0 ; set as basis of what we do for test /test execution At the team level and personal goals: - flexibility so that as we work on these goals that we have an element of agility - top level org goals, team goals, personal goals can have adjustment - Dave Hunt's goals are impressive : stretch and super stretch goals - we look to him to complete the main goals,... stretch goals awesome, super stretch walk on water - backlog as we go forward if the super stretch aren't hit. - mevans likes that. - Bob's expectation : QA team is head/shoulders ahead. rest of Eng should follow suit - kudos. level of thoughts are good - great opportunity to do a hard reset - is what we're doing really working, what can we do more of, what can we do better - it's obvious that we aren't big enough to handle this ourselves - need to work smarter, scalable, and more community - if we don't we're going to "die" - process review and refinement? - doesn't necessarily have to be top level goals - could be about business - external facing as well - need to get things done - pushing the rope right now for QA, need to figure out how to do this better - let's for real do it now. - capture the idea of the hard reset? - look at what we're doing, plan our work and then work our plan - if we don't do that at some point, our day jobs will overcome us - looking at each of the teams, and task what does 2012 look like? - what is going to give us the time to - this quarter is a pivotal quarter - time frame is tight - fennec, open web apps, b2g, etc. is something that we're going to have to adjust to - given the team and the curtailing of hiring, we're going to have to make some hard choices - how do we handle expectations - what is the proper course of action : smarter, focus - most focused - prioritization on how much we can make an impact - meeting next week : bring in ideas - mevans to pool together ideas - this quarter and future - there is a fair amount of planning that has been done : qa automation, infrastructure - need to have up on the board - complete reset ideas - dare to be great - find a way to find bugs and make the fixes happen - move forward and throw stuff on the table - the energy from the meeting might be worth the effort; mail could work, but meeting might be better - train coming/light of the end of the tunnel - chaotic : rapid release, we're doing it - define the scope of the problem, try to come up with solutiosn to the problem - solutions beyond that - understanding what the problem is - initiaves that the company doing is rapidly outstripping what we're doing - services - platform - desktop - fennec - more chaos coming our way - need to adopt testing in bigger chunks? - down to the hard choices, where do we focus - how do we make the services that we provide the most effective? - use that evaulation currently - given the context on what's being released, how do we handle it - what's on our plate currently - how big of a plate can we have - how ... etc. - dilemma : - try to examine the old way of doing things is hard - mozmill, mochitest runnning on desktop - should we be looking at that? - where we want to go? - trying to look at the old way - new things are developing are based on the old ways of doing things - hit the reset, and look where we need to go - understanding the historical thing - mobile may be very right for a reset - not much on automation there comparitive - from traditional FF approach, whole new tools might need to be evaluated - time/effort/money already put in, ... finding a happy medium - how do we approach current things, what's coming down the pike, and does it map - this is going to be our new approach - current approach - how to evaluate it - what is the sensible approach - what is happening coming down the pike - moving forward : - advancement in what mozilla does, and is investing is ... is changing all the time - evaluation and moving forward - where do we want to be excellent - services, browser id, app sync, sync service - OWA, fennec, desktop - where do we want to have excellent test framework and all around - we have to start somewhere. - where do we want to start and where do we want to put our efforts. - align with company goals - we should be listening to the company goals - paste link into the channel : http://www.t3.org/tangledwebs/07/tw0706.html Expectations of what this team can be Scrumpad: keep as is : looking for information that you want to relay to other team members ( not daily status ) - moving to new server - native fennec : huge priority - meeting with google and win 8 restrictions - problems with team slides - useful for mozcamp asia - take away : easiest way to get someone involved on your team - provide template please? : marcia action item - events are coming up; can be repurposed, tweaked, modified - slides from vancouver? some might, but need crisper slides - post mortem for sync server meltdown - community invites have gone out, no update for staff for mozcamps; hoping that comes through - proposal : henrik, marcia for festival, but nothing from the mozcamps - request : mary colvig > bob to chase that down - automation : Action Items: - mevans to pool together ideas for next meeting about this quarter and future - marcia to create a template for team slides - Bob to chase down mozcamp (mary colvig) - Please sign up for test day on friday BrowserID