QA/StaffMeetings/QA-Staff-01-25-2012: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(41 intermediate revisions by 17 users not shown)
Line 23: Line 23:


* Usefulness of qa-contribute mailing list -- too much spam recently; what is it used for again? (Aaron)
* Usefulness of qa-contribute mailing list -- too much spam recently; what is it used for again? (Aaron)
* Test Day 1/20 Reflection (Jason)
* (DavidB) Check out [http://powerball.theautomatedtester.co.uk powerball] and try a simple [http://people.mozilla.org/~jmaher/power-ball-10.xpi addon] to earn points for bugs you file and comment on.
* Follow up on team discussion that stephend started last evening
* Virtual Machine image maintenance transitioning from Al to Geo. Want to try setting update schedule at every six months. Is this OK?
* Has QMO handoff happened?


== QA Team Updates  ==
== QA Team Updates  ==
Line 30: Line 35:
** Firefox 10 sign off meeting is today at 2:30 PM in WarpCore
** Firefox 10 sign off meeting is today at 2:30 PM in WarpCore
*** If you know of any blocking issues please raise them to Juan, Marcia, or Anthony
*** If you know of any blocking issues please raise them to Juan, Marcia, or Anthony
* Silent Update
** Default Compatible Add-ons:
*** Huge thank you to every who helped with Default Compatible Add-ons testing; shipping for add-ons back to Firefox 4
** Add-on Hotfix
*** Hotfix add-on is live on AMO for beta users, will enable us to simplify the chemspill process for *some* chemspills
* Crash Investigation
* Crash Investigation
** https://wiki.mozilla.org/Platform/2012-01-24#Stability_Report
** https://wiki.mozilla.org/Platform/2012-01-24#Stability_Report
** Firefox 9.0.1 has a number of issues.
** Firefox 9.0.1 has a number of issues.
* Maintenance and Security (Al)
* Maintenance and Security (ashughes)
** Firefox 3.6.25
** Firefox 3.6.26
***
*** Testing is underway, targeting push to Beta tomorrow, release on Tuesday


=== Browser Technologies (Tony)  ===
=== Browser Technologies (Tony)  ===
*Issues
*Issues
** Tracy taking Services QA Community stewardship.
** Fennec Native is at risk for 1/31 beta date
*** [https://docs.google.com/spreadsheet/ccc?key=0AocUyLHteCtSdEJjT0hCM0ZOb2ZtRHBQdy16WktjSEE&hl=en_US#gid=0 List of things] we need done by beta and release
*Fennec (kbrosnan/aaronmt)
**  
**  
*Fennec (aakash)  
*Sync Client (tracy)
**  
** Android Native Sync code drop .3 is nearly ready. It should go to m-c today or tomorrow.
*Sync (Tracy)
** Add-on sync is bubbling up the channels (Fx 11) gps has added a series of test to TPS for this feature.
**  
** Switching from Mozmill to Sikuli for UI automation of sync
*Test pilot (tracy)
*Test pilot (tracy)
**  
** nothing to report
*Identity (tracy)
*Sync Server (jbonacci, jrgm)
**
** The backlog of deployments is being addressed now that some of the pressure is off OPs for BrowserID work
*** https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120124
*** https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120125
** QA work will begin very soon on handling server-side test requirements for Sync+BrowserID
*BrowserID (jbonacci, jrgm)
** Trains continue to roll through
*** Train 17: Bug 713961 - QA and deploy BrowserID train-2011.12.28 to production
*** Train 18: Bug 715728 - QA and deploy BrowserID train-2012.01.05 to production
*** Train 19: Bug 719243 - QA and deploy BrowserID train-2012.01.18 to production
** Big additions in the current and next train:
*** Support for primaries
*** Support for localization
*** UI changes
** BrowserID team is actively pursuing leads to act as primaries (email domains, etc).
** QA work continues towards client-side automation
** QA is also working with Dev on front-end and back-end unit tests
** Other upcoming activities include build-out and deployment of PHX colo
*Jetpack(?)
*Jetpack(?)
**  
**  
Line 54: Line 84:
** Dria (Deb Richardson) is now officially PM
** Dria (Deb Richardson) is now officially PM
** Talked to Browser ID/WebQA team to figure out how to best tackle the issue of testing Browser ID and having infrastructure to have selenium tests for Pancake to run daily.
** Talked to Browser ID/WebQA team to figure out how to best tackle the issue of testing Browser ID and having infrastructure to have selenium tests for Pancake to run daily.
* webapis (jhammink)
** presently main focus with these is now with marionette on B2G (P2), mainly
***SMS
***Telephony
**Insofar as they tie into work with B2G
* b2G (jhammink)
** now building, test driving marionette against b2g on both devices and emulator; have gaia (frontend) working with marionette, python, and JS
****coming soon - blog post coming for all of the above
**Currently manually testing Telephony, SMS, GAIA UI, very hard!
**Test plan for MWC congress demo (last week Feb)
***test plan will cover mainly functionality (use-cases), (but also reliability, performance, security tbd)
**attending B2G workweek in feb
**Two new interns to onboard
**wanna use Vlad Maniak as a resource - and he has expressed interest!


=== WebQA (Stephen)  ===
=== WebQA (Stephen)  ===
* Affiliates
** have a new contributor, Blayne Wilson.
** setting up rshetty as one of the primary qa contacts on the project
* AMO
* AMO
**
**[https://github.com/mozilla/Addon-Tests/pull/313 Pull request for a paypal test submitted]
**[http://qa-selenium.mv.mozilla.com:8080/job/apps.preview.dev/ jenkins job in place for smoke tests in place of temporary site apps-preview-dev]
* Engagement Projects
* Engagement Projects
**  
** De Todos Para Todos: Phase II release went out, added Portuguese site
** Mozilla Careers is in progress- should release this week
* Input
* Input
**
** Small regression caught by [https://bugzilla.mozilla.org/show_bug.cgi?id=720191 automation].
* Mozilla.com
* Mozilla.com
**
** Shipped 1.2 and 1.3 [https://bugzilla.mozilla.org/buglist.cgi?order=Importance&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&resolution=INCOMPLETE&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&component=www.mozilla.org%2Ffirefox&product=Websites&target_milestone=1.1&target_milestone=1.2&target_milestone=1.3&target_milestone=1.4 bugs]
* MDN
* MDN
**
**Working on [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=mdn_21&sharer_id=403306&list_id=2141702 2.1]
* Mozillians
** [http://goo.gl/mrVqy 1.3 milestone] still in active development
** on-boarding a new contributor, Austin Davis
* Socorro
* Socorro
**
** Code freeze today, shipping [http://goo.gl/6Cz4E 2.4.1 milestone] next Wednesday 2/1/2012.
*SUMO
*SUMO
**
**[http://moxie.jamessocol.com/bugstats/sumo/2012-01-24 1/24] release went out
**[http://moxie.jamessocol.com/bugstats/sumo/2012-01-31 1/31] release in progress


=== Community (Marcia)  ===
=== Community (Marcia)  ===
*  
* QA Community Team meeting today
** https://wiki.mozilla.org/QA/QA_Community/Meetings/2012-01-25
* Community Action Plan
** [https://quality.mozilla.org/2012/01/qa-community-action-plan/ Discussion] to develop an action plan complete with long and short-term goals; '''please participate in the discussion''
* Upcoming Testdays/BugDays/Meetups
* Upcoming Testdays/BugDays/Meetups
**  
** [https://quality.mozilla.org/2012/01/firefox-10-rc-or-beta-6-testday/ Firefox 10 Testday] (juanb)
* Video Work
* Video Work
**  
**  
*QMO.next (Al):
*QMO.next (Al):
**  
**
'


=== QA Automation Service (Henrik, David)  ===
=== QA Automation Service (Henrik, David)  ===
* General
* General
**
** Selenium 2.17 has been released
*Mozmill
*Mozmill
**
** Mozmill 1.5.8 release has been targeted for today (including support for running tests in parallel)
 
====Project updates====
====Project updates====
; Prototype of a Fennec Driver (DavidB)
; CI system for on-demand Firefox builds (Geo)
*
* Window systems turned out to be challenging re: setup
* Currently finalizing job setup to get prototype fully rolled


; New Members and Mentors Pages
; CI system for daily and l10n Firefox Builds (Henrik)
*
* System runs stable within the last week - no fall-outs from Pulse
* Due to lack of documentation for Pulse some issues with queues had to be fixed


; CI system for daily and l10n Firefox Builds (Henrik)
; Automated testing for add-ons default to compatible
*
* Currently a WIP patch is available for testing
* '''Waiting for the final list of add-ons to test including platforms and the MPT machines'''


; Selenium Grid (DavidB)
; MemChaser extension
*
* Need to finish up the logging feature before we can go public on AMO
* '''Working with the JS team on getting an API for garbage collection implemented'''
* '''Collecting ideas for a zombie compartment detector'''
* Feedback from various groups is absolutely positive


; [https://wiki.mozilla.org/QA/TCM Case Conductor]
; [https://wiki.mozilla.org/QA/TCM Case Conductor]
*
* Doing preliminary testing on 0.8 (pure-Django) drop
* Feature list for 0.8 up-to-date in [https://www.pivotaltracker.com/projects/280483# Tracker]
** Everything above the ".8 Feature Complete" bar is expected to be in 0.8 release
* On-track for 0.8 complete on 2/3
* [https://intranet.mozilla.org/QA/Q12012_QAgoals#Case_Conductor Q1/2012 Goals] created.
 
===Apps (David)===
 
* Test Day on 1/20 Results (All)
** We built a well documented software project, with github pages, pydocs ..etc.
** ~7 contributors, ran into issues getting code up and running
** Gained insight on future direction of apps extension automation framework
** Began shift to java + selenium + sikuli + jbehave.
 
* Mochi Tests
** Met with Clint, got details on mochi tests, how to write / where to check them in ..etc.
** Began experimentation process
 
* Mobile (Aaron)
** Reviewing sites on native fennec, for possible Android soup from Webkit -> gecko move.
 
* Dashboard (jsmith)
** In progress of getting a QA Bugzilla Dashboard for Web Apps through Metrics team
** https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=bugzilla/solr/QA&file=QA.wcdf
 
* Goals
** Seed new test framework
** Prepare for MWC.


== Notes, Takeaways and Action Items  ==
== Notes, Takeaways and Action Items  ==
Line 105: Line 199:
=== Today ===
=== Today ===
* Add notes here
* Add notes here
**
Ideas around structure for QA
- no names attach
- general buckets
- general work needing to be done
- not cast in stone
- subject for talks in/out of
- quickly go through it
 
 
high points:
 
Desktop
- Jan 21st.
- 2:30 warpcore: sign off on 10
- chase.com not working ex.  try to catch it all
- general web usage testing
- if there is any bugs, is it blocking?
- plans have been discussed
- mitigating plans
- so far, no wiki for it
- Anthony on 3.6
- in the wiki
 
Mobile
- risk for 1/31
- maybe adjusting the release date for Beta
- Moving parts
- reiterate try to use mobile and get performance in it, responsiveness, pulling up tabs, etc.
- Services side
- new work for back end
- browserid: next 2 trains are big
- b2g
- goal for world congress
- ux demo
- how to flash gecko/gia on a device
 
WebQA
- Affliates
- new contributer
- rajiv is doing onboarding
- Mozillians
- contributer Austin
 
Community
- planning community goals
- Matt approached bmoss for a headset for contributer
- if you want to send a thank you/or tool for a contributer, we will find a way to help
- community giving program
- funding; something small; non specific, just for contributing/in the community; if we get into specifics then it lends to mercenaries.
- studies on compensation.  We don't want mercenaries we want community
 
Automation services
- speed improvements due to support for running in parallel
- memchaser
 
Case Conductor
- 0.8 release
- 1.0 release for the quarter
- ~ 3/1 0.9
- middle of march stabilization
- end of march 1.0
- small projects into case conductor
 
Note:
- rules of snakes
- don't send Bob in a knife to a gunfight
- when you mean a ton of time...
- how much are we going to save ahead of time
- give data for the time, for the bug
- if it doesnt make sense to them, you're not going to persuade them
- give the next level of information
 
Raymond - took over QMO
 
Org:
META groups:
- Community (Marcia)
- triage (bugs) :
- help people log bugs
- pick up the phone; help understand issues to get a solid bug
- QMO
- Automation
- tools : some set that make the right set of tools happen; QA/community can help implement, but no necessarily set it up (permenant)
- tests : need people to actually write the tests in automated tools (nonpermenant/pull to various groups/everyone eventually get there)
 
* Web - Correctness
App Store (UI)
M. O.
Mozillians
Socorro
 
* Server - Performance testing/Load testing/Failover testing/Crypto/Data Store
Sync
Browser ID -> Services
App Store
Socorro
 
* Client (Native; Mac, Win, Android)
Mobile, Desktop
Apps
B2G (might be different)
 
-> we stand behind it because we said it... => shell -> of passing it to waverly.
-> at some point we have to say no.  what willing to stop?
 
--> What's the problem that we're trying to solve?
--> Learn to say "No"
 
- Automation Team versus QA?  Probably no difference.
 
[Clear understanding what the load is / What's going to break?]
 
[Amount of work is limited -> get the automation; maintenance work, = > do it incrementally; maintenance]
 
[Testing/Security testing is in all three buckets]
 
[Security Team]
[Taros : Snappy, Telemetry]
[try to have a standardization - one off ]
 
[strattling buckets?  Community and Web/Server/Client?  ]
 
 
[ Be able to cross train and do what needs to be done; specialty seems broken, expertise should be there though]
 
[ Accountability for the tools? 
- make sure that the group is happy; tool is useful?
- check back with the customer
- break down of communication/management]
 
[ Do it through influence/not authority ]
 
[ Skill sets are different for Web, Server, Client, ex selenium for web, but not client; server doesn't necessary have a ui, JSON thingy, ]
Next Step is to figure out details
 
=== Last Meeting ===
=== Last Meeting ===
* Action Items
* Action Items
**
**

Latest revision as of 19:42, 3 February 2012

« previous mtg | QA Staff Home | next mtg »

QA Staff Meeting Notes January 25, 2012

Meeting Details

 # Every other Weds @ 1:30p Pacific Time - Mozilla HQ, Warp Core 
 #
 # Vidyo:  https://v.mozilla.com/flex.html?roomdirect.html&key=UK1zyrd7Vhym
 # Phone:  +1 650 903 0800 or 650-215-1282 x92 Conf# 95312 (US/INTL)
 #         +1 800 707 2533 (pin 369) Conf# 95312 (US Toll Free)
 #         +1 416 848 3114 x92 Conf# 95312 (Canada)
 # IRC:    irc://irc.mozilla.org:6697/#qa

QA Org Items

  • New Hires & Interviews
    • item
  • Upcoming Events
    • item
  • PTOs

Discussion Items

  • Usefulness of qa-contribute mailing list -- too much spam recently; what is it used for again? (Aaron)
  • Test Day 1/20 Reflection (Jason)
  • (DavidB) Check out powerball and try a simple addon to earn points for bugs you file and comment on.
  • Follow up on team discussion that stephend started last evening
  • Virtual Machine image maintenance transitioning from Al to Geo. Want to try setting update schedule at every six months. Is this OK?
  • Has QMO handoff happened?

QA Team Updates

Desktop Firefox (Juan)

  • Fx.Next
    • Firefox 10 sign off meeting is today at 2:30 PM in WarpCore
      • If you know of any blocking issues please raise them to Juan, Marcia, or Anthony
  • Silent Update
    • Default Compatible Add-ons:
      • Huge thank you to every who helped with Default Compatible Add-ons testing; shipping for add-ons back to Firefox 4
    • Add-on Hotfix
      • Hotfix add-on is live on AMO for beta users, will enable us to simplify the chemspill process for *some* chemspills
  • Crash Investigation
  • Maintenance and Security (ashughes)
    • Firefox 3.6.26
      • Testing is underway, targeting push to Beta tomorrow, release on Tuesday

Browser Technologies (Tony)

  • Issues
    • Tracy taking Services QA Community stewardship.
    • Fennec Native is at risk for 1/31 beta date
  • Fennec (kbrosnan/aaronmt)
  • Sync Client (tracy)
    • Android Native Sync code drop .3 is nearly ready. It should go to m-c today or tomorrow.
    • Add-on sync is bubbling up the channels (Fx 11) gps has added a series of test to TPS for this feature.
    • Switching from Mozmill to Sikuli for UI automation of sync
  • Test pilot (tracy)
    • nothing to report
  • Sync Server (jbonacci, jrgm)
  • BrowserID (jbonacci, jrgm)
    • Trains continue to roll through
      • Train 17: Bug 713961 - QA and deploy BrowserID train-2011.12.28 to production
      • Train 18: Bug 715728 - QA and deploy BrowserID train-2012.01.05 to production
      • Train 19: Bug 719243 - QA and deploy BrowserID train-2012.01.18 to production
    • Big additions in the current and next train:
      • Support for primaries
      • Support for localization
      • UI changes
    • BrowserID team is actively pursuing leads to act as primaries (email domains, etc).
    • QA work continues towards client-side automation
    • QA is also working with Dev on front-end and back-end unit tests
    • Other upcoming activities include build-out and deployment of PHX colo
  • Jetpack(?)
  • Pancake (nhirata)
    • Infrastructure changes, UX changes still occurring with nothing to test on the front end.
    • Dria (Deb Richardson) is now officially PM
    • Talked to Browser ID/WebQA team to figure out how to best tackle the issue of testing Browser ID and having infrastructure to have selenium tests for Pancake to run daily.
  • webapis (jhammink)
    • presently main focus with these is now with marionette on B2G (P2), mainly
      • SMS
      • Telephony
    • Insofar as they tie into work with B2G
  • b2G (jhammink)
    • now building, test driving marionette against b2g on both devices and emulator; have gaia (frontend) working with marionette, python, and JS
        • coming soon - blog post coming for all of the above
    • Currently manually testing Telephony, SMS, GAIA UI, very hard!
    • Test plan for MWC congress demo (last week Feb)
      • test plan will cover mainly functionality (use-cases), (but also reliability, performance, security tbd)
    • attending B2G workweek in feb
    • Two new interns to onboard
    • wanna use Vlad Maniak as a resource - and he has expressed interest!

WebQA (Stephen)

Community (Marcia)

'

QA Automation Service (Henrik, David)

  • General
    • Selenium 2.17 has been released
  • Mozmill
    • Mozmill 1.5.8 release has been targeted for today (including support for running tests in parallel)

Project updates

CI system for on-demand Firefox builds (Geo)
  • Window systems turned out to be challenging re: setup
  • Currently finalizing job setup to get prototype fully rolled
CI system for daily and l10n Firefox Builds (Henrik)
  • System runs stable within the last week - no fall-outs from Pulse
  • Due to lack of documentation for Pulse some issues with queues had to be fixed
Automated testing for add-ons default to compatible
  • Currently a WIP patch is available for testing
  • Waiting for the final list of add-ons to test including platforms and the MPT machines
MemChaser extension
  • Need to finish up the logging feature before we can go public on AMO
  • Working with the JS team on getting an API for garbage collection implemented
  • Collecting ideas for a zombie compartment detector
  • Feedback from various groups is absolutely positive
Case Conductor
  • Doing preliminary testing on 0.8 (pure-Django) drop
  • Feature list for 0.8 up-to-date in Tracker
    • Everything above the ".8 Feature Complete" bar is expected to be in 0.8 release
  • On-track for 0.8 complete on 2/3
  • Q1/2012 Goals created.

Apps (David)

  • Test Day on 1/20 Results (All)
    • We built a well documented software project, with github pages, pydocs ..etc.
    • ~7 contributors, ran into issues getting code up and running
    • Gained insight on future direction of apps extension automation framework
    • Began shift to java + selenium + sikuli + jbehave.
  • Mochi Tests
    • Met with Clint, got details on mochi tests, how to write / where to check them in ..etc.
    • Began experimentation process
  • Mobile (Aaron)
    • Reviewing sites on native fennec, for possible Android soup from Webkit -> gecko move.
  • Goals
    • Seed new test framework
    • Prepare for MWC.

Notes, Takeaways and Action Items

Today

  • Add notes here

Ideas around structure for QA - no names attach - general buckets - general work needing to be done - not cast in stone - subject for talks in/out of - quickly go through it


high points:

Desktop - Jan 21st. - 2:30 warpcore: sign off on 10 - chase.com not working ex. try to catch it all - general web usage testing - if there is any bugs, is it blocking? - plans have been discussed - mitigating plans - so far, no wiki for it - Anthony on 3.6 - in the wiki

Mobile - risk for 1/31 - maybe adjusting the release date for Beta - Moving parts - reiterate try to use mobile and get performance in it, responsiveness, pulling up tabs, etc. - Services side - new work for back end - browserid: next 2 trains are big - b2g - goal for world congress - ux demo - how to flash gecko/gia on a device

WebQA - Affliates - new contributer - rajiv is doing onboarding - Mozillians - contributer Austin

Community - planning community goals - Matt approached bmoss for a headset for contributer - if you want to send a thank you/or tool for a contributer, we will find a way to help - community giving program - funding; something small; non specific, just for contributing/in the community; if we get into specifics then it lends to mercenaries. - studies on compensation. We don't want mercenaries we want community

Automation services - speed improvements due to support for running in parallel - memchaser

Case Conductor - 0.8 release - 1.0 release for the quarter - ~ 3/1 0.9 - middle of march stabilization - end of march 1.0 - small projects into case conductor

Note: - rules of snakes - don't send Bob in a knife to a gunfight - when you mean a ton of time... - how much are we going to save ahead of time - give data for the time, for the bug - if it doesnt make sense to them, you're not going to persuade them - give the next level of information

Raymond - took over QMO

Org: META groups: - Community (Marcia) - triage (bugs) : - help people log bugs - pick up the phone; help understand issues to get a solid bug - QMO

- Automation - tools : some set that make the right set of tools happen; QA/community can help implement, but no necessarily set it up (permenant) - tests : need people to actually write the tests in automated tools (nonpermenant/pull to various groups/everyone eventually get there)

  • Web - Correctness

App Store (UI) M. O. Mozillians Socorro

  • Server - Performance testing/Load testing/Failover testing/Crypto/Data Store

Sync Browser ID -> Services App Store Socorro

  • Client (Native; Mac, Win, Android)

Mobile, Desktop Apps B2G (might be different)

-> we stand behind it because we said it... => shell -> of passing it to waverly. -> at some point we have to say no. what willing to stop?

--> What's the problem that we're trying to solve? --> Learn to say "No"

- Automation Team versus QA? Probably no difference.

[Clear understanding what the load is / What's going to break?]

[Amount of work is limited -> get the automation; maintenance work, = > do it incrementally; maintenance]

[Testing/Security testing is in all three buckets]

[Security Team] [Taros : Snappy, Telemetry] [try to have a standardization - one off ]

[strattling buckets? Community and Web/Server/Client? ]


[ Be able to cross train and do what needs to be done; specialty seems broken, expertise should be there though]

[ Accountability for the tools? - make sure that the group is happy; tool is useful? - check back with the customer - break down of communication/management]

[ Do it through influence/not authority ]

[ Skill sets are different for Web, Server, Client, ex selenium for web, but not client; server doesn't necessary have a ui, JSON thingy, ]


Next Step is to figure out details

Last Meeting

  • Action Items
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/StaffMeetings/QA-Staff-01-25-2012&oldid=394012"