MozillaWiki

QA/Browser Technologies/JulyWorkWeek2011: Difference between revisions

Page Discussion

QA/Browser Technologies/JulyWorkWeek2011 (view source)

Revision as of 01:52, 2 August 2011

12,456 bytes added ,  2 August 2011
m
→‎Action Items
 
(29 intermediate revisions by 5 users not shown)
Line 7: Line 7:
|- valign="top"
|- valign="top"
| '''Date:'''  
| '''Date:'''  
| June 20th-25, 2011
| July 18th-22, 2011
|- valign="top"
|- valign="top"
| '''Attendees:'''  
| '''Attendees:'''  
Line 14: Line 14:
| '''What:'''  
| '''What:'''  
| Browser Technologies QA Workweek
| Browser Technologies QA Workweek
|- valign="top"
| '''Main Scrumpad:'''
| http://mozqa.sync.in/bt-work-week-july-2011
|}
|}


= Meeting Space =
= Meeting Space =


Offsite: Half moon bay
Offsite:  
Onsite: Zombocom
  [http://maps.google.com/maps?q=171+Coronado+Ave,+half+moon+bay+ca&oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&hq=&hnear=0x808f739c6551fea1:0x83f656cbb6bd1730,171+Coronado+Ave,+Half+Moon+Bay,+CA+94019&gl=us&ei=HIgkTvOOM5CqsAP64p1L&sa=X&oi=geocode_result&ct=title&resnum=1&ved=0CBUQ8gEwAA 171 Coronado Ave, HMB]
  Leslie: (650) 703-8993
<br>Onsite: Zombocom
 
= Agenda  =
= Agenda  =


Line 54: Line 59:
== Monday  ==
== Monday  ==


2pm: Breakdown for the week.  Start a idea scrumpad, lay out the expectations
*2pm: Breakdown for the week.  idea scrumpad: http://mozqa.sync.in/bt-work-week-july-2011
http://mozqa.sync.in/bt-work-week-july-2011
*5pm: Head to Offsite.  See scrumpad for details
5pm: Head to Offsite.  See scrumpad for details
*8:30pm: Dinner at [http://maps.google.com/maps?oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&q=sam%27s+chowder+house&fb=1&gl=us&hq=sam%27s+chowder+house&hnear=0x808fb7495bec0189:0x7c17d44a466baf9b,Mountain+View,+CA&ei=7LskTr_3FYTEsgaC9IjBCQ&sa=X&oi=local_result&ct=image&ved=0CAQQtgM&cid=0,0,2828005579592320106 Sam's Chowderhouse], reservations under Tony


== Tuesday  ==
== Tuesday  ==


*8am: Mobile Waverley Call [https://wiki.mozilla.org/QA/Fennec/Waverley/2011-07-19 (Notes)]
*8am: Mobile Waverley Call [https://wiki.mozilla.org/QA/Fennec/Waverley/2011-07-19 (Notes)]
*9am : Large Group Introduction & Discussions
*9am : Large Group Introduction & Discussions ( [http://mozqa.sync.in/vision-casting Vision Casting] )
*10am: Mobile Discussion / Services Work Session
*10am: Mobile Discussion / Services Work Session ( [http://mozqa.sync.in/community-discussion Community Discussion] )
*11am: Mobile Discussion / Services Work Session
*11am: Mobile Discussion / Services Work Session ( [http://mozqa.sync.in/mobile-automation Mobile Automation] )
*12pm - 12:45pm: Lunch
*12pm ~ 12:32 Mobile Discussion ([http://mozqa.sync.in/waverly-mobile Waverly Mobile])
*1pm: Mobile Discussion / Services Work Session
*12:32 pm - 1:30pm: Lunch
*2pm: Services Discussion / Mobile Work Session
*1:30pm ~ 2:30 : Team Building Exercises
*3pm: Services Discussion / Mobile Work Session
*2:30pm ~ 4:00 pm : Mobile Work Session ([http://mozqa.sync.in/mobile-coffeetalk mobile coffetalk]) /  Services Discussion
*4pm: Services Discussion / Mobile Work Session
*4pm: Services Discussion / Mobile Work Session ( [http://mozqa.sync.in/bt-lightningtalks Lightning Talk Notes])
*5pm: Half Moon Bay fun, Group Picture
*5pm: Half Moon Bay fun, Group Picture  
*7pm: Dinner at [http://maps.google.com/maps?q=131+Mirada+Rd+,+half+moon+bay,+ca&oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&hq=&hnear=0x808f739c6d2e7fcb:0xc8ef1addb552e514,131+Mirada+Rd,+Half+Moon+Bay,+CA+94019&gl=us&ei=WLskTpyeOMTvsgbSzZmeCQ&sa=X&oi=geocode_result&ct=image&resnum=1&ved=0CBYQ8gEwAA Miramar Beach Restaurant], reservations under Tony


== Wednesday  ==
== Wednesday  ==
Line 80: Line 86:
== Thursday ==
== Thursday ==


*12pm: Lunch for Aakash
*11am: Sync Bug Triage
*2pm: BT Project demo,  (video recorded, 10 Fwd)
*12pm: Farewell Lunch for Aakash.  Pho Garden on Castro
*2pm: BT Project demo,  (video recorded on airmozilla, 10 Fwd)
*3pm: Sync Server Unit, Load, Automated Testing: Dev/Ops/QA Discussion
*3pm: mobile release test planning (aaron, kevin)
*TBD: TPS vs. Funkload for Sync Server API automation/smokes: James, Owen, Jonathon, other interested parties


== Friday  ==
== Friday  ==


* TBD
* TBD: TPS vs. Funkload for Sync Server API automation/smokes: James, Owen, Jonathon, other interested parties
* TBD: other stuff
* Tracy returns to KC (7:10 am flight)
* Aaron returns to TO (noon flight)


= Action Items  =
= Action Items  =
Items that the group felt would merit followup on
 
Collaborating Notes from Workweek (7/25)
 
#'''Vision &amp; Goals'''
#*Summary:
#**Responsible for emerging technologies and environments like Sync, Mobile, Experimental Lab projects (Identity, Share, WebApps), and server environments
#**More resources to help with building out automation, support for new projects, and a better usage of tasks like project investigation, more exploratory testing, interacting with other teams, and defining processes
#***Less focus on routine tasks like regression testing, and beefing automation while leveraging outsource tools
#*Takeaways:
#**'''(Tony)''' Defining Process: what is the future of sync and mobile in respect to Browser Tech?
#**'''(Tony)''' Come up with a BT service agreement for emerging projects, that's somewhat carbon copied to hand off to teams
#**'''(Tony)''' define our goals to other teams (marketing, support, devs, l10n)
#*Sandboxed for Future:
#**What are our responsibilities, current vs future? As a team?
#**What's our hiring and resourcing plan?
#'''Community'''
#*Summary:
#**Community engagement is a challenge for Mobile and Services. Both have very small userbase now.
#**provide a clear list of tasks and a regular schedule:
#***QMO cleanup (owen and rbillings are asking around)
#***Testday posts (send this earlier! also, needs to have direct list of tasks for those that want it, but exploratory for others)
#****there should be an assistance from dev&nbsp;: ideally it would be nice if they attended events
#**Have a playground environment for experimenting with client/server testing and automation
#***Mobile can be device anywhere, create scripts for easier and faster setup
#***Services would be a QA infrastructure
#*Takeaways:
#**'''(aaron, tracy)''' use Moz Reps; additional means of communications
#***other focus channels (ie Reddit, selenium, Android)
#**provide a test environment that's easy accessible from outside
#***'''(kevin, aaron)''' Device Anywhere for mobile
#***'''(tracy, james)''' Sync server with access for Services)
#**'''(All)''' Clearer Testday posts, earlier, and list of tangible tasks to execute
#***'''(All)''' Include Devs in testday channels, and ask them to be strategic on what to do
#**'''(james, owen, tracy)''' Setup VM environments and documentation for services. Video Sync help
#*Sandboxed for Future:
#**have a forum, survey, summary of bugs, feedback, summary of bugs, list of tasks outside of IRC; not limiting to one day.
#**introduce folks to emerging technologies + projects, so early adopters can play with it (eg labs)
#**Having physical meetups. Need to expand on the ideas listed, and drive a purpose
#**How to handle feedback from outside, and incorporate them in the most effective way into testing
#'''Waverley'''
#*Summary:
#**Waverley's feedback revolved around:
#***How can they help more?
#***Litmus tests - creating, updating, maintaining?
#***More regular interaction with us?
#***what are document source of truth
#***Mozilla to provide more guidance, attention, and answering when needed
#****Having special office hours, more regular interaction outside of tuesday mornings, and keeping them informed on projects and bugs
#***Encourage Waverley to interact with developers directly in bugzilla, irc, and any other avenues more often. Doing a great job now, but more of it is good.
#*** Include Waverley on our monday 9am triage calls?
#**** Need to provide better feedback on litmus test case writing.  Also, writing good feature testplans
#*Takeaways:
#**'''(Tony)''' flagging in-litmus + bugs and ask them/us to create tests
#** '''(aaron, Kevin)''' provide our knowledge of bugs and features status in our features in test plans
#** '''(aaron, Kevin)''' Follow up with feature sign off / status.  Look through their wiki page, also get in touch directly.  They have a PM now, so we can work through that channel.
#** '''(aaron, Kevin)''' Need to provide better feedback on litmus test case writing.  Also, writing good feature testplans
#***Create a template for a testplan
#** '''(Tony, Tracy)''' Introduce Services projects (sync triaging, client-side testing for now) to Waverley.
#**''' (All)''' regression-wanted; build config
#* Sandboxed for Future:
#** Future projects and expectations for Waverley
# '''Services Goals'''
#* Summary:
#** client-side automation, explore mozmill and TPS
#** server-side automation, explore funkload and TPS
#** Continuous integration is needed
#** Full QA against Load Cluster (combined Staging)
#** Pull/package FF build with it.  Build out a VM that can be used by anyone
#* Takeaways:
#** '''(tracy, owen)''' What sets of tests do we automate?
#** '''(tracy)''' develop mozmill automation for client sync smoketests
#** '''(Owen)''' Funkload tests with load. Add reporting
#** '''(James)''' Get production reports so we can analyze baselines and customized load tests
#** '''(James)''' Get two servers up and running so we can snapshot them as reference system
#** '''(tracy, owen, James)''' Have proper documentation for all the above
#* Sandboxed for Future:
#** Sync Server distribution and support + Reference Setup
#** Creating VMs with reference setup openly
#** Building up all components for distribution (nginx, gunicorn, sync server, etc...)
#*** Make available for community to contribute
# '''Mobile Automation'''
#* Summary:
#** Android is a top tier platform focus - so we should be adding automation to the mix
#*** No current client mobile framework on our plate now.
#*** QA can work with A-team and releng to work on fixing broken mochitests
#*** crowdsource addon&nbsp;: get out there
#** We know Dev team has lack of automation now, and aware of it
#** Discussed pros and cons of
#* Takeaways:
#** '''(martijn)''' continue to work with joel on getting the failed mochitests on tinderbox down (Q3 Goal)
#** '''(martijn, aaron)''' continue to get the crowdsource addon in the hands of testers quickly (Q3 Goal)  (see wiki page)
#**''' (tchung, aaron)''' work with QA Automation team to create a wish-list of what a client tool should look like
#*** In addition, talk to Clint's team to see what they have in mind
#* Sandboxed for Future:
#** A better understanding of how dev automation is today, and their roadmap
#*** What other tools can be used?  (eg. more Test Harness like - what about performance testing?)
#** Investigate usage of 3rd party automation tools in DA, and going down the private cloud route
#** continued client automation solution in house
#** L10n Coverage has nothing.  what about memory usage, telemetry, performance?
#** Who can write more browser-chrome tests?
#*** Ask developers as well for their input: mochitests are being written from them
# '''Sync Discussion'''
#* Summary:
#** Weekly train work, defined with monday client-side handoff, and wednesday server-side handoff.  Important to define and document what is being tested each week, in details and in public.
#** Questions around the right automation tools for client.  Mozmill or TPS?  Need to weigh the two
#** Server side automation will continue with funkload.  QA sign-off needs to happen on Funkload results/reports since we are not controlling the activity
#** Test/Automation of Server-Side
#*** 1. Weekly Train - Stage
#*** 2. Maintenance Releases
#*** 3. Load Testing
#*** 4. Continuous Integration
#** Staging VMs are being repurposed to make a CI ENV for Dev and OPs.  All QA work will be moved over to the physical LT cluster
#* Takeaways:
#**'''(james)''' build out Sync environment for QA, and host it with viewable IP
#** '''(tracy, owen)''' to talk to Henrik this week (hopefully) about usefullness of Mozmill
#** '''(Owen)''' already in touch with Jonathon about what is/is not working with TPS
#** '''(Tracy)''' to look at updating his Sync Server TP and the test cases (new, current) in LItmus.  helpful to community use on Aurora/Beta branchs
#* Sandboxed for Future:
#** We should talk to Jesse Ruderman about FireFox Fuzz Testing and using ideas/techniques for Sync automation
#** What's left of Eggplant?
# Mobile Coffee Talk
#* Summary:
#** What's the best way to know when beta builds are coming? (usually HG to get the changes, usually a handful of small fixes, see URL link for Beta Changes (from Kevin)
#** Haven't spent much time reviewing what Waverley results from beta and release testing.  Need to spend more resources reviewing their work
#** Device Anywhere is pretty good with turn around with their bugs.  But need to figure out ways to setup quicker, cleaner state, deal with power issues
#** Crasher Triage has challenges.  libc bugs dont tell us much, and symbols are missing on Honeycomb
#** Device Compatibility, how to increase exposure and participation?  need better market feedback, QA more stern about issues found, more aggressive in bugzilla on devices, tracking devices better.
#** Feature signoff earlier.  Shouldnt wait until beta, but midpoint of Aurora.  what to do about late string freezes?  Also, bad tracking of features getting backed out, need better tracking criteria.
#** Aurora in the marketplace
#* Takeaways:
#** '''(aaron, kevin)''' Write up device anywhere wiki page (how to use, etc.)
#** '''(naoki)''' follow up with Thomas or Android and see how to get more symbols for honeycomb
#** '''(All)''' trouble shooting documentation for Waverly / Device Anywhere
#** '''(All)''' help waverly do some troubleshooting before submitting ticket
#**''' (kevin, aaron)''' Weekly dig&nbsp;: add feedback analysis for Aurora
#** '''(martijn)''' deploy crowdsource addon!
#* Sandboxed for Future:
#** Device Anywhere automation
#** Is there a better release process we can push forward?
#** Integration with Desktop scheduling
# '''Beta Environment'''
#* Summary:
#** Defined: quick, experimental projects in an isolated dev environment (its own channel, repository, etc)
#** QA's signoff criteria: (Solid extension tested, &amp; Loose support of product sign-off (eg. identity, Account Portal))
#*** QA level of support: monitor feedback, smoketest levels of product
#* Takeaways:
#** '''(tony, james)''' How are we getting QA knowledge of incoming projects?  What is our signoff process)
#* Sandboxed for Future:
#** If Beta gets big, how do we integrate other team projects into this environment?  (eg. web, mobile, desktop)
#*** And how would QA signoff work then?  Services team owns the addon and the infra
#*** Should we also have our own QA beta environment?
Tchung
Confirmed users
6,361

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/330348...335289"