MozillaWiki

QA/Browser Technologies/JulyWorkWeek2011: Difference between revisions

Page Discussion

QA/Browser Technologies/JulyWorkWeek2011 (view source)

Revision as of 01:52, 2 August 2011

9,465 bytes added ,  2 August 2011
m
→‎Action Items
 
(34 intermediate revisions by 5 users not shown)
Line 7: Line 7:
|- valign="top"
|- valign="top"
| '''Date:'''  
| '''Date:'''  
| June 20th-25, 2011
| July 18th-22, 2011
|- valign="top"
|- valign="top"
| '''Attendees:'''  
| '''Attendees:'''  
Line 14: Line 14:
| '''What:'''  
| '''What:'''  
| Browser Technologies QA Workweek
| Browser Technologies QA Workweek
|- valign="top"
| '''Main Scrumpad:'''
| http://mozqa.sync.in/bt-work-week-july-2011
|}
|}


= Meeting Space =
= Meeting Space =


Offsite: Half moon bay
Offsite:  
Onsite: Zombocom
  [http://maps.google.com/maps?q=171+Coronado+Ave,+half+moon+bay+ca&oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&hq=&hnear=0x808f739c6551fea1:0x83f656cbb6bd1730,171+Coronado+Ave,+Half+Moon+Bay,+CA+94019&gl=us&ei=HIgkTvOOM5CqsAP64p1L&sa=X&oi=geocode_result&ct=title&resnum=1&ved=0CBUQ8gEwAA 171 Coronado Ave, HMB]
  Leslie: (650) 703-8993
<br>Onsite: Zombocom
 
= Agenda  =
= Agenda  =
The Workweek will consist of Project planning, Discussions, Lightning Talks, and Work sessions.  Use the offsite for peer discussions, but also use the onsite to catch up with devs, pm's and other colleagues.


{| class="fullwidth-table" style="width: 939px; height: 139px;"
{| class="fullwidth-table" style="width: 939px; height: 139px;"
Line 52: Line 59:
== Monday  ==
== Monday  ==


Breakdown for the week.  Start a idea scrumpad, lay out the expectations
*2pm: Breakdown for the week.  idea scrumpad: http://mozqa.sync.in/bt-work-week-july-2011
http://mozqa.sync.in/bt-work-week-july-2011
*5pm: Head to Offsite.  See scrumpad for details
*8:30pm: Dinner at [http://maps.google.com/maps?oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&q=sam%27s+chowder+house&fb=1&gl=us&hq=sam%27s+chowder+house&hnear=0x808fb7495bec0189:0x7c17d44a466baf9b,Mountain+View,+CA&ei=7LskTr_3FYTEsgaC9IjBCQ&sa=X&oi=local_result&ct=image&ved=0CAQQtgM&cid=0,0,2828005579592320106 Sam's Chowderhouse], reservations under Tony


== Tuesday  ==
== Tuesday  ==


*8am: Mobile Waverley Call [https://wiki.mozilla.org/QA/Fennec/Waverley/2011-07-19 (Notes)]
*8am: Mobile Waverley Call [https://wiki.mozilla.org/QA/Fennec/Waverley/2011-07-19 (Notes)]
*9am : Large Group Introduction & Discussions
*9am : Large Group Introduction & Discussions ( [http://mozqa.sync.in/vision-casting Vision Casting] )
*10am: Mobile Discussion / Services Work Session
*10am: Mobile Discussion / Services Work Session ( [http://mozqa.sync.in/community-discussion Community Discussion] )
*11am: Mobile Discussion / Services Work Session
*11am: Mobile Discussion / Services Work Session ( [http://mozqa.sync.in/mobile-automation Mobile Automation] )
*12pm - 12:45pm: Lunch
*12pm ~ 12:32 Mobile Discussion ([http://mozqa.sync.in/waverly-mobile Waverly Mobile])
*1pm: Mobile Discussion / Services Work Session
*12:32 pm - 1:30pm: Lunch
*2pm: Services Discussion / Mobile Work Session
*1:30pm ~ 2:30 : Team Building Exercises
*3pm: Services Discussion / Mobile Work Session
*2:30pm ~ 4:00 pm : Mobile Work Session ([http://mozqa.sync.in/mobile-coffeetalk mobile coffetalk]) /  Services Discussion
*4pm: Services Discussion / Mobile Work Session
*4pm: Services Discussion / Mobile Work Session ( [http://mozqa.sync.in/bt-lightningtalks Lightning Talk Notes])
*5pm: Half Moon Bay fun, Group Picture
*5pm: Half Moon Bay fun, Group Picture  
*7pm: Dinner at [http://maps.google.com/maps?q=131+Mirada+Rd+,+half+moon+bay,+ca&oe=utf-8&rls=org.mozilla:en-US:unofficial&client=firefox-a&um=1&ie=UTF-8&hq=&hnear=0x808f739c6d2e7fcb:0xc8ef1addb552e514,131+Mirada+Rd,+Half+Moon+Bay,+CA+94019&gl=us&ei=WLskTpyeOMTvsgbSzZmeCQ&sa=X&oi=geocode_result&ct=image&resnum=1&ved=0CBYQ8gEwAA Miramar Beach Restaurant], reservations under Tony


== Wednesday  ==
== Wednesday  ==
Line 77: Line 86:
== Thursday ==
== Thursday ==


*12pm: Lunch for Aakash
*11am: Sync Bug Triage
*2pm: BT Project demo,  (video recorded, 10 Fwd)
*12pm: Farewell Lunch for Aakash.  Pho Garden on Castro
*2pm: BT Project demo,  (video recorded on airmozilla, 10 Fwd)
*3pm: Sync Server Unit, Load, Automated Testing: Dev/Ops/QA Discussion
*3pm: mobile release test planning (aaron, kevin)
*TBD: TPS vs. Funkload for Sync Server API automation/smokes: James, Owen, Jonathon, other interested parties


== Friday  ==
== Friday  ==


*10am - 10:30am: Automating load &amp; performance testing in the CI and Se2
* TBD: TPS vs. Funkload for Sync Server API automation/smokes: James, Owen, Jonathon, other interested parties
*10:35am - 11:05am: Our mobile automation story - spikes and goals
* TBD: other stuff
* Tracy returns to KC (7:10 am flight)
* Aaron returns to TO (noon flight)


= Action Items  =
= Action Items  =
Items that the group felt would merit followup on
== Tuesday's Notes  ==
[[https://docs.google.com/document/d/1iRjaQlm24khgPoAJpSMifCTae0Fd2-524DGWcVODSOA/edit?hl=en_US full meeting minutes]]
=== uTest  ===
*Krupa, Rebecca, and Matt to work on getting a checklist for uTest projects things we need their testers to adhere to, and whether they file directly in Bugzilla, etc.
=== Waverly ===
*Assign one tester to:
**AMO
**Socorro
*Process outline for utilizing Waverly
**daily standups
**Office hours
**Point them to the Guidelines
**WebQA point person for automation needs in Tuesday 8am
**Get Matt the point person and ask for a direct meeting invite for that contact person
*Team to try using GitHub issues, talk to someone who’s using it
**Raymond said to ask Dietrich
== Wednesday's Notes  ==
[[https://docs.google.com/document/d/1xIp1-6ix6pi4EayOC1eZ-j8ShyqNHj2_nwdrUyK8J4E/edit?hl=en_US full meeting minutes]]
=== Load Testing ===
*Stephen to talk to Justin Lazarao about the load-testing cluster
*Stephen to talk to James Bonacci
*Investigate multi-node JMeter?
*Can trigger JMeter from a simple ant script in CI
== Thursday's Notes  ==
[[https://docs.google.com/document/d/1wDq70o6bmDWOcLFZoD1J7BWsw5rmls9mM3InNHCqtW8/edit?hl=en_US full meeting minutes]]
=== Git  ===
*Stephen &amp; Matt B. to create a visual diagram of the git clone/branch/merge/review process.
*Team members can consider sitting through test reviews with one another to learn from one another.
*Use Pivitol Tracker
*Stephen to talk to Dave Dash and see about co-developing a fuzzing tool a la PowerFuzzer
=== Garmr  ===
*pip installable
*https://github.com/AutomatedTester/Garmr
*meant to check security items on our security check list - https://wiki.mozilla.org/WebAppSec/Secure_Coding_QA_Checklist


== Friday's Notes<br>  ==
Collaborating Notes from Workweek (7/25)


[[https://docs.google.com/document/d/1c4Rk4DjoTUjnUUBhRnNfUQqOWZezHcequm6lVXXgX6Q/edit?hl=en_US full meeting minutes]]
#'''Vision &amp; Goals'''
=== Our Automation Env ===
#*Summary:  
*remove the blocker dependency, get Se2 running
#**Responsible for emerging technologies and environments like Sync, Mobile, Experimental Lab projects (Identity, Share, WebApps), and server environments
*finding a project that we can try out BrowserMob with, and do a trial run
#**More resources to help with building out automation, support for new projects, and a better usage of tasks like project investigation, more exploratory testing, interacting with other teams, and defining processes
*Need a real strategy for the environments that we need to support
#***Less focus on routine tasks like regression testing, and beefing automation while leveraging outsource tools
**we should be using Webtrends+anticipated users’ platforms (we want to support other platforms for conversion)  
#*Takeaways:
*Grid 2 is again a dependency/blocker
#**'''(Tony)''' Defining Process: what is the future of sync and mobile in respect to Browser Tech?
*Nightly, Aurora, Beta, RC FF
#**'''(Tony)''' Come up with a BT service agreement for emerging projects, that's somewhat carbon copied to hand off to teams
**need to explore an automated way to upgrade Nightly &amp; Aurora
#**'''(Tony)''' define our goals to other teams (marketing, support, devs, l10n)
**doing this cleanly would be a nice contribution to the community
#*Sandboxed for Future:
**Ensure Se supports Nightly &amp; Aurora envs
#**What are our responsibilities, current vs future? As a team?
*Latest Chrome, Safari (limited/non-existent Selenium support at present), Opera (you’re welcome Esteban)  
#**What's our hiring and resourcing plan?
*Linux, OSX, Win XP, Vista?,Win 7
#'''Community'''
*Consider Android, iPad, iPhone envs
#*Summary:
*explore targeting specific platforms with Se (Firefox x on Windows x, etc.) -- granularity with environments for test runs
#**Community engagement is a challenge for Mobile and Services. Both have very small userbase now.  
*Look at resurrecting an Ubuntu Linux environment for Selenium
#**provide a clear list of tasks and a regular schedule:
*What about Mac OS X Lion support? Do we want to maintain dual environments?  
#***QMO cleanup (owen and rbillings are asking around)
**Which testruns run in Snow Leopard, which run in Lion? Do we want a blend of both?  
#***Testday posts (send this earlier! also, needs to have direct list of tasks for those that want it, but exploratory for others)
**Modify the current environment string to differentiate between the two versions
#****there should be an assistance from dev&nbsp;: ideally it would be nice if they attended events
#**Have a playground environment for experimenting with client/server testing and automation
#***Mobile can be device anywhere, create scripts for easier and faster setup
#***Services would be a QA infrastructure
#*Takeaways:
#**'''(aaron, tracy)''' use Moz Reps; additional means of communications
#***other focus channels (ie Reddit, selenium, Android)
#**provide a test environment that's easy accessible from outside
#***'''(kevin, aaron)''' Device Anywhere for mobile
#***'''(tracy, james)''' Sync server with access for Services)
#**'''(All)''' Clearer Testday posts, earlier, and list of tangible tasks to execute
#***'''(All)''' Include Devs in testday channels, and ask them to be strategic on what to do
#**'''(james, owen, tracy)''' Setup VM environments and documentation for services. Video Sync help
#*Sandboxed for Future:
#**have a forum, survey, summary of bugs, feedback, summary of bugs, list of tasks outside of IRC; not limiting to one day.
#**introduce folks to emerging technologies + projects, so early adopters can play with it (eg labs)
#**Having physical meetups. Need to expand on the ideas listed, and drive a purpose
#**How to handle feedback from outside, and incorporate them in the most effective way into testing
#'''Waverley'''
#*Summary:
#**Waverley's feedback revolved around:
#***How can they help more?
#***Litmus tests - creating, updating, maintaining?
#***More regular interaction with us?
#***what are document source of truth
#***Mozilla to provide more guidance, attention, and answering when needed
#****Having special office hours, more regular interaction outside of tuesday mornings, and keeping them informed on projects and bugs
#***Encourage Waverley to interact with developers directly in bugzilla, irc, and any other avenues more often. Doing a great job now, but more of it is good.
#*** Include Waverley on our monday 9am triage calls?
#**** Need to provide better feedback on litmus test case writing.  Also, writing good feature testplans
#*Takeaways:
#**'''(Tony)''' flagging in-litmus + bugs and ask them/us to create tests
#** '''(aaron, Kevin)''' provide our knowledge of bugs and features status in our features in test plans
#** '''(aaron, Kevin)''' Follow up with feature sign off / status.  Look through their wiki page, also get in touch directly.  They have a PM now, so we can work through that channel.
#** '''(aaron, Kevin)''' Need to provide better feedback on litmus test case writing.  Also, writing good feature testplans
#***Create a template for a testplan
#** '''(Tony, Tracy)''' Introduce Services projects (sync triaging, client-side testing for now) to Waverley.
#**''' (All)''' regression-wanted; build config
#* Sandboxed for Future:
#** Future projects and expectations for Waverley
# '''Services Goals'''
#* Summary:
#** client-side automation, explore mozmill and TPS
#** server-side automation, explore funkload and TPS
#** Continuous integration is needed
#** Full QA against Load Cluster (combined Staging)
#** Pull/package FF build with it.  Build out a VM that can be used by anyone
#* Takeaways:
#** '''(tracy, owen)''' What sets of tests do we automate?
#** '''(tracy)''' develop mozmill automation for client sync smoketests
#** '''(Owen)''' Funkload tests with load. Add reporting
#** '''(James)''' Get production reports so we can analyze baselines and customized load tests
#** '''(James)''' Get two servers up and running so we can snapshot them as reference system
#** '''(tracy, owen, James)''' Have proper documentation for all the above
#* Sandboxed for Future:
#** Sync Server distribution and support + Reference Setup
#** Creating VMs with reference setup openly
#** Building up all components for distribution (nginx, gunicorn, sync server, etc...)
#*** Make available for community to contribute
# '''Mobile Automation'''
#* Summary:
#** Android is a top tier platform focus - so we should be adding automation to the mix
#*** No current client mobile framework on our plate now.
#*** QA can work with A-team and releng to work on fixing broken mochitests
#*** crowdsource addon&nbsp;: get out there
#** We know Dev team has lack of automation now, and aware of it
#** Discussed pros and cons of
#* Takeaways:
#** '''(martijn)''' continue to work with joel on getting the failed mochitests on tinderbox down (Q3 Goal)
#** '''(martijn, aaron)''' continue to get the crowdsource addon in the hands of testers quickly (Q3 Goal)  (see wiki page)
#**''' (tchung, aaron)''' work with QA Automation team to create a wish-list of what a client tool should look like
#*** In addition, talk to Clint's team to see what they have in mind
#* Sandboxed for Future:
#** A better understanding of how dev automation is today, and their roadmap
#*** What other tools can be used?  (eg. more Test Harness like - what about performance testing?)
#** Investigate usage of 3rd party automation tools in DA, and going down the private cloud route
#** continued client automation solution in house
#** L10n Coverage has nothing.  what about memory usage, telemetry, performance?
#** Who can write more browser-chrome tests?
#*** Ask developers as well for their input: mochitests are being written from them
# '''Sync Discussion'''
#* Summary:
#** Weekly train work, defined with monday client-side handoff, and wednesday server-side handoff.  Important to define and document what is being tested each week, in details and in public.
#** Questions around the right automation tools for client.  Mozmill or TPS?  Need to weigh the two
#** Server side automation will continue with funkload.  QA sign-off needs to happen on Funkload results/reports since we are not controlling the activity
#** Test/Automation of Server-Side
#*** 1. Weekly Train - Stage
#*** 2. Maintenance Releases
#*** 3. Load Testing
#*** 4. Continuous Integration
#** Staging VMs are being repurposed to make a CI ENV for Dev and OPs.  All QA work will be moved over to the physical LT cluster
#* Takeaways:
#**'''(james)''' build out Sync environment for QA, and host it with viewable IP
#** '''(tracy, owen)''' to talk to Henrik this week (hopefully) about usefullness of Mozmill
#** '''(Owen)''' already in touch with Jonathon about what is/is not working with TPS
#** '''(Tracy)''' to look at updating his Sync Server TP and the test cases (new, current) in LItmus.  helpful to community use on Aurora/Beta branchs
#* Sandboxed for Future:
#** We should talk to Jesse Ruderman about FireFox Fuzz Testing and using ideas/techniques for Sync automation
#** What's left of Eggplant?
# Mobile Coffee Talk
#* Summary:
#** What's the best way to know when beta builds are coming? (usually HG to get the changes, usually a handful of small fixes, see URL link for Beta Changes (from Kevin)
#** Haven't spent much time reviewing what Waverley results from beta and release testing.  Need to spend more resources reviewing their work
#** Device Anywhere is pretty good with turn around with their bugs.  But need to figure out ways to setup quicker, cleaner state, deal with power issues
#** Crasher Triage has challenges.  libc bugs dont tell us much, and symbols are missing on Honeycomb
#** Device Compatibility, how to increase exposure and participation? need better market feedback, QA more stern about issues found, more aggressive in bugzilla on devices, tracking devices better.
#** Feature signoff earlier.  Shouldnt wait until beta, but midpoint of Aurora.  what to do about late string freezes?  Also, bad tracking of features getting backed out, need better tracking criteria.
#** Aurora in the marketplace
#* Takeaways:
#** '''(aaron, kevin)''' Write up device anywhere wiki page (how to use, etc.)
#** '''(naoki)''' follow up with Thomas or Android and see how to get more symbols for honeycomb
#** '''(All)''' trouble shooting documentation for Waverly / Device Anywhere
#** '''(All)''' help waverly do some troubleshooting before submitting ticket
#**''' (kevin, aaron)''' Weekly dig&nbsp;: add feedback analysis for Aurora
#** '''(martijn)''' deploy crowdsource addon!
#* Sandboxed for Future:
#** Device Anywhere automation
#** Is there a better release process we can push forward?
#** Integration with Desktop scheduling
# '''Beta Environment'''
#* Summary:
#** Defined: quick, experimental projects in an isolated dev environment (its own channel, repository, etc)
#** QA's signoff criteria: (Solid extension tested, &amp; Loose support of product sign-off (eg. identity, Account Portal))
#*** QA level of support: monitor feedback, smoketest levels of product
#* Takeaways:
#** '''(tony, james)''' How are we getting QA knowledge of incoming projects? What is our signoff process)
#* Sandboxed for Future:
#** If Beta gets big, how do we integrate other team projects into this environment? (eg. web, mobile, desktop)
#*** And how would QA signoff work then?  Services team owns the addon and the infra
#*** Should we also have our own QA beta environment?
Tchung
Confirmed users
6,361

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/330343...335289"