|
|
| Line 18: |
Line 18: |
|
| |
|
| = Contributions = | | = Contributions = |
| == 2018 H2 ==
| |
| '''Legend''': ☑ Completed task | ☐ Upcoming task | | '''Legend''': ☑ Completed task | ☐ Upcoming task |
| | | {| class="wikitable mw-collapsible" style="overflow:auto; width:800px" |
| | ! style="text-align:left" | <big>2018 H2</big> |
| | |- |
| | | |
| ; Professional Development | | ; Professional Development |
| : ☐ Read ''Peopleware'' | | : ☐ Read ''Peopleware'' |
| Line 38: |
Line 40: |
| : ☑ Transition Alin Rus from WebRender benchmarking to IRIS test development | | : ☑ Transition Alin Rus from WebRender benchmarking to IRIS test development |
| : ☑ Transition Roxana Robotin from WebRTC test maintenance to IRIS test development | | : ☑ Transition Roxana Robotin from WebRTC test maintenance to IRIS test development |
| | | |} |
| == 2018 H1 ==
| |
| === PI Requests v2 ===
| |
| ; OKR: Drive creation of a proof-of-concept tool to replace the mailing list / spreadsheet ([https://wiki.mozilla.org/User:Ashughes/PI_Requests_v2 wiki])
| |
| | |
| ; Tasks
| |
| : ☑ Develop clear requirements for the ideal PI Request system
| |
| :: ☑ Conduct user interviews
| |
| :: ☑ Document requirements of the ideal system based on user feedback
| |
| : ☑ Fix low-hanging fruit within the current system
| |
| : ☑ Develop a workflow design of the PI Request system
| |
| :: ☑ Draft a workflow design encompassing the minimum viable end-to-end workflow
| |
| :: ☑ Get feedback on workflow design from key stakeholders (Softvision, PI Management, Service Now)
| |
| :: ☑ Revise MVP workflow design based on feedback
| |
| : ⇒ Conduct a feasibility study of Service Now as a drop-in replacement of the PI Request system
| |
| :: ☑ Review implementation requirements of MVP workflow design in Service Now
| |
| :: ☐ Engage Service Now team in development of MVP system for testing
| |
| :: ☐ Test implementation of Service Now based MVP for compliance to workflow
| |
| :: ☐ Test implementation of Service Now based MVP with a PI test group
| |
| : ⇒ Conduct a feasibility study of JIRA Service Desk
| |
| ::
| |
| | |
| === Benchmarking ===
| |
| ; <big>''OKR: Lead development and maintenance of automated tests for Quantum benchmarking''</big>
| |
| {|
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Owners</u>:
| |
| | Alin Rus (reporting to Milan)
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Code</u>:
| |
| | https://github.com/Top5Softvision/Top5_Benchmark
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Dashboard</u>:
| |
| | https://docs.google.com/spreadsheets/d/1LxEcm8Fz1tRU9s9HhIOlHdktO_a5UEEtw6bemfjq0h4/edit?usp=sharing
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Documentation</u>:
| |
| | https://docs.google.com/document/d/1UT8GTcVUQDYqCySNNhSoRBnIOiJa7TjN0GAkDyFLEFA/edit
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Next Steps</u>:
| |
| | ☐ Finish writing tests against Youtube<br />☐ Begin work on automated dashboard
| |
| |}
| |
| | |
| === WebRTC Test Automation ===
| |
| ; <big>''OKR: Lead development and maintenance of automated tests''</big>
| |
| * '''Owners''': Roxana Robotin (reporting to Jan-Yvar)
| |
| * '''Code''': [https://hg.mozilla.org/mozilla-central mozilla-central]
| |
| * '''Tracking''': [https://docs.google.com/spreadsheets/d/1bi89fKZ8rGkHxi_4aZwz1ZnxfUYQa7oODsTR_WmC61s/edit#gid=1282602213 spreadsheet]
| |
| * '''Dashboard''': ?
| |
| * '''Status''':
| |
| ** '''2018-04-03''': 6 Open, 0 Resolved, 0% Complete
| |
| * '''Tasks''':
| |
| ** Update and Run WebRTC mochitests via {{bug|1454644}} - [https://docs.google.com/spreadsheets/d/17rWf-eAI42Bv6_c2IvhHYhck8IOZeFbK0uXcaB6kncg/edit#gid=1527710909 doc]
| |
| | |
| | |
| What are the metrics for this project?
| |
| * number of defects found by automation?
| |
| * time spent on implementation of a specific test?
| |
| * time spent on maintenance of a specific test?
| |
| * time saved vs manual testing?
| |
| * key deliverable such as a dashboard or audit?
| |
| | |
| <bugzilla>{"blocks":"1405876"}</bugzilla>
| |
| | |
| === Firefox UI Tests ===
| |
| ; <big>''OKR: Determine path forward for so-called deprecated Firefox UI tests''</big>
| |
| {|
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Owners</u>:
| |
| | ?
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Code</u>:
| |
| | https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests/
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Dashboard</u>:
| |
| | ''<link to results dashboard>''
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Documentation</u>:
| |
| | Matt Howell [https://docs.google.com/document/d/1KFtZgd2zbolo7G_7lq8Nle8gnOoUr21l4r-vdwiS-G0/edit?ts=5aa2df6b explains the reason] the Update tests are disabled. Functional and Puppeteer tests need to be audited.<br />
| |
| <u>Local Testrun - Linux x86_64 - 2018-03-21</u>:
| |
| <pre>
| |
| SUMMARY
| |
| -------
| |
| 7:22.40 INFO passed: 86
| |
| 7:22.40 INFO failed: 1
| |
| 7:22.40 INFO todo: 3 (skipped: 1)
| |
| 7:22.40 INFO
| |
| FAILED TESTS
| |
| -------
| |
| 7:22.40 INFO test_initial_download.py test_initial_download.TestSafeBrowsingInitialDownload.test_safe_browsing_initial_download
| |
| 7:22.40 SUITE_END
| |
| </pre>
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Next Steps</u>:
| |
| | ☐ Set up a local environment to run the tests locally and see what happens<br />☐ Analyze the result of the tests
| |
| |}
| |
| | |
| === Metrics-Graphics-GFX ===
| |
| ; <big>''OKR: Determine path forward for metrics-graphics-gfx dashboard''</big>
| |
| {|
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Owners</u>:
| |
| |
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Code</u>:
| |
| |
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Dashboard</u>:
| |
| |
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Documentation</u>:
| |
| |
| |
| |-
| |
| | style="text-align:right; vertical-align:top" | <u>Next Steps</u>:
| |
| | ☐ Anthony to review project status with Milan<br />
| |
| |}
| |
| | |
| == 2017 ==
| |
| === 2017 Q1 ===
| |
| * ☐ something to do
| |
| * ☑ something done
| |
| * ✓ something done
| |
| * ✔ something done
| |
| | |
| ☐ something to do <br/>
| |
| ☑ something done <br/>
| |
| ✓ something done <br/>
| |
| ✔ something done <br/>
| |
| | |
| == 2016 ==
| |
| === 2016 Q4 ===
| |
| {|
| |
| |-
| |
| | '''1. Prototype a dashboard of long-term Graphics stability trends'''
| |
| * {{done|}} architect a system to automate updating chart data periodically beyond the 6-month Socorro API limit
| |
| ** {{done|}} create a python script to clone crash data from Socorro
| |
| ** {{done|}} set up python script on a 24-hour cron job
| |
| * {{done|}} prototype a series of long-term charts to track:
| |
| ** {{done|}} WebGL success rate - rpi3/#psd-webgl
| |
| ** {{done|}} WebGL crash rate - rpi3/#psd-webgl-rate
| |
| ** {{done|}} Shutdown crashes - rpi3/#psd-shutdown
| |
| ** {{done|}} Startup crashes - rpi3/#psd-startup
| |
| ** {{done|}} Driver crashes - rpi3/#psd-driver-rate
| |
| ** {{prev|}} High-volume crashes
| |
| ** {{done|}} Overall Graphics crash rate - rpi3/#psd-gfx-rate
| |
| * {{done|}} prototype a dashboard to display all long-term graphs - See <server>/index.htm#psd-dashboard
| |
| * {{done|}} [add'l] Move infrastructure to a Mac Mini behind VPN
| |
| * {{prev|}} host the dashboard on metrics.mozilla.com/gfx (if possible)
| |
| | |
| |-
| |
| | '''2. Prototype a small-scale, automated emunittest lab to improve discoverability of WebGL regressions in Nightly'''
| |
| * {{done|}} set up a machine to be used for testing on a daily basis
| |
| * {{done|}} get a script from Jukka Jylänki's to automate publishing test results to his local server in Finland
| |
| * {{done|}} test the script on a dummy rig at my desk in Vancouver
| |
| * {{done|}} let the machine run for several days to capture long-term results - See [http://clbri.com:6932/results?uuid=9f09167b-a166-4402-a20d-7536574b5d06 clbri.com]
| |
| * {{prev|}} optionally, get Jukka to create a mobile version of his test site that we can point to WebQA's Saucelabs instance
| |
| | |
| |-
| |
| | '''3. Utilize the Graphics test plan framework to validate shipping SKIA Content on Windows to Aurora in Firefox 52'''
| |
| * document the test plan framework and telemetry experiment lessons learned
| |
| * work with Mason to document milestones and release criteria
| |
| * ensure these criteria are measured and met along the way
| |
| |-
| |
| | '''4. Maintenance tasks to ensure GFX quality does not slip
| |
| * Daily bug triage via http://ashughes1.github.io/graphicstriage/
| |
| * Find a new home for One & Done tasks (testrail?)
| |
| * Execute testing via Betabreakers for new features in Aurora
| |
| |}
| |
| | |
| ==== GPU Process ====
| |
| * Develop a test plan to qualify MVP
| |
| * Deploy a Telemetry Experiment in Nightly to A/B test stability (blocked by {{bug|1323304}})
| |
| | |
| === 2016 Q3 ===
| |
| ; Tasks
| |
| * ...
| |
| | |
| ; Bugs
| |
| <bugzilla display="list">
| |
| {
| |
| "emailreporter1":"1",
| |
| "emaillongdesc1":"1",
| |
| "emailtype1":"exact",
| |
| "email1":"anthony.s.hughes@gmail.com",
| |
| "chfield":"[Bug creation]",
| |
| "chfieldfrom":"2016-07-01",
| |
| "chfieldto":"2016-09-30",
| |
| "include_fields":"id,summary,status"
| |
| }
| |
| </bugzilla>
| |
| | |
| === 2016 Q2 ===
| |
| ; Tasks
| |
| * Prototyped a crash-stats dashboard with auto-updating charts
| |
| * Conducted a talk at London all-hands ([https://docs.google.com/presentation/d/17CFTU_WWu0tWd_kHfSghaMbtsgPrcvavEytcat4O5MI/edit#slide=id.g14538819fd_2_263 presentation])
| |
| * Conducted Animometer benchmarking ([https://drive.google.com/open?id=1xo6Yfds1YFn8FAtFdQeb5R_JKMWej-g9cVZWgfK8Rso Mac OS], [https://drive.google.com/open?id=1vCkgNv_EGncd-naj_89s-2Ep9iL7I2tuzTYYhAOV0WA Windows 7])
| |
| * Created an add-on to integrate crash-stats visualization with Bugzilla ([https://addons.mozilla.org/en-US/firefox/addon/bugzilla-socorro-lens/ v0.1])
| |
| * Executed Firefox 46 testing to support {{bug|1255281}} ([[QA/Platform/Graphics/Betabreakers/Firefox_46|testplan]])
| |
| * Executed Firefox 48 testing to support {{bug|1260507}} ([[QA/Platform/Graphics/Betabreakers/Firefox_48|testplan]])
| |
| * Set up weekly triage of Top Crashes, Cold Crashes, Cold Trackers, Untriaged, and Help Wanted bugs with a dashboard to track progress
| |
| | |
| ; Bugs
| |
| <bugzilla display="list">
| |
| {
| |
| "emailreporter1":"1",
| |
| "emaillongdesc1":"1",
| |
| "emailtype1":"exact",
| |
| "email1":"anthony.s.hughes@gmail.com",
| |
| "chfield":"[Bug creation]",
| |
| "chfieldfrom":"2016-04-01",
| |
| "chfieldto":"2016-06-30",
| |
| "include_fields":"id,summary,status"
| |
| }
| |
| </bugzilla>
| |
| | |
| === 2016 Q1 ===
| |
| * Filed a bug to enable special character searches on Socorro ({{bug|1239402}})
| |
| * Prototyped a dashboard of Socorro data
| |
| * Prototyped a dashboard of Bugzilla data
| |
| * Executed a testrun to vet Developer Edition 45 for WinXP + D3D9 + E10S re bug 1237769
| |
| | |
| ; Bugs
| |
| <bugzilla display="list">
| |
| {
| |
| "emailreporter1":"1",
| |
| "emaillongdesc1":"1",
| |
| "emailtype1":"exact",
| |
| "email1":"anthony.s.hughes@gmail.com",
| |
| "chfield":"[Bug creation]",
| |
| "chfieldfrom":"2016-01-01",
| |
| "chfieldto":"2016-03-31",
| |
| "include_fields":"id,summary,status"
| |
| }
| |
| </bugzilla>
| |
| | |
| == 2015 ==
| |
| === 2015 Q4 ===
| |
| * Drafted documentation on graphics device/driver blocklisting ''[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Graphics_Blocklisting]''
| |
| * Executed testruns to sanity check [[QA/Platform/Graphics/Firefox_43|Firefox 43]] and [[QA/Platform/Graphics/Firefox_44|Firefox 44]] in Aurora
| |
| * Organized Android sanity [[QA/Platform/Graphics/Android|testing]] for SKIA update (due Dec 7, 2015)
| |
| * Proposed inclusion of GPU chipset information in Socorro, tracked via {{bug|1192351}}
| |
| * Prototyped a graphics quality [https://metrics.mozilla.com/gfx/ dashboard]
| |
| * Prototyped a periodic triage workflow to close old bugs and escalate lost issues
| |
| * Prototyped a [http://ashughes1.github.io/testday-metrics/charts/index.htm dashboard] to measure engagement during testdays
| |
| | |
| ; Bugs
| |
| <bugzilla display="list">
| |
| {
| |
| "emailreporter1":"1",
| |
| "emaillongdesc1":"1",
| |
| "emailtype1":"exact",
| |
| "email1":"anthony.s.hughes@gmail.com",
| |
| "chfield":"[Bug creation]",
| |
| "chfieldfrom":"2015-10-01",
| |
| "chfieldto":"2015-12-31",
| |
| "include_fields":"id,summary,status"
| |
| }
| |
| </bugzilla>
| |
| | |
| === 2015 Q3 ===
| |
| * Developed a process to create a mission statement of and for the community ([[QA/Future|result]])
| |
| * Conducted one-to-one meetings with peers and collect feedback to inform the mission statement
| |
| * Drafted a recommendation for Hossain Al Ikram to participate in 2015 Global Gatherings
| |
| * Provided advice to Jamie Charlton about planning Firefox OS testing events
| |
| * Dogfooded Windows 10 to report blocker bugs ahead of release
| |
| * Created a process for [http://bit.ly/1JZqzMc daily sanity checks] via the lab in Toronto
| |
| * Executed outsourced sanity checking for [[QA/Platform/Graphics/Firefox_41|Firefox Aurora 41]]
| |
| * Executed outsourced sanity checking for [[QA/Platform/Graphics/Firefox_42|Firefox Aurora 42]]
| |
| * Extended the sanity checking to volunteers via One & Done ([https://docs.google.com/a/mozilla.com/forms/d/1AoaAU6__rebVxOwjb8ZOM_fgZkdqk-DK1hITr5QDzFo/edit# results form])
| |
| * Documented best practices on MDN ([https://developer.mozilla.org/en-US/docs/Mozilla/QA/Helping_the_Graphics_Team artifact])
| |
| * Developed a catalog of graphics hardware to facilitate device-specific testing ([[QA/Platform/Graphics/Inventory|artifact]])
| |
| * Created an [https://docs.google.com/spreadsheets/d/12jNc8rlZP4MDk5xOHyOWlcO2W5J39KKofzvcVZxVFJk/edit#gid=0 internal hardware catalog] as a proof-of-concept
| |
| * Migrated the internal catalog to a
| |
| * Improved readability of the inventory based on peer review
| |
| | |
| ; Bugs
| |
| <bugzilla display="list">
| |
| {
| |
| "id":"1112371,1130195,1145143,1173579,1176147,1180561,1187464,1187466,1189266,1189940,1191060,1194933,1198765,1201679,1201960,1202696,1202700,1202748,1202786,1204337,1206626,1207326,1208579,1209617"
| |
| } | |
| </bugzilla>
| |
