Confirmed users
3,816
edits
Auto-tools/Goals/2012Q2: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| Line 2: | Line 2: | ||
* {{done|Extend Mobile Platform Automation for B2G and Fennec Native to extend our automation systems to work with specific phone hardware and new development boards for both products.}} | * {{done|Extend Mobile Platform Automation for B2G and Fennec Native to extend our automation systems to work with specific phone hardware and new development boards for both products.}} | ||
* {{done|Deploy Datazilla new graph server UI into production to make it easier and simpler to track all our performance numbers across our growing sets of performance and endurance data we are collecting on our products.}} | * {{done|Deploy Datazilla new graph server UI into production to make it easier and simpler to track all our performance numbers across our growing sets of performance and endurance data we are collecting on our products.}} | ||
** ''NOTE:'' Revised deliverable in light of developer feedback and changed course mid-quarter to deliver a system that would turn TBPL orange/green based on per-push regression detection. This means we won't deliver a UI by end of quarter, but the necessary block and tackle for the UI will land (and we should have a system on production hardware in early Q3 - it's just waiting on hardware delivery). | ** '''NOTE:''' Revised deliverable in light of developer feedback and changed course mid-quarter to deliver a system that would turn TBPL orange/green based on per-push regression detection. This means we won't deliver a UI by end of quarter, but the necessary block and tackle for the UI will land (and we should have a system on production hardware in early Q3 - it's just waiting on hardware delivery). | ||
* {{done|Signal From Noise Phase II - Make the same noise-reduction changes we made on Tp5 on all the other page-load tests and ensure all performance tests are sending raw observations to Datazilla.}} | * {{done|Signal From Noise Phase II - Make the same noise-reduction changes we made on Tp5 on all the other page-load tests and ensure all performance tests are sending raw observations to Datazilla.}} | ||
* {{done|Refactor how rapid-release tracking flags are implemented for improved performance and maintainability}} | * {{done|Refactor how rapid-release tracking flags are implemented for improved performance and maintainability}} | ||
** ''NOTE:'' Code complete, awaiting deployment with IT | ** '''NOTE:''' Code complete, awaiting deployment with IT | ||
* {{done|Upgrade bugzilla.mozilla.org to Bugzilla version 4.2}} | * {{done|Upgrade bugzilla.mozilla.org to Bugzilla version 4.2}} | ||
** ''NOTE:'' We are code complete, should be deploying with IT in early weeks of Q3 | ** '''NOTE:''' We are code complete, should be deploying with IT in early weeks of Q3 | ||
* {{done|Reduce android test automation instability and make it easier for the web QA and desktop QA teams to write and run automated tests.}} | * {{done|Reduce android test automation instability and make it easier for the web QA and desktop QA teams to write and run automated tests.}} | ||
** ''NOTE:'' If you look at the metrics, it looks like we attained this goal and then lost it. What actually happened is that we attained this, driving android automation failures down to less than 10%, but then we enabled 250,000 more tests and our instability percentage went back up. The big shift here is that for the first time in Fennec's history, we went from running "only what will pass" on mobile to running "everything except the known failures" and we have developers working on the failures. | ** '''NOTE:''' If you look at the metrics, it looks like we attained this goal and then lost it. What actually happened is that we attained this, driving android automation failures down to less than 10%, but then we enabled 250,000 more tests and our instability percentage went back up. The big shift here is that for the first time in Fennec's history, we went from running "only what will pass" on mobile to running "everything except the known failures" and we have developers working on the failures. | ||
== Official Q2 Goals == | == Official Q2 Goals == | ||
| Line 70: | Line 70: | ||
** {{miss|Create tools to monitor noise in talos numbers so that we know when talos numbers become unacceptably noisy again}} | ** {{miss|Create tools to monitor noise in talos numbers so that we know when talos numbers become unacceptably noisy again}} | ||
** {{done|Ensure that all raw data values from all talos tests flow into Datazilla database backend}} | ** {{done|Ensure that all raw data values from all talos tests flow into Datazilla database backend}} | ||
*** ''NOTE:'' We worked with metrics on the "noisyness" tool, but it will only work with full raw data for datazilla. And so we dropped this support in favor of getting the per-push calculations correct which will take this idea of increasing noisyness into account. | *** '''NOTE:''' We worked with metrics on the "noisyness" tool, but it will only work with full raw data for datazilla. And so we dropped this support in favor of getting the per-push calculations correct which will take this idea of increasing noisyness into account. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 122: | Line 122: | ||
** {{done|Create JSON upload to templeton/datazilla}} | ** {{done|Create JSON upload to templeton/datazilla}} | ||
** {{done|Create dashboard for analysis of results (either as standalone templeton or as Datazilla plugin view)}} | ** {{done|Create dashboard for analysis of results (either as standalone templeton or as Datazilla plugin view)}} | ||
*** ''NOTE:'' Not sure if the pulse listener was ever needed. Never heard any mention of it. | *** '''NOTE:''' Not sure if the pulse listener was ever needed. Never heard any mention of it. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 136: | Line 136: | ||
** {{done|Move to new Pulse hardware in PHX}} | ** {{done|Move to new Pulse hardware in PHX}} | ||
** {{done|Improve durable Queue system}} | ** {{done|Improve durable Queue system}} | ||
*** ''NOTE:'' Added monitoring to handle durable queues but the true fix is in the new library which is not done | *** '''NOTE:''' Added monitoring to handle durable queues but the true fix is in the new library which is not done | ||
** {{skip|Create new library for Mozilla pulse not dependent on carrot}} | ** {{skip|Create new library for Mozilla pulse not dependent on carrot}} | ||
*** '' | *** '''NOTE:''' Deferred in favor of B2G work | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
** All Pulse based Automation systems | ** All Pulse based Automation systems | ||
| Line 152: | Line 152: | ||
** {{miss|Resolve MAC address issue}} | ** {{miss|Resolve MAC address issue}} | ||
** {{miss|Resolve reboot issues}} | ** {{miss|Resolve reboot issues}} | ||
*** ''NOTE:'' We could not resolve the panda mac issues nor the reboot issues this quarter. We did get mochitests running on pandas but we got derailed by constantly looking for kernel solutions to our other problems with the pandas (hands off flashing, stable mac addresses etc). Going forward, we are going to re-tailor the goals and expectations of what is possible on panda dev boards so we can deploy these in Q3. | *** '''NOTE:''' We could not resolve the panda mac issues nor the reboot issues this quarter. We did get mochitests running on pandas but we got derailed by constantly looking for kernel solutions to our other problems with the pandas (hands off flashing, stable mac addresses etc). Going forward, we are going to re-tailor the goals and expectations of what is possible on panda dev boards so we can deploy these in Q3. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 193: | Line 193: | ||
* '''Q2 Outcomes''' | * '''Q2 Outcomes''' | ||
** {{miss| Create a device checkout system for foopy-less management of systems}} | ** {{miss| Create a device checkout system for foopy-less management of systems}} | ||
*** ''NOTE:'' It was a nice to have, deferred in favor of work on android reliability | *** '''NOTE:''' It was a nice to have, deferred in favor of work on android reliability | ||
** {{done|Aid with cultivating best of breed SUT managment tools (in coordination with releng)}} | ** {{done|Aid with cultivating best of breed SUT managment tools (in coordination with releng)}} | ||
** {{done|Fix dependencies so that Mozbase tools (like talos) can be easily integrated with Mozharness}} | ** {{done|Fix dependencies so that Mozbase tools (like talos) can be easily integrated with Mozharness}} | ||
** {{done|Deploy simple pypi server inside our infrastructure so that slaves can easily perform dependency management at runtime (versus at slave-image time via puppet)}} | ** {{done|Deploy simple pypi server inside our infrastructure so that slaves can easily perform dependency management at runtime (versus at slave-image time via puppet)}} | ||
*** ''NOTE:'' The remaining work to deploy the pypi server is all IT side now. | *** '''NOTE:''' The remaining work to deploy the pypi server is all IT side now. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 209: | Line 209: | ||
** {{done|Public web pages, updated daily, indicating total number of closed blockers and nonblockers for each person involved in Fennec, split by team, with links to Bugzilla.}} | ** {{done|Public web pages, updated daily, indicating total number of closed blockers and nonblockers for each person involved in Fennec, split by team, with links to Bugzilla.}} | ||
** {{skip|Include number of comments-on-blocker-bugs per person. | ** {{skip|Include number of comments-on-blocker-bugs per person. | ||
*** ''NOTE:''This comments-per-person feature was not easily obtained via Bugzilla API, so this is a stretch goal.}} | *** '''NOTE:'''This comments-per-person feature was not easily obtained via Bugzilla API, so this is a stretch goal.}} | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 224: | Line 224: | ||
** {{done|Evangelize the use of the system}} | ** {{done|Evangelize the use of the system}} | ||
** {{miss|Attend crash-kill work week to find out how to best support that team with BugHunter}} | ** {{miss|Attend crash-kill work week to find out how to best support that team with BugHunter}} | ||
*** ''NOTE:'' Communication Failure: Did not get invited soon enough to book travel | *** '''NOTE:''' Communication Failure: Did not get invited soon enough to book travel | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 252: | Line 252: | ||
** {{skip|Tailor automation to be buildbot-ready by end of quarter}} | ** {{skip|Tailor automation to be buildbot-ready by end of quarter}} | ||
** {{skip|Deploy into buildbot automation (stretch)}} | ** {{skip|Deploy into buildbot automation (stretch)}} | ||
*** ''NOTE:'' Dburns has been helping out with this, but we deferred several tasks because their automation won't be ready for deploy by end of Q. | *** '''NOTE:''' Dburns has been helping out with this, but we deferred several tasks because their automation won't be ready for deploy by end of Q. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 288: | Line 288: | ||
** {{done|Implement Orange Seed feature}} | ** {{done|Implement Orange Seed feature}} | ||
** {{miss|Deploy to production using staging/production VMs}} | ** {{miss|Deploy to production using staging/production VMs}} | ||
*** ''NOTE:'' Our staging system was a miss due to pending transfer of ES control to IT so that a development ES database to be available that could be used so that OF can use it. Need IT to make a decision about whether to upgrade capacity on IT's dev ES cluster or create a dev ES cluster on the metrics ES instance. | *** '''NOTE:''' Our staging system was a miss due to pending transfer of ES control to IT so that a development ES database to be available that could be used so that OF can use it. Need IT to make a decision about whether to upgrade capacity on IT's dev ES cluster or create a dev ES cluster on the metrics ES instance. | ||
**** ES also needs to be upgraded (IT needed) | **** ES also needs to be upgraded (IT needed) | ||
**** Also needs to be re-indexed (IT needed) | **** Also needs to be re-indexed (IT needed) | ||
| Line 302: | Line 302: | ||
** {{done|(new) Add V8 to the test matrix}} | ** {{done|(new) Add V8 to the test matrix}} | ||
** {{skip|Add mobile support to the tests so that we run the canvas demo on phones}} | ** {{skip|Add mobile support to the tests so that we run the canvas demo on phones}} | ||
*** ''NOTE:'' Skipped in favor of getting autophone more reliable to be used as the basic platform here. | *** '''NOTE:''' Skipped in favor of getting autophone more reliable to be used as the basic platform here. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 313: | Line 313: | ||
* '''Q2 Outcomes''' | * '''Q2 Outcomes''' | ||
** {{drop|Integrate Peptest with telemetry probes to better measure responsiveness}} | ** {{drop|Integrate Peptest with telemetry probes to better measure responsiveness}} | ||
*** ''NOTE:'' Dropped in favor of other higher priority work for mobile | *** '''NOTE:''' Dropped in favor of other higher priority work for mobile | ||
** {{skip|Aid developers with writing peptest patches}} | ** {{skip|Aid developers with writing peptest patches}} | ||
*** ''NOTE:'' Skipped in favor of meeting with developers, figuring out their new needs and resolving to work toward that with a small project in Q3. | *** '''NOTE:''' Skipped in favor of meeting with developers, figuring out their new needs and resolving to work toward that with a small project in Q3. | ||
** {{done|Complete peptest-talos style reporting system}} | ** {{done|Complete peptest-talos style reporting system}} | ||
** {{done|Integrate Peptest reporting into Datazilla}} | ** {{done|Integrate Peptest reporting into Datazilla}} | ||
*** ''NOTE:'' Verified that peptest will write to datazilla, but did not turn on default peptest writing to datazilla yet. | *** '''NOTE:''' Verified that peptest will write to datazilla, but did not turn on default peptest writing to datazilla yet. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 345: | Line 345: | ||
** {{miss|Complete code for mirroring solution}} | ** {{miss|Complete code for mirroring solution}} | ||
** {{miss|Deploy to VM for automation}} | ** {{miss|Deploy to VM for automation}} | ||
*** ''NOTE:'' Missed due to poor leadership from mentor on this project (ctalbert). | *** '''NOTE:''' Missed due to poor leadership from mentor on this project (ctalbert). | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 364: | Line 364: | ||
* '''Q2 Outcomes''' | * '''Q2 Outcomes''' | ||
** {{skip|Drive consensus around a comprehensive architecture for addon test automation}} | ** {{skip|Drive consensus around a comprehensive architecture for addon test automation}} | ||
*** ''NOTE:'' Attempted to start this conversation, but the addons and related teams were not interested in the automation. Left the conversation to be revisited in the future and focused on higher priority mobile, b2g, and bugzilla work instead. | *** '''NOTE:''' Attempted to start this conversation, but the addons and related teams were not interested in the automation. Left the conversation to be revisited in the future and focused on higher priority mobile, b2g, and bugzilla work instead. | ||
* '''Stakeholders''' | * '''Stakeholders''' | ||
| Line 385: | Line 385: | ||
* {{done|Establish best practices to become the best community integrated development team at Mozilla}} | * {{done|Establish best practices to become the best community integrated development team at Mozilla}} | ||
* {{miss|Blog about those practices (as we prove them)}} | * {{miss|Blog about those practices (as we prove them)}} | ||
** ''NOTE:'' We didn't blog about our practices any better than we did last quarter | ** '''NOTE:''' We didn't blog about our practices any better than we did last quarter | ||
* {{done|Promote two community folks to Mentor status.}} | * {{done|Promote two community folks to Mentor status.}} | ||
* {{miss|Set individual blogging targets and meet them.}} | * {{miss|Set individual blogging targets and meet them.}} | ||
** ''NOTE:'' See above, the super stars on our team of blogging did their blogging, but the rest of us didn't. | ** '''NOTE:''' See above, the super stars on our team of blogging did their blogging, but the rest of us didn't. | ||