QA/BrowserID/OPsBuildOut: Difference between revisions

From MozillaWiki
< QA‎ | BrowserID
Jump to navigation Jump to search
 
(40 intermediate revisions by the same user not shown)
Line 10: Line 10:
* QA Test Schedule per four environments: November/December
* QA Test Schedule per four environments: November/December


* Missing latest information from #identity: 11/14 and newer...
== QA  ==


== QA ==
{| width="100%" border="1" style="border-collapse:collapse"
<table width="100%" border="1" style="border-collapse:collapse">
|-
<tr><td width="25%">Task/Project</td><td width="15%">Owner</td><td width="15%">Status</td><td width="45%">Notes, Links, Tickets, Issues</td></tr>
| width="25%" | Task/Project  
| width="15%" | Owner  
| width="15%" | Status  
| width="45%" | Notes, Links, Tickets, Issues


<tr><td valign="top">Track OPs BrowserID build-out planning and HW orders</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
|-
* see Bug 697477</td></tr>
| valign="top" | Track Test/CI build-out  
<tr><td valign="top">BrowserID arch review</td><td valign="top">jbonacci</td><td valign="top">completed</td><td valign="top">
| valign="top" | jbonacci  
* https://intranet.mozilla.org/Services/Ops/BrowserID#Architecture</td></tr>
| valign="top" | Planned
<tr><td valign="top">Track RPM and deployment work in Dev</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
| valign="top" |
* see Bug 695596</td></tr>
*see Bug 695958
<tr><td valign="top">Track Dev build-out</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
*http://hudson.build.mtv1.svc.mozilla.com/job/browserid-server
* see Bug 695957</td></tr>
<tr><td valign="top">Track Stage build-out</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
* see Bugs 695941, 695944</td></tr>
<tr><td valign="top">Track Prod build-out</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* see Bugs 695941, 695943, 695946, 695955, 695956, 695960, 695972, 698044</td></tr>
<tr><td valign="top">Track Test/CI build-out</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* see Bug 695958</td></tr>
<tr><td valign="top">Track DEV load test development</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
* see Issue 504</td></tr>
<tr><td valign="top">Track OPs load test deployment</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
* see Bug 695952</td></tr>
<tr><td valign="top">Track OPs dashboard/monitoring work</td><td valign="top">jbonacci</td><td valign="top">in progress</td><td valign="top">
* see Bugs 695961, 695963, 695967</td></tr>
<tr><td valign="top">Load test tutorial</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Logging for QA work</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Monitoring/dashboards tutorial</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Stability testing Dev env</td><td valign="top">QA</td><td valign="top">planned</td><td valign="top">
* need help from Dev and OPs</td></tr>
<tr><td valign="top">Stability testing Stage env</td><td valign="top">QA</td><td valign="top">planned</td><td valign="top">
* need help from Dev and OPs</td></tr>
<tr><td valign="top">Load testing Stage env</td><td valign="top">QA/OPs</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Stability testing Prod env</td><td valign="top">QA</td><td valign="top">planned</td><td valign="top">
* need help from Dev and OPs</td></tr>
<tr><td valign="top">Stability testing Test/CI env</td><td valign="top">QA</td><td valign="top">planned</td><td valign="top">
* need from with Dev and OPs</td></tr>
<tr><td valign="top">QA node/npm "easy" installs</td><td valign="top">jbonacci</td><td valign="top">ongoing</td><td valign="top">
* https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install</td></tr>
<tr><td valign="top">Complete ID2 configuration</td><td valign="top">jbonacci/lloyd/petef</td><td valign="top">in progress</td><td valign="top">
* working with Lloyd and Pete on this</td></tr>
<tr><td valign="top">QA RPM installs to ID2</td><td valign="top">jbonacci</td><td valign="top">ongoing</td><td valign="top">
* need RPM info from Dev and OPs</td></tr>
<tr><td valign="top">QA Test Plan updates</td><td valign="top">jbonacci</td><td valign="top">ongoing</td><td valign="top">
* https://wiki.mozilla.org/QA/BrowserID/TestPlan</td></tr>
<tr><td valign="top">QA Test wiki updates</td><td valign="top">jbonacci</td><td valign="top">ongoing</td><td valign="top">
* https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta</td></tr>
<tr><td valign="top">QA internal wiki</td><td valign="top">jbonacci</td><td valign="top">planned</td><td valign="top">
* TBD</td></tr>
<tr><td valign="top">Develop Manual Test App</td><td valign="top">jbonacci/ozten</td><td valign="top">planned</td><td valign="top">
* working with Austin on this</td></tr>
<tr><td valign="top">Port Unit Tests to Stage</td><td valign="top">jbonacci/stomlinson</td><td valign="top">planned</td><td valign="top">
* working with Shane on this</td></tr>
<tr><td valign="top">API automation</td><td valign="top">QA</td><td valign="top">planned</td><td valign="top">
* TBD</td></tr>


</table>
|-
| valign="top" | Track new "cross-team env" build-out
| valign="top" | jbonacci
| valign="top" | Planned
| valign="top" |
*This will be a VM environment like old Beta
*This VM env is to be shared by QA teams for BrowserID, AMO, Soup, etc.




* How to access the new environments from the "front end"?
|-
* REF: https://intranet.mozilla.org/Services/Ops/BrowserID/ZeusConfig
| valign="top" | Stability testing Test/CI env
    dev: dev-browserid.services.mozilla.com
| valign="top" | QA
    stage: stage-browserid.services.mozilla.com
| valign="top" | Planned
    prod: browserid.org
| valign="top" |
* I assume then the same clients will now point to the above...
*need from with Dev and OPs


== Dev ==
|-
<table width="100%" border="1" style="border-collapse:collapse">
| valign="top" | Stability testing new "cross-team env"  
<tr><td width="25%">Task/Project</td><td width="15%">Owner</td><td width="15%">Status</td><td width="45%">Notes, Links, Tickets, Issues</td></tr>
| valign="top" | QA
| valign="top" | Planned
| valign="top" |
*need from with Dev and OPs
*This will be a VM environment like old Beta


<tr><td valign="top">Initial plan for Scaling to 1 million users</td><td valign="top">Dev team</td><td valign="top">completed</td><td valign="top">
|-
* https://github.com/mozilla/browserid/wiki/Scaling-to-1M</td></tr>
| valign="top" | Track OPs BrowserID build-out planning and HW orders
<tr><td valign="top">Metrics dashboard</td><td valign="top">Dev team</td><td valign="top">completed</td><td valign="top">
| valign="top" | jbonacci
* see Bug 690107</td></tr>
| valign="top" | completed  
<tr><td valign="top">Maintain "all-in-one" BrowserID</td><td valign="top">lloyd</td><td valign="top">in progress</td><td valign="top">
| valign="top" |
* none</td></tr>
*see Bug 697477 - fixed
<tr><td valign="top">GitHub repository reorganization</td><td valign="top">lloyd</td><td valign="top">completed</td><td valign="top">
* Issue 503: improve repository organization</td></tr>
<tr><td valign="top">Better logging</td><td valign="top">lloyd</td><td valign="top">in progress</td><td valign="top">
* GitHub Issues 119, 537, 541, 536, 530, 529, etc.</td></tr>
<tr><td valign="top">Process splitting: browserid, verifier, keysigner, dbwriter</td><td valign="top">lloyd</td><td valign="top">completed</td><td valign="top">
* GitHub Issue 460</td></tr>
<tr><td valign="top">Second client (dev.)(beta.)myfavoritebooze.org</td><td valign="top">lloyd</td><td valign="top">completed</td><td valign="top">
* support 'keep me signed in' feature</td></tr>
<tr><td valign="top">Developer Engagement Plan</td><td valign="top">Dev/PM teams</td><td valign="top">in progress</td><td valign="top">
* https://wiki.mozilla.org/Engagement/Developer_Engagement/BrowserIDPlan</td></tr>
<tr><td valign="top">Crypto discussions and work</td><td valign="top">Dev/Security teams</td><td valign="top">in progress</td><td valign="top">
* GitHub issus: various</td></tr>
<tr><td valign="top">Work with OPs: RPM creation for deployment</td><td valign="top">lloyd/petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695956</td></tr>
<tr><td valign="top">Work with OPs: Dev deployment</td><td valign="top">lloyd/petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695957</td></tr>
<tr><td valign="top">Work with OPs: initial Dev test</td><td valign="top">lloyd/petef</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Work with OPs: load test code and deploy</td><td valign="top">lloyd/petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695952
* Issue 504: repair load_gen</td></tr>
<tr><td valign="top">Update GitHub Scaling to 1M</td><td valign="top">lloyd</td><td valign="top">in progress</td><td valign="top">
* https://github.com/mozilla/browserid/wiki/Scaling-to-1M</td></tr>


</table>
|-
| valign="top" | BrowserID arch review
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*https://intranet.mozilla.org/Services/Ops/BrowserID#Architecture


== Table for OPs ==
|-
<table width="100%" border="1" style="border-collapse:collapse">
| valign="top" | Track RPM and deployment work in Dev
<tr><td width="25%">Task/Project</td><td width="15%">Owner</td><td width="15%">Status</td><td width="45%">Notes, Links, Tickets, Issues</td></tr>
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Bug 695596


<tr><td valign="top">Build-out planning</td><td valign="top">OPs</td><td valign="top">in progress</td><td valign="top">
|-
* https://intranet.mozilla.org/Services/Ops/BrowserID
| valign="top" | Track Dev build-out  
* 11/12-11/14 posting of all four environments on OPs wiki</td></tr>
| valign="top" | jbonacci
<tr><td valign="top">VM setup/config for Dev and Test/CI</td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
| valign="top" | completed
* https://intranet.mozilla.org/Services/Ops/BrowserID</td></tr>
| valign="top" |
<tr><td valign="top">HW orders for Prod:SCL2</td><td valign="top">OPs</td><td valign="top">in progress</td><td valign="top">
*see Bug 695957 - fixed
* Bugs 695941, 697477
* https://intranet.mozilla.org/Services/Ops/BrowserID</td></tr>
<tr><td valign="top">HW orders for Prod:PHX1</td><td valign="top">OPs</td><td valign="top">in progress</td><td valign="top">
* Bugs 695941, 697477
* https://intranet.mozilla.org/Services/Ops/BrowserID
* prod scl2 hardware has also landed
* still waiting on new load balancers (not a blocker for testing)
* also waiting for HW for PHX1</td></tr>
<tr><td valign="top">HW orders for Stage</td><td valign="top">OPs</td><td valign="top">in progress</td><td valign="top">
* Bugs 695941, 697477
* https://intranet.mozilla.org/Services/Ops/BrowserID
* staging/loadtest environment hardware has landed and is mostly built
* still waiting on NetOPs</td></tr>
<tr><td valign="top">Logging </td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
* Bugs 695916, 695963, 695967</td></tr>
<tr><td valign="top">Metrics </td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
* Bugs 695916, 695963, 695967</td></tr>
<tr><td valign="top">RPM design/build/test</td><td valign="top">petef/lloyd</td><td valign="top">in progress</td><td valign="top">
* Bug 695956
* Latest pull request was 11/14</td></tr>
<tr><td valign="top">Set up Load test env</td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695952</td></tr>
<tr><td valign="top">Zeus configuration</td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695943</td></tr>
<tr><td valign="top">MySQL</td><td valign="top">petef</td><td valign="top">in progress</td><td valign="top">
* Bug 695972
* mysql replication/monitoring strategy fleshed out (doc coming)</td></tr>
<tr><td valign="top">Dev environment deploy and test</td><td valign="top">petef/lloyd</td><td valign="top">in progress</td><td valign="top">
* Bug 695957
* 11/7-11/14 DEV env build-out and test deployments
* on track for week of 11/13</td></tr>
<tr><td valign="top">Stage environment deploy and test</td><td valign="top">petef/lloyd</td><td valign="top">planned</td><td valign="top">
* Bug 695946
* on track for week of 11/13</td></tr>
<tr><td valign="top">Stage environment load test</td><td valign="top">petef/QA</td><td valign="top">planned</td><td valign="top">
* none</td></tr>
<tr><td valign="top">Prod environment deploy and test</td><td valign="top">petef</td><td valign="top">planned</td><td valign="top">
* Bugs 695946, 695960, 698044</td></tr>
<tr><td valign="top">Test/CI environment deploy and test</td><td valign="top">petef</td><td valign="top">planned</td><td valign="top">
* Bug 695958
* Planned for week of 11/20</td></tr>
<tr><td valign="top">Other: working with infra-sec on documentation, vuln scans, more reliable CEF logging, and RSBAC rollout</td><td valign="top">OPs</td><td valign="top">in progress</td><td valign="top">
* none</td></tr>


</table>
|-
| valign="top" | Track Stage build-out
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Bugs 695941 (fixed), 695944 - fixed, 703497


== Quick Summaries: July - Oct ==
|-
| valign="top" | Track Prod build-out
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Bugs 695941 (fixed), 695943, 695946, 695955, 695956 (fixed), 695960, 695972, 698044, 703497
 
|-
| valign="top" | Track DEV load test development
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Issue 504
 
|-
| valign="top" | Track OPs load test deployment
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Bug 695952
 
|-
| valign="top" | Track OPs dashboard/monitoring work
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*see Bugs 695961, 695963, 695967
 
|-
| valign="top" | Unit test tutorial/wiki
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*Front-End unit tests, Back-End unit tests, Headless unit tests
*https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install
 
|-
| valign="top" | Stability testing Dev env
| valign="top" | QA
| valign="top" | completed
| valign="top" | 
*Server: https://dev-browserid.services.mozilla.com
*RP: http://beer.mtv1.dev.svc.mozilla.com
 
|-
| valign="top" | Stability testing Stage env
| valign="top" | QA
| valign="top" | completed
| valign="top" |
*RP1: http://beta.myfavoritebeer.org/
*RP2: http://beta.myfavoritebooze.org/
*RP3: http://beta.myfavoriteshow.org/
*RP4: http://carrera.databits.net:9999/
*Server: https://diresworb.org/
 
|-
| valign="top" | Track Load testing Stage env
| valign="top" | QA/OPs
| valign="top" | completed
| valign="top" |
*none
 
|-
| valign="top" | Stability testing Prod env
| valign="top" | QA
| valign="top" | completed
| valign="top" |
*none
 
|-
| valign="top" | Load testing Prod env
| valign="top" | QA/Dev/OPs
| valign="top" | completed
| valign="top" |
*none
 
|-
| valign="top" | QA node/npm "easy" installs
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*Final steps: completed information for installing using unit tests
*https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install
 
|-
| valign="top" | Discussion of Android compatibility for BrowserID
| valign="top" | QA/Dev
| valign="top" | completed
| valign="top" |
*2.2, 2.3, 3.x and newer vs. supporting older versions
* Percentage-wise (of users) 2.2 and newer seems ok
 
|-
| valign="top" | Final QA wiki updates
| valign="top" | jbonacci
| valign="top" | completed
| valign="top" |
*https://wiki.mozilla.org/QA/BrowserID/TestPlan
*https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta
*https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install
*https://wiki.mozilla.org/QA/Browser_Technologies/Services/Releases/BrowserID
*https://intranet.mozilla.org/QA/StageLoadTesting
*https://intranet.mozilla.org/QA/StageAndProdMonitoring
*https://wiki.mozilla.org/QA/BrowserID/OPsBuildOut
*Internal site: TBD
 
 
|}
 
<br>
 
== Dev  ==
 
{| width="100%" border="1" style="border-collapse:collapse"
|-
| width="25%" | Task/Project
| width="15%" | Owner
| width="15%" | Status
| width="45%" | Notes, Links, Tickets, Issues
|-
| valign="top" | Initial plan for Scaling to 1 million users
| valign="top" | Dev team
| valign="top" | completed
| valign="top" |
*https://github.com/mozilla/browserid/wiki/Scaling-to-1M
 
|-
| valign="top" | Metrics dashboard
| valign="top" | Dev team
| valign="top" | completed
| valign="top" |
*see Bug 690107
 
|-
| valign="top" | Maintain "all-in-one" BrowserID
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*Git and RPM methods are always available
 
|-
| valign="top" | GitHub repository re-organization
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*Issue 503: improve repository organization
 
|-
| valign="top" | Better logging
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*GitHub Issues 119, 537, 541, 536, 530, 529, etc.
*Dev needs to work with OPs to get *-error logs, archiving, and log rotation
 
|-
| valign="top" | Process splitting: browserid, verifier, keysigner, dbwriter
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*GitHub Issue 460
 
|-
| valign="top" | Second client (dev.)(beta.)myfavoritebooze.org
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*support 'keep me signed in' feature
 
|-
| valign="top" | Developer Engagement Plan
| valign="top" | Dev/PM teams
| valign="top" | completed
| valign="top" |
*https://wiki.mozilla.org/Engagement/Developer_Engagement/BrowserIDPlan
 
|-
| valign="top" | Crypto discussions and work
| valign="top" | Dev/Security teams
| valign="top" | completed
| valign="top" |
*GitHub issues: various
 
|-
| valign="top" | Work with OPs: RPM creation for deployment
| valign="top" | lloyd/petef
| valign="top" | completed
| valign="top" |
*Bug 695956 - fixed
 
|-
| valign="top" | Work with OPs: Dev deployment
| valign="top" | lloyd/petef
| valign="top" | completed
| valign="top" |
*Bug 695957 - fixed
 
|-
| valign="top" | Work with OPs: initial Dev test
| valign="top" | lloyd/petef
| valign="top" | completed
| valign="top" |
* See bullets below
 
|-
| valign="top" | Work with OPs: code for Stage
| valign="top" | lloyd/petef
| valign="top" | completed
| valign="top" |
* Important open issues for Dev and Stage:
* 504: repair load_gen
* 560: move bcrypt to webhead
* 561: handle mysql master failover in dbwriter process
* 566: heartbeat should not check backend servers
* 582: improved file based configuration
 
|-
| valign="top" | Work with OPs: load test code and deploy
| valign="top" | lloyd/petef
| valign="top" | completed
| valign="top" |
*Bug 695952
*Issue 504: repair load_gen
*https://github.com/mozilla/browserid/blob/dev/bin/load_gen
 
|-
| valign="top" | Update GitHub Scaling to 1M
| valign="top" | lloyd
| valign="top" | completed
| valign="top" |
*https://github.com/mozilla/browserid/wiki/Scaling-to-1M
 
|-
| valign="top" | Use of node.js v0.6.x
| valign="top" | lloyd, benadida
| valign="top" | completed
| valign="top" |
*none
 
|-
| valign="top" | Unit tests, headless front-end tests
| valign="top" | stomlinson, bendadida
| valign="top" | completed
| valign="top" |
*none
 
|}
 
<br>
 
== OPs  ==
 
{| width="100%" border="1" style="border-collapse:collapse"
|-
| width="25%" | Task/Project
| width="15%" | Owner
| width="15%" | Status
| width="45%" | Notes, Links, Tickets, Issues
 
|-
| valign="top" | Test/CI environment deploy and test
| valign="top" | petef
| valign="top" | Planned
| valign="top" |
*Bug 695958
*http://hudson.build.mtv1.svc.mozilla.com/job/browserid-server
*Need new DEV public, so just need to get a public VIP/VLAN on our dev/test zeus cluster after that and auto deployment, old dev and move to new dev new dev --> dev.diresworb.org (or equivalent)
*Request sent to NetOPs for external IPs for Dev and Test
*Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
*Stage to be up in same time period, tested, with the addition of load testing
 
|-
| valign="top" | New "cross-team env" VM deploy and test
| valign="top" | lloyd/petef
| valign="top" | Planned
| valign="top" |
*Need an OPs bug for this
*This Env to be shared by various teams: BrowserID, Soup, AMO, etc.
 
|-
| valign="top" | Jenkins/Hudson for BrowserID-Server
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*http://hudson.build.mtv1.svc.mozilla.com/job/browserid-server
 
|-
| valign="top" | Build-out planning
| valign="top" | OPs
| valign="top" | completed<br>
| valign="top" |
*https://intranet.mozilla.org/Services/Ops/BrowserID
*11/12-11/14 posting of all four environments on OPs wiki
 
|-
| valign="top" | VM setup/config for Dev and Test/CI
| valign="top" | petef
| valign="top" | completed<br>
| valign="top" |
*https://intranet.mozilla.org/Services/Ops/BrowserID
 
|-
| valign="top" | HW orders for Prod:SCL2
| valign="top" | OPs
| valign="top" | completed<br>
| valign="top" |
*Bugs 695941 (fixed), 697477 - fixed
*https://intranet.mozilla.org/Services/Ops/BrowserID
 
|-
| valign="top" | HW orders for Prod:PHX1
| valign="top" | OPs
| valign="top" | completed<br>
| valign="top" |
*Bugs 695941 (fixed), 697477 - fixed
*https://intranet.mozilla.org/Services/Ops/BrowserID
*prod scl2 hardware has also landed
*still waiting on new load balancers (not a blocker for testing)
*also waiting for HW for PHX1
 
|-
| valign="top" | HW orders for Stage
| valign="top" | OPs
| valign="top" | completed
| valign="top" |
*Bugs 695941 (fixed), 697477 - fixed
*https://intranet.mozilla.org/Services/Ops/BrowserID
*staging/loadtest environment hardware has landed and is mostly built
*still waiting on NetOPs
 
|-
| valign="top" | Logging
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*Bugs 695916, 695963, 695967
*/var/log/browserid on webheads, secure webheads, keysigners
*/var/log/zeus on Zeus servers
* Still need log archiving and rotation
* Still need *-error logs
* TBD for database
 
|-
| valign="top" | Metrics
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*Bugs 695916, 695963, 695967
 
|-
| valign="top" | RPM design/build/test
| valign="top" | petef/lloyd
| valign="top" | completed
| valign="top" |
*Bug 695956 - fixed
*Latest pull request was 11/14
 
|-
| valign="top" | Set up Load test env
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*Bug 695952
 
|-
| valign="top" | Monitoring/Dashboards
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*All in Graphite/Pencil
 
|-
| valign="top" | Zeus configuration
| valign="top" | petef/atoll
| valign="top" | completed
| valign="top" |
*Bugs 695943, 705922, 705922, 706408, 706410, 706411, 704638, 708321
 
|-
| valign="top" | MySQL
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*Bug 695972
*mysql replication/monitoring strategy fleshed out (doc coming)
*DB/MySQL failover work, Bug 695972
 
|-
| valign="top" | Initial Dev environment deploy and test
| valign="top" | petef/lloyd
| valign="top" | completed
| valign="top" |
*Bug 695957 - fixed
*11/7-11/14 DEV env build-out and test deployments
*First official deploymant: train-2011.11.17
*Dev Server: https://dev-browserid.services.mozilla.com
*Dev RP (temp): http://beer.mtv1.dev.svc.mozilla.com
*OPs needs to configure persistence on the test beer site otherwise beer names are not sticking across sessions
*Need to see what happens on Stage (once it is up and running again)
*OPs needs to puppetize and config mongodb on Dev, so my guess is Stage will need this also
*Request sent to NetOPs for external IPs for Dev and Test
*Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
*Stage to be up in same time period, tested, with the addition of load testing
 
|-
| valign="top" | Initial Stage environment deploy and test
| valign="top" | petef/lloyd
| valign="top" | completed
| valign="top" |
*Bug 695946, 703497, 695944 - fixed
*First official deploymant: train-2011.11.17
*Stage Server: https://stage-browserid.services.mozilla.com
*Stage RP (temp): http://carrera.databits.net:9999
*Delayed from last week: dbwriter, networking and our new load balancers
*Stage still needs log rotation and archiving
*Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
*Stage to be up in same time period, tested, with the addition of load testing
 
|-
| valign="top" | Stage environment load test
| valign="top" | petef/QA
| valign="top" | completed
| valign="top" |
*Planned: benchmarks and performance in Stage.
*QA needs more detailed instructions for running load_gen off of the "client" load generators.
 
|-
| valign="top" | Prod environment deploy, test, and load test
| valign="top" | petef
| valign="top" | completed
| valign="top" |
*Bugs 695946, 695960, 698044, 705922, 703497
*Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
*Stage to be up in same time period, tested, with the addition of load testing
 
|-
| valign="top" | Other: working with infra-sec on documentation, vuln scans, more reliable CEF logging, heartbeats
| valign="top" | OPs
| valign="top" | completed
| valign="top" |
*none
 
|-
| valign="top" | Puppet configuration (transferable to Stage)
| valign="top" | OPs
| valign="top" | completed
| valign="top" |
*Bug 695956 - fixed
 
|-
| valign="top" | RSBAC, crypto, ACLs work
| valign="top" | OPs
| valign="top" | completed
| valign="top" |
*Bug 695955
*Completed on Dev
*Planned for Stage and then Prod
 
|}
 
<br>
 
== Quick Summaries/History: July - Oct ==
QA:
QA:
* Focus for Summer and Fall
* Focus for Summer and Fall
Line 229: Line 604:
* Bug 650863 - SSL cert needed for browserid.org
* Bug 650863 - SSL cert needed for browserid.org
* Bug 695955 - browserid: write rsbac security policies for all host types
* Bug 695955 - browserid: write rsbac security policies for all host types
 
* Bug 705023 - Set browserid's user UID to something lower than 500
* Bug 705033 - Change the way nodejs starts browserid (no functionality change)
* Bug 705922 - zlb1/2/3.pub.scl2.svc.m.c: please kickstart, yum update, reboot, puppetize
* Bug 706408 - monitor zeus log sync queue size
* Bug 706410 - deploy zeus log sync ssh key to zlb*.pub.scl2.svc
* Bug 706411 - fix zeus log sync script to report rsync errors via cronmail
* Bug 704638 - deploy zlb*.pub.scl2


== GitHub Issues ==
== GitHub Issues ==

Latest revision as of 19:51, 19 December 2011

Back To ==> Mozilla BT Services

Intro

This is the main QA site for the BrowserID OPs Build-Out. All tasks, projects, status, and other important information related to the Build-Out will be listed here and updated frequently.

  • Overall/Master Schedule (details): to be added
  • Build-Out Schedule per four environments: November/December
  • Deployment Schedule per four environments: November/December
  • Load Test Schedule per four environments: November/December
  • QA Test Schedule per four environments: November/December

QA

Task/Project Owner Status Notes, Links, Tickets, Issues
Track Test/CI build-out jbonacci Planned
Track new "cross-team env" build-out jbonacci Planned
  • This will be a VM environment like old Beta
  • This VM env is to be shared by QA teams for BrowserID, AMO, Soup, etc.


Stability testing Test/CI env QA Planned
  • need from with Dev and OPs
Stability testing new "cross-team env" QA Planned
  • need from with Dev and OPs
  • This will be a VM environment like old Beta
Track OPs BrowserID build-out planning and HW orders jbonacci completed
  • see Bug 697477 - fixed
BrowserID arch review jbonacci completed
Track RPM and deployment work in Dev jbonacci completed
  • see Bug 695596
Track Dev build-out jbonacci completed
  • see Bug 695957 - fixed
Track Stage build-out jbonacci completed
  • see Bugs 695941 (fixed), 695944 - fixed, 703497
Track Prod build-out jbonacci completed
  • see Bugs 695941 (fixed), 695943, 695946, 695955, 695956 (fixed), 695960, 695972, 698044, 703497
Track DEV load test development jbonacci completed
  • see Issue 504
Track OPs load test deployment jbonacci completed
  • see Bug 695952
Track OPs dashboard/monitoring work jbonacci completed
  • see Bugs 695961, 695963, 695967
Unit test tutorial/wiki jbonacci completed
Stability testing Dev env QA completed
Stability testing Stage env QA completed
Track Load testing Stage env QA/OPs completed
  • none
Stability testing Prod env QA completed
  • none
Load testing Prod env QA/Dev/OPs completed
  • none
QA node/npm "easy" installs jbonacci completed
Discussion of Android compatibility for BrowserID QA/Dev completed
  • 2.2, 2.3, 3.x and newer vs. supporting older versions
  • Percentage-wise (of users) 2.2 and newer seems ok
Final QA wiki updates jbonacci completed



Dev

Task/Project Owner Status Notes, Links, Tickets, Issues
Initial plan for Scaling to 1 million users Dev team completed
Metrics dashboard Dev team completed
  • see Bug 690107
Maintain "all-in-one" BrowserID lloyd completed
  • Git and RPM methods are always available
GitHub repository re-organization lloyd completed
  • Issue 503: improve repository organization
Better logging lloyd completed
  • GitHub Issues 119, 537, 541, 536, 530, 529, etc.
  • Dev needs to work with OPs to get *-error logs, archiving, and log rotation
Process splitting: browserid, verifier, keysigner, dbwriter lloyd completed
  • GitHub Issue 460
Second client (dev.)(beta.)myfavoritebooze.org lloyd completed
  • support 'keep me signed in' feature
Developer Engagement Plan Dev/PM teams completed
Crypto discussions and work Dev/Security teams completed
  • GitHub issues: various
Work with OPs: RPM creation for deployment lloyd/petef completed
  • Bug 695956 - fixed
Work with OPs: Dev deployment lloyd/petef completed
  • Bug 695957 - fixed
Work with OPs: initial Dev test lloyd/petef completed
  • See bullets below
Work with OPs: code for Stage lloyd/petef completed
  • Important open issues for Dev and Stage:
  • 504: repair load_gen
  • 560: move bcrypt to webhead
  • 561: handle mysql master failover in dbwriter process
  • 566: heartbeat should not check backend servers
  • 582: improved file based configuration
Work with OPs: load test code and deploy lloyd/petef completed
Update GitHub Scaling to 1M lloyd completed
Use of node.js v0.6.x lloyd, benadida completed
  • none
Unit tests, headless front-end tests stomlinson, bendadida completed
  • none


OPs

Task/Project Owner Status Notes, Links, Tickets, Issues
Test/CI environment deploy and test petef Planned
  • Bug 695958
  • http://hudson.build.mtv1.svc.mozilla.com/job/browserid-server
  • Need new DEV public, so just need to get a public VIP/VLAN on our dev/test zeus cluster after that and auto deployment, old dev and move to new dev new dev --> dev.diresworb.org (or equivalent)
  • Request sent to NetOPs for external IPs for Dev and Test
  • Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
  • Stage to be up in same time period, tested, with the addition of load testing
New "cross-team env" VM deploy and test lloyd/petef Planned
  • Need an OPs bug for this
  • This Env to be shared by various teams: BrowserID, Soup, AMO, etc.
Jenkins/Hudson for BrowserID-Server petef completed
Build-out planning OPs completed
VM setup/config for Dev and Test/CI petef completed
HW orders for Prod:SCL2 OPs completed
HW orders for Prod:PHX1 OPs completed
HW orders for Stage OPs completed
Logging petef completed
  • Bugs 695916, 695963, 695967
  • /var/log/browserid on webheads, secure webheads, keysigners
  • /var/log/zeus on Zeus servers
  • Still need log archiving and rotation
  • Still need *-error logs
  • TBD for database
Metrics petef completed
  • Bugs 695916, 695963, 695967
RPM design/build/test petef/lloyd completed
  • Bug 695956 - fixed
  • Latest pull request was 11/14
Set up Load test env petef completed
  • Bug 695952
Monitoring/Dashboards petef completed
  • All in Graphite/Pencil
Zeus configuration petef/atoll completed
  • Bugs 695943, 705922, 705922, 706408, 706410, 706411, 704638, 708321
MySQL petef completed
  • Bug 695972
  • mysql replication/monitoring strategy fleshed out (doc coming)
  • DB/MySQL failover work, Bug 695972
Initial Dev environment deploy and test petef/lloyd completed
  • Bug 695957 - fixed
  • 11/7-11/14 DEV env build-out and test deployments
  • First official deploymant: train-2011.11.17
  • Dev Server: https://dev-browserid.services.mozilla.com
  • Dev RP (temp): http://beer.mtv1.dev.svc.mozilla.com
  • OPs needs to configure persistence on the test beer site otherwise beer names are not sticking across sessions
  • Need to see what happens on Stage (once it is up and running again)
  • OPs needs to puppetize and config mongodb on Dev, so my guess is Stage will need this also
  • Request sent to NetOPs for external IPs for Dev and Test
  • Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
  • Stage to be up in same time period, tested, with the addition of load testing
Initial Stage environment deploy and test petef/lloyd completed
  • Bug 695946, 703497, 695944 - fixed
  • First official deploymant: train-2011.11.17
  • Stage Server: https://stage-browserid.services.mozilla.com
  • Stage RP (temp): http://carrera.databits.net:9999
  • Delayed from last week: dbwriter, networking and our new load balancers
  • Stage still needs log rotation and archiving
  • Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
  • Stage to be up in same time period, tested, with the addition of load testing
Stage environment load test petef/QA completed
  • Planned: benchmarks and performance in Stage.
  • QA needs more detailed instructions for running load_gen off of the "client" load generators.
Prod environment deploy, test, and load test petef completed
  • Bugs 695946, 695960, 698044, 705922, 703497
  • Goal is to have Dev, Test/CI, and Prod up and running and tested by QA by 12/9
  • Stage to be up in same time period, tested, with the addition of load testing
Other: working with infra-sec on documentation, vuln scans, more reliable CEF logging, heartbeats OPs completed
  • none
Puppet configuration (transferable to Stage) OPs completed
  • Bug 695956 - fixed
RSBAC, crypto, ACLs work OPs completed
  • Bug 695955
  • Completed on Dev
  • Planned for Stage and then Prod


Quick Summaries/History: July - Oct

QA:

  • Focus for Summer and Fall
    • Weekly deployments to Beta environment
    • Verification of deployments to current Production environment
    • Test Plan and Weekly Trains wikis
  • Tracking Dev activity for scaling to one million users
  • Tracking OPs activity for BrowserID environment build-outs

DEV:

  • Scalability planning started in the Summer.
  • Planning path for security review and production build-out was started in the Summer.
  • Metrics dashboard for identity
  • Code migration to MySQL
  • Command-line load generation and performance analysis
  • Debug improvements
  • Other areas shared with OPs: monitoring, logging improvements, RPM generation script, schema changes to improve database performance and scalability

OPs:

  • BrowserID env designs
  • Capacity planning
  • Hardware was ordered in October
  • Dev environment planning, build-out and configuration was started
  • RPM generation work was started
  • Other areas of work: node cluster, mysql strategy, monitoring strategy, logging, DB performance, scalability, zeus vips setup, nginx/apache, webhead/secure webhead planning
  • Dependent on Dev for keysigner design and split, dbwriter design and split, logging improvements, repository restructuring, process split, code integration for packaging, and the load-gen code
  • REF: Bug 695940 - BrowserID production tracking bug

Important Links


OPs Tickets

  • Bug 695940 - BrowserID production tracking bug
    • This is a meta bug for all the other OPs tickets


Other Tickets

  • Bug 690107 - Build dashboards for identity
  • Bug 644776 - Security Review For BrowserID
  • Bug 692247 - Accessibility review for revised design of BrowserID
  • Bug 694073 - Need socketlabs SMTP parameters for browserid
  • Bug 650863 - SSL cert needed for browserid.org
  • Bug 695955 - browserid: write rsbac security policies for all host types
  • Bug 705023 - Set browserid's user UID to something lower than 500
  • Bug 705033 - Change the way nodejs starts browserid (no functionality change)
  • Bug 705922 - zlb1/2/3.pub.scl2.svc.m.c: please kickstart, yum update, reboot, puppetize
  • Bug 706408 - monitor zeus log sync queue size
  • Bug 706410 - deploy zeus log sync ssh key to zlb*.pub.scl2.svc
  • Bug 706411 - fix zeus log sync script to report rsync errors via cronmail
  • Bug 704638 - deploy zlb*.pub.scl2

GitHub Issues


BrowserID Environments

See the main OPs BrowserID site for full details:


Load Testing

  • Load_Gen code and test - Dev work is still in progress
  • See the following bug and issue:
    • Bug 695952 - browserid: loadtest
    • Issue 504: repair load_gen
  • Load Test hardware (per OPs)
    • There are eight dedicated client machines to send load
      • Need machine name information
    • There is also a dedicated VLAN that does not hit the firewall to push more load, if required.
  • Load Test information per four environments: TBD


Dashboards and Monitoring


Bugs vs. GitHub Issues

  • TBD


Back To ==> Mozilla BT Services