Back To ==> Mozilla BT Services
Intro
This is the main QA site for the BrowserID OPs Build-Out. All tasks, projects, status, and other important information related to the Build-Out will be listed here and updated frequently.
- Overall/Master Schedule (details): to be added
- Build-Out Schedule per four environments: November/December
- Deployment Schedule per four environments: November/December
- Load Test Schedule per four environments: November/December
- QA Test Schedule per four environments: November/December
- Missing latest information from #identity: 11/14 and newer...
QA
| Task/Project | Owner | Status | Notes, Links, Tickets, Issues |
| Track OPs BrowserID build-out planning and HW orders | jbonacci | in progress |
|
| BrowserID arch review | jbonacci | completed | |
| Track RPM and deployment work in Dev | jbonacci | in progress |
|
| Track Dev build-out | jbonacci | in progress |
|
| Track Stage build-out | jbonacci | in progress |
|
| Track Prod build-out | jbonacci | planned |
|
| Track Test/CI build-out | jbonacci | planned |
|
| Track DEV load test development | jbonacci | in progress |
|
| Track OPs load test deployment | jbonacci | in progress |
|
| Track OPs dashboard/monitoring work | jbonacci | in progress |
|
| Load test tutorial | jbonacci | planned |
|
| Logging for QA work | jbonacci | planned |
|
| Monitoring/dashboards tutorial | jbonacci | planned |
|
| Stability testing Dev env | QA | planned |
|
| Stability testing Stage env | QA | planned |
|
| Load testing Stage env | QA/OPs | planned |
|
| Stability testing Prod env | QA | planned |
|
| Stability testing Test/CI env | QA | planned |
|
| QA node/npm "easy" installs | jbonacci | ongoing | |
| Complete ID2 configuration | jbonacci/lloyd/petef | in progress |
|
| QA RPM installs to ID2 | jbonacci | ongoing |
|
| QA Test Plan updates | jbonacci | ongoing | |
| QA Test wiki updates | jbonacci | ongoing | |
| QA internal wiki | jbonacci | planned |
|
| Develop Manual Test App | jbonacci/ozten | planned |
|
| Port Unit Tests to Stage | jbonacci/stomlinson | planned |
|
| API automation | QA | planned |
|
- How to access the new environments from the "front end"?
- REF: https://intranet.mozilla.org/Services/Ops/BrowserID/ZeusConfig
dev: dev-browserid.services.mozilla.com stage: stage-browserid.services.mozilla.com prod: browserid.org
- I assume then the same clients will now point to the above...
Dev
| Task/Project | Owner | Status | Notes, Links, Tickets, Issues |
| Initial plan for Scaling to 1 million users | Dev team | completed | |
| Metrics dashboard | Dev team | completed |
|
| Maintain "all-in-one" BrowserID | lloyd | in progress |
|
| GitHub repository reorganization | lloyd | completed |
|
| Better logging | lloyd | in progress |
|
| Process splitting: browserid, verifier, keysigner, dbwriter | lloyd | completed |
|
| Second client (dev.)(beta.)myfavoritebooze.org | lloyd | completed |
|
| Developer Engagement Plan | Dev/PM teams | in progress | |
| Crypto discussions and work | Dev/Security teams | in progress |
|
| Work with OPs: RPM creation for deployment | lloyd/petef | in progress |
|
| Work with OPs: Dev deployment | lloyd/petef | in progress |
|
| Work with OPs: initial Dev test | lloyd/petef | planned |
|
| Work with OPs: load test code and deploy | lloyd/petef | in progress |
|
| Update GitHub Scaling to 1M | lloyd | in progress |
Table for OPs
| Task/Project | Owner | Status | Notes, Links, Tickets, Issues |
| Build-out planning | OPs | in progress | https://intranet.mozilla.org/Services/Ops/BrowserID, 11/12-11/14 posting of all four environments on OPs wiki |
| VM setup/config for Dev and Test/CI | petef | in progress | https://intranet.mozilla.org/Services/Ops/BrowserID |
| HW orders for Prod:SCL2 | OPs | in progress | Bugs 695941, 697477, https://intranet.mozilla.org/Services/Ops/BrowserID |
| HW orders for Prod:PHX1 | OPs | in progress | Bugs 695941, 697477, https://intranet.mozilla.org/Services/Ops/BrowserID, prod scl2 hardware has also landed, still waiting on new load balancers (not a blocker for testing), also waiting for HW for PHX1 |
| HW orders for Stage | OPs | in progress | Bugs 695941, 697477, https://intranet.mozilla.org/Services/Ops/BrowserID, staging/loadtest environment hardware has landed and is mostly built, still waiting on NetOPs |
| Logging | petef | in progress | Bugs 695916, 695963, 695967 |
| Metrics | petef | in progress | Bugs 695916, 695963, 695967 |
| RPM design/build/test | petef/lloyd | in progress | Bug 695956, Latest pull request was 11/14 |
| Set up Load test env | petef | in progress | Bug 695952 |
| Zeus configuration | petef | in progress | Bug 695943 |
| MySQL | petef | in progress | Bug 695972 - browserid: figure out & test mysql master/failover strategy, mysql replication/monitoring strategy fleshed out (doc coming) |
| Dev environment deploy and test | petef/lloyd | in progress | Bug 695957, 11/7-11/14 DEV env build-out and test deployments, on track for week of 11/13 |
| Stage environment deploy and test | petef/lloyd | planned | Bug 695946, on track for week of 11/13 |
| Stage environment load test | petef/QA | planned | none |
| Prod environment deploy and test | petef | planned | Bugs 695946, 695960, 698044 |
| Test/CI environment deploy and test | petef | planned | Bug 695958, Planned for week of 11/20 |
| Other: working with infra-sec on documentation, vuln scans, more reliable CEF logging, and RSBAC rollout | OPs | in progress | none |
Quick Summaries: July - Oct
QA:
- Focus for Summer and Fall
- Weekly deployments to Beta environment
- Verification of deployments to current Production environment
- Test Plan and Weekly Trains wikis
- Tracking Dev activity for scaling to one million users
- Tracking OPs activity for BrowserID environment build-outs
DEV:
- Scalability planning started in the Summer.
- Planning path for security review and production build-out was started in the Summer.
- Metrics dashboard for identity
- Code migration to MySQL
- Command-line load generation and performance analysis
- Debug improvements
- Other areas shared with OPs: monitoring, logging improvements, RPM generation script, schema changes to improve database performance and scalability
OPs:
- BrowserID env designs
- Capacity planning
- Hardware was ordered in October
- Dev environment planning, build-out and configuration was started
- RPM generation work was started
- Other areas of work: node cluster, mysql strategy, monitoring strategy, logging, DB performance, scalability, zeus vips setup, nginx/apache, webhead/secure webhead planning
- Dependent on Dev for keysigner design and split, dbwriter design and split, logging improvements, repository restructuring, process split, code integration for packaging, and the load-gen code
- REF: Bug 695940 - BrowserID production tracking bug
Important Links
- IRC: #identity
- BrowserID QA:
- BrowserID Dev:
- BrowserID OPs:
OPs Tickets
- Bug 695940 - BrowserID production tracking bug
- This is a meta bug for all the other OPs tickets
Other Tickets
- Bug 690107 - Build dashboards for identity
- Bug 644776 - Security Review For BrowserID
- Bug 692247 - Accessibility review for revised design of BrowserID
- Bug 694073 - Need socketlabs SMTP parameters for browserid
- Bug 650863 - SSL cert needed for browserid.org
- Bug 695955 - browserid: write rsbac security policies for all host types
GitHub Issues
- Link for Server issues: https://github.com/mozilla/browserid/issues
- Link for Client issues: https://github.com/lloyd/myfavoritebeer.org/issues
- OPs-related issues in GitHub:
- 560: move bcrypt to webhead
- 561: handle mysql master failover in dbwriter process
- 566: heartbeat should not check backend servers
- 571: update rpm build to match new layout
- 575: [rpmbuild] work around npm bug: re-fetch jwcrypto every time
BrowserID Environments
See the main OPs BrowserID site for full details:
- Dev: https://intranet.mozilla.org/Services/Ops/BrowserID#Development_Environment
- Test/CI: https://intranet.mozilla.org/Services/Ops/BrowserID#Test.2FCI_Environment
- Stage: https://intranet.mozilla.org/Services/Ops/BrowserID#Stage_Environment
- Prod: https://intranet.mozilla.org/Services/Ops/BrowserID#Prod_Environment
Load Testing
- Load_Gen code and test - Dev work is still in progress
- See the following bug and issue:
- Bug 695952 - browserid: loadtest
- Issue 504: repair load_gen
- Load Test hardware (per OPs)
- There are eight dedicated client machines to send load
- Need machine name information
- There is also a dedicated VLAN that does not hit the firewall to push more load, if required.
- There are eight dedicated client machines to send load
- Load Test information per four environments: TBD
Dashboards and Monitoring
- Metrics team links:
- Bug 690107 - Build dashboards for identity
- https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics2&path=identity/&file=identity.wcdf
- Requires LDAP access
- OPs bugs:
- Bug 695961 - browserid: verify statsd collection, build pencil dashboards
- Bug 695963 - browserid: coordinate log collection with metrics
- Bug 695967 - browserid: setup monitoring
- OPs links per four environments: TBD
Bugs vs. GitHub Issues
- TBD
Back To ==> Mozilla BT Services