CloudServices/Chronicle/Testing Notes

From MozillaWiki
Jump to: navigation, search

Test coverage

  • Unit Tests: yes
  • Smoke Tests: need
  • Regression Tests:
  • Integration Tests
  • e2e Tests
  • functional testing in firefox, uses theintern. need to see how we can use or add to this
  • Load and Performance Tests: neither of these two (load/performance) are likely to be an issue yet, but could be looked at later

How are deployments done?

will follow the standard jenkins deploy pipeline by end of q1

commit into branch -> PR -> code review -> travis build -> lint it manually -> run it -> merge

master is currently deployed manually to staging, will be formalized soon by benson

merge to master will trigger a build in travis which will then deploy to stage/jenkins

ops will do staging first and then production later

2 week sprints, deploy at the end of each, promoted to staging

Does the project automatically execute the required test types on checkin via a build server like jenkins or travis? yes, it will when the jenkins work is complete

Can/should each of the test types fail the build on failed runs? unit and e2e should fail, load is unknown until required

Are the test results visible in treeherder or another tool? just in travis/jenkins, will need to standardize

Dev build should run: unit, functional tests, eventually a smoke/small load test

Stage build should run: unit, functional tests, eventually some additional e2e or api tests and a full load test

Next Steps

  • Make sure unit and e2e/functional tests execute and can fail the build on jenkins
  • Expand e2e tests as needed to cover all features
  • Create a small load test to run in dev for every PR in travis (as much load as you can put on in ~5 minute max)
  • Create any scripts or tools that would ease the dev->stage->prod process (any repetitive qa steps), best if they can run in jenkins. automated linting is one example.
  • Create a full load test that can be run in stage in jenkins (where do we think the load will be in 6 months?)