CloudServices/SimplePushServer/Testing Notes

From MozillaWiki
Jump to: navigation, search

Tools are written by the team, used by QA (load test, etc.)

Tools

Does the project need/have:

Unit Tests

  • run in travis before merges

Smoke Tests

  • python smoke test that isn’t tied to anything yet in dev cluster before stage

Regression Tests

Integration Tests

e2e Tests

Load Tests

  • small load test that isn’t tied to anything yet of dev cluster before stage

Performance Tests

How are deployments done?

travis builds pr's

  • before merge

jenkins builds ‘dev’ (master)

  • deploy to dev cluster
  • run smoke and load and then move to stage

staging doesn’t update that regularly

will follow the standard jenkins deploy by mid-late march

  • who?

no git event that is triggering stage

no testing at stage yet

nothing that triggers from stage to production

Does the project automatically execute the required test types on checkin via a build server like jenkins or travis?

Can/should each of the test types fail the build on failed runs?

  • smoke test should fail
  • load test shouldn’t unless there is a regression - percent failure

percent failure - load test logs to influxdb, can use db to query whether or not the test regressed

Are the test results visible in treeherder?

  • no, just travis right now

For a specific build, does it require acceptance testing or manual sign-off?

  • minimal need during qa sign off
  • push could be e2e tested via fmd
  • might not be needed once prod slow rollout is added (ensure up time for servers, rollback if needed)

maybe a smoke test in production to validate client

infrequent releases, is it worth doing automated production smoke testing?

production monitoring tests could be valuable

There are 3 projects built for ‘push’:

  • simplepush - not updated regularly, will be legacy but currently used
  • loop push - same code base as simplepush but different configs, used only for loop/hello
  • web push - not happening a while (late 2015?), the long term push solution

simplepush 1.5 will carry data, release to stage next week-ish

As of FF36 we will unthrottle loop, meaning we should see 3x the connection count from simple push and will need more servers

Have tested: 100k people connected, 1 notification per minute, load test push 2000 notification per second so it’s well covered. We have typically been seeing no more than 10s of notifications per second.

Capacity: 160k per server, 1.6mm across 10-node cluster

Next Steps

  • Automate load test via FMD, ticket creation and server validation for deployments
  • Determine production monitoring and smoke tests