Tools are written by the team, used by QA (load test, etc.)
Does the project need/have:
- run in travis before merges
- python smoke test that isn’t tied to anything yet in dev cluster before stage
- small load test that isn’t tied to anything yet of dev cluster before stage
How are deployments done?
travis builds pr's
- before merge
jenkins builds ‘dev’ (master)
- deploy to dev cluster
- run smoke and load and then move to stage
staging doesn’t update that regularly
will follow the standard jenkins deploy by mid-late march
no git event that is triggering stage
no testing at stage yet
nothing that triggers from stage to production
Does the project automatically execute the required test types on checkin via a build server like jenkins or travis?
Can/should each of the test types fail the build on failed runs?
- smoke test should fail
- load test shouldn’t unless there is a regression - percent failure
percent failure - load test logs to influxdb, can use db to query whether or not the test regressed
Are the test results visible in treeherder?
- no, just travis right now
For a specific build, does it require acceptance testing or manual sign-off?
- minimal need during qa sign off
- push could be e2e tested via fmd
- might not be needed once prod slow rollout is added (ensure up time for servers, rollback if needed)
maybe a smoke test in production to validate client
infrequent releases, is it worth doing automated production smoke testing?
production monitoring tests could be valuable
There are 3 projects built for ‘push’:
- simplepush - not updated regularly, will be legacy but currently used
- loop push - same code base as simplepush but different configs, used only for loop/hello
- web push - not happening a while (late 2015?), the long term push solution
simplepush 1.5 will carry data, release to stage next week-ish
As of FF36 we will unthrottle loop, meaning we should see 3x the connection count from simple push and will need more servers
Have tested: 100k people connected, 1 notification per minute, load test push 2000 notification per second so it’s well covered. We have typically been seeing no more than 10s of notifications per second.
Capacity: 160k per server, 1.6mm across 10-node cluster
- Automate load test via FMD, ticket creation and server validation for deployments
- Determine production monitoring and smoke tests