Services/ServerAppGuidelines

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

• Server Environments
  • Sandbox
    • No puppet, just do stuff
    • Internal only
  • Dev
    • Uses puppet
    • Builds on checkin
    • Runs functional + unit tests on checkin
    • Devs have access to debug
    • Same config as stage (and prod, if possible)
  • Stage
    • Uses puppet
    • Requires an explicit update (see release process)
    • No app gets into stage without full functional and stress test coverage
  • Production
    • Uses puppet
    • Explicit push as per process below
    • Functional tests run periodically in prodl
• Release cycles
  • Emergency pushes
    • Must hit stage and pass functional tests. An unplanned outage (503 at Zeus level) is of higher value than pushing unverified code to prod.
    • Requires sign-off from key people (TBD) before we short-circuit process
    • QA signoff preferred, not mandated unless deemed necessary as a part of sign-off
  • Normal maintenance
    • Checkpoint on Wed AM, identify apps with pending updates that are stable + passing tests
    • Stage by EOD Wed, hand off to QA Thursday AM, start stress (95% load) test
    • QA signoff by EOD Monday (five days, three working days)
    • Push to prod on Tuesday
    • Debrief as needed at Wed meeting
    • If we fail on stress/QA tests, we punt to the next week cycle
  • Feature releases
    • Checkpoint on Wed AM
    • Ensure all supporting documentation for new/changed features is updated and in primary doc location
    • Push to stage by EOD Wed
    • QA + stress test as with normal releases, but for an extra week (12 days, eight working days) to ensure time to test new feature fully + extended run time to shake out issues

• Documentation
  • All new features and functional changes must be documented before the change can be pushed to stage
  • Format is TBD, but current front-runner is Sphinx

• Build and Test
  • Packaging
    • All apps must use the same package and install mechanism, with services-core as a base to provide consistent behaviour
    • Build scripts must follow a format that can be used with Hudson
  • Continuous Integration
    • All projects must be buildable by Hudson, and will build on each commit
    • Build/test failures must be fixed immediately or the change reverted
  • Functional and unit tests
    • Tests are required for all apps as a prerequisite for pushing to stage
    • Apps that are already in stage/prod require tests for all changes
    • All features must be tested fully (not just default paths)
    • Tests must run on each checkin, any test failures block push to stage