Bouncer 2 was staged. See https://bugzilla.mozilla.org/show_bug.cgi?id=304343.
Bouncer 2 is tentatively scheduled to be deployed in the second week of december.
Deployment will be a two-phase process.
- The update system for 220.127.116.11 will be used to test bouncer2 through the build snippets, which will point to a download2.mozilla.org.* URI for complete and partial .mar files.
- Once we have verified that works, a full migration will take place and for the 18.104.22.168 release we should be relying solely on Bouncer2.
The TODO list below describes what will happen chronologically.
Deployment TODO List
Stage the code on stage.m.c
Get Bouncer2 onto main cluster oremj, Dec 6
- Only PHP bounce script really needs to be on cluster!
- Create download1.mozilla.org; point at current bouncer installation
- Create download2.mozilla.org; point at bouncer2 installation
- CNAME download.mozilla.org to download1.mozilla.org
- Code part is pretty much done
- Need LDAP info from Aravind (talked to justin about it)
- Do we need to waterfall auth methods (fallback)?
Verify v2 links and file sums for all available products
- Load test URIs using grinder to assure req/s
- Load testing to be done on/around Friday the 8th.
Script to migrate mirrors
- Additional tables to migrate (v1->v2)
mirror_locations->files(re-load with loader)
mirror_location_mirror_map->downloadables(generated by sentry)
mirror_os->oss(manual entry is fine -- only 3)
mirror_products->product_versions(re-load with loader)
Auto-create per-locale entries for each product-- based on what locales? (will be handled by loader)
- Migrate patches for update system and partner updates as well. (identifiable by .complete.mar, .partial.mar)
- loader can put these files in just fine with this command:
./loader.py -L complete.mar -P /firefox/releases/1.5.08/update -S -W complete.mar -X complete.mar -x overwrite <SHA1SUMS
- Migrate download product counts from version 1 to version 2 (Lars will write a special script for this)
- When all is ready, switch to CNAME download2.mozilla.org Before 22.214.171.124
Load Testing Notes
- Possible bottlenecks that could have affected tests
- bandwidth limits
- connection limits (tcp, for test nodes)
- definitely NOT cpu limits (test nodes were bored)
- Areas we could tweak
- keys in downloadables table
- sql in main bounce script
- fix joins
- fix order by clause
- remove count increments, move all stats to http logs (could end up caching a particular mirror and perform DDOS attacks by mistake on mirror network)
Products to Migrate
Crossed out ones are already migrated to bouncer2. If one of the below are not to be migrated, remove them from the list and put them in "Products to NOT Migrate".
Products to NOT Migrate