User:Ewong/post mortem
From MozillaWiki
There were a total of 9 builds sendchanges [yes. it was THAT painful, but it could've been more than 9 as I had cheated and moved the tags forward.].
- I had included the following additional m-r csets:
- build 1 : [Sept. 27]
- failed at source step due to configure failure [couldn't find pkg-config and was fixed by adding --disable-compile-environment to mozconfig]
- aside for osx64 (which actually completed the build), Linux* and Windows compile failed.
- build 2 : [Sept. 28]
- source step due to a bad mozconfig items (no_tooltool=1, ac_add_options --disable-compile-environment)
- for Windows, failed at the tooltool step [missing tooltool token]
- for Linux*, failed at compile.
- for osx64, repacks failed at wget_enus due to bug 1307371.
- build 3 : [Sept. 29]
- failed at compile step due to what I had thought it was a clobber issue, so clobbered a few times.
- osx64 failed at repacks: bug 1307371
- solution: backout http://hg.mozilla.org/releases/mozilla-release/rev/b76bb66cef3e on relbranch.
- considered installing updated wget, but thought backing out the above cset was easier. Updating wget can be done later.
- solution: backout http://hg.mozilla.org/releases/mozilla-release/rev/b76bb66cef3e on relbranch.
- build 4 : [Oct. 3]
- compile step failure finally determined to be attributed to bug 1306543
- build 5 : [Oct. 3]
- Windows: failed at tooltool step (tooltool token still missing: but the error was a misnomer. The problem was that the fingerprints of hg.mo had changed. Needed to update all slaves.]
- Linux repacks failed: invalid mozconfig option --with--ccache
- OSX64 repacks failed: bug 1307371
- build 6: [Oct. 4]
- Windows: failed at compile step: ignored the vs2015u2 tooltool package
- fixed by adding . "$topsrcdir/build/mozconfig-win.common" to mozconfig
- Linux64: failed at compile due to bug 1293928 [retriggered and it was ok]
- Linux repacks: failed due to bug 1231349
- Windows: failed at compile step: ignored the vs2015u2 tooltool package
[At this point, we spent A LOT of time figuring bug 1231349; but at the end, we were just going nowhere. After consulting with the council, we decided to not ship Inspector, Chatzilla and Calendar for 2.46]
- build 7: [Dec. 12]
- Linux* : make_pkg failed : forgot to remove a line from suite/locales/Makefile.in
- OSX64: failed at compile step: was somehow still looking for lightning-based artifacts.
- build 8: [Dec 13]
- OSX64: failed at compile step: was somehow still looking for lightning-based artifacts.
- found out it was because a line in the mozconfig was still refering to lightning's mk file.
- OSX64: failed at compile step: was somehow still looking for lightning-based artifacts.
- build 9: [Dec 13]
- OSX64: failed at configure step: ignored the clang tooltool package
- 1) added $topsrcdir/build/universal/mozconfig to the macosx64's release-l10n
- failed as it couldn't find the necessary config path.
- 2) changed the |make config| buildbotcustom step to run |make config| on both i386 and x86_64 objdirs
- failed as it couldn't get the en-US builds (invalid objdir)
- in hindsight, had I sat down and thought more carefully (not so easy when in chicken-little mode), repacking didn't require both i386 and x86_64 configs. It only needed one.
- 3) removed the buildbotcustom changes and simply changed the "$topsrcdir/build/universal/mozconfig" (from 1), to "$topsrcdir/build/universal/local-mozconfig.common"
- Finally fixed the osx64 repacks. [Note: Instead of doing build 10, I had opted to just move the SEAMONKEY_2_46_BUILD9 and SEAMONKEY_2_46_RELEASE tags to the newer csets in c-r.]
- 1) added $topsrcdir/build/universal/mozconfig to the macosx64's release-l10n
- OSX64: failed at configure step: ignored the clang tooltool package
Once the builds and repacks completed, it was time to do the fakesign.
However, since archive.mozilla.org is cached, any uploads to the 'servers' needed some time before they appeared. As such, since fakesign triggered the l10n verifications and ultimately the updates and updates verifications, there would be a long wait. But the problem that I had with this triggering process is that FtpPoller wasn't ever triggered or if it did, it never polled the candidates directory.
So, after much head-desking/walling, I went and did the following:
- bin/buildbot sendchange --username=ewong --master=localhost:9010 --branch=post_signing -c "SeaMonkey 2.46build9" doit
And of course, since we're doing 2.46, things are busted with the l10n verifications and updates step.
- l10n verifications (iirc) wasn't that necessary so I let it go as red; though I'll go back and figure out what's busting it.
- linux l10n verification has *always* been red. After some 'fixes' (if you can really call it that), *everything* in the l10n verification is green. (sounds wrong, right?)
- Updates failed due to not finding the nightly-based candidates; but I fixed that with a few pushes.
- had to attempt at it a few times as after the fifth fix and the updates step was still failing, I had realized that I did not reconfigured the master. (facepalm)
- had quite a few instances of using nightly/ instead of candidates, so I went ahead and changed them in a manual patch on the master (hasn't been pushed).
- Note: I'm suggesting we just move all release related 'temp' candidates folders to the candidates/ section. Before changing the stage.mozilla.org to S3, I don't recall how nightly/ got copied to candidates/ (I suspect I'll find out probably later), but it's worth keeping candidates in candidates/.
- there are still some reminant instances of nightly somewhere. Need to spend some time to find them.
- aus2' authorized_keys2 was not updated with seabld's new key. Had to ping jlaz to help us out on that.
On to the langpackmove.sh step et. al.
- langpackmove.sh ran 'flawlessly'
- zipcopy.sh ran flawlessly
- make-checksums failed.
- the langpacks in the candidates directory all referred to 2.40 and at first I thought I had fubar'd the whole release and then realized that langpackmove actually copied the wrong files.
- it wasn't that case. The files were 'right'. Just the names were wrong.
- modified the langpackmove.sh script and re-ran it. Skipped the zipcopy script since it was ok.
- make-checksums still failed as I didn't have sha1sum available and so I copied the script from : http://superuser.com/questions/362304/installing-sha1sum-on-git-bash-mingw and adapted it to the make-checksums.sh script.
- reran make-checksums and everything ran smoothly.
- the langpacks in the candidates directory all referred to 2.40 and at first I thought I had fubar'd the whole release and then realized that langpackmove actually copied the wrong files.
- viruspush.sh failed
- for some reasons, my .netrc was clobbered (now empty).
- even with a .netrc, ftp wasn't ok and it probably stems from me doing the ftp from a windows system, which has no concept of 'chmod 600 .netrc'.
- so figured a different way and viruspush.sh went ok.
- mirrorpush.sh went smoothly
- final verification was green
- website patch was pushed.
- concerned that the cron job that updated the website might not be working, like what happened in 2.40.
- had to : ln -s WINNT_x86-msvc WINNT_x86-msvc-{x86,x64}
- perhaps we need to include this change in the part that uploads the files to aus2.
- concerned that the cron job that updated the website might not be working, like what happened in 2.40.
- backupsnip and pushsnip went well.
- Posted the "SeaMonkey 2.46 release" message to mozilla.dev.seamonkey and mozilla.support.seamonkey
- Realized the MD5SUM and SHA1SUMS files were missing, so I uploaded those files.
Notes: On a whole, I think this release could have been done a lot better than I did, especially with my chicken-little mentality when the release process went south at build #2. So yeah, looking forward to the next release.
