Release Engineers

Nick Thomas (nthomas)

Bugs

bug 515421 - tracking bug for 3.5.x to 3.6.y
bug 515417 - tracking bug for 3.0.z to 3.6.y

3.5.x to 3.6.y

Based on notes for 3.0.15 -> 3.5.5 MU (rather than 2.0.0.20 one), with more "improvements".

3.5.5 to 3.6b2

Create these config files

CVS: mozilla/tools/patcher-configs/moz191-branch-major-update-patcher2.cfg
HG: build/tools/release/updates/moz191-firefox-{linux,mac,win32}-major.cfg

Snippet Generation

On fx-linux-1.9-slave2:

mkdir -p /builds/3.6b2-major-update-tests
cd /builds/3.6b2-major-update-tests/

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
cvs co -d patcher-configs mozilla/tools/patcher-configs
## get un-checked in configs 
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=413970 > patcher-configs/moz191-branch-major-update-patcher2.cfg
## end of configs get
cvs co -r UPDATE_PACKAGING_R10 -d patcher mozilla/tools/patcher
cd patcher
cvs co -r UPDATE_PACKAGING_R10 -d MozBuild mozilla/tools/release/MozBuild

# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R10 --config=../patcher-configs/moz191-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz191-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
mkdir 3.5.5-3.6b2
ln -s ../3.6b2/ftp 3.5.5-3.6b2/ftp
# Also delete 3.5.5 mars that are only using up space
rm -rf 3.5.5
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz191-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verify

Not really required when we're only going to push the test channels

Check that releasetest = beta = release. We know betatest should be different and point to ftp.m.o rather than the mirrors.

cd temp/firefox/3.5.5-3.6b2
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

Only pushing test snippets over here (for production also copy aus2.beta and aus2)

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091112-Firefox-3.5.5-3.6b2-MU-test

Enable test snippets

Leveraging an earlier backupsnip

# cltbld@aus2-staging
~/bin/pushsnip   20091112-Firefox-3.5.5-3.6b2-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-355-36b2-major/
cd /builds/verify/firefox-355-36b2-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
## get un-checked in configs
wget --no-check-certificate -O patch https://bug515421.bugzilla.mozilla.org/attachment.cgi?id=413972
patch -p3 < patch
## end of config get 
platform=linux   # or mac or win32
./verify.sh -c moz191-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Update verify analysis

Linux output (no locale differences):

diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/linux_build/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/moz2_slave/linux_build/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");
Only in source/firefox/icons: mozicon16.xpm
Only in source/firefox/icons: mozicon50.xpm
Only in source/firefox: old-homepage-default.properties
Only in source/firefox/res: broken-image.gif
Only in source/firefox/res: loading-image.gif

channel-prefs.js is fine - comment change ignorable, channel change is expected
the two xpm's were removed in bug 499226 but not added to removed-files.in
old-homepage-default.properties was removed in bug 504953 but not added to removed-files.in
the two gifs were converted to pngs in bug 420811 but added to a section of removed-files.in which is #ifdef XP_MACOSX

Mac output (no locale differences):

Only in source/Firefox.app/Contents/MacOS/components: oji.xpt
diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/macosx_build/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/moz2_slave/macosx_build/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");
Only in source/Firefox.app/Contents/MacOS: libwidget.rsrc
Only in source/Firefox.app/Contents/MacOS: old-homepage-default.properties
Only in source/Firefox.app/Contents: Plug-Ins
Only in source/Firefox.app/Contents/Resources: firefox-bin.rsrc
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target
1687563284    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov  2 15:59 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
1687563285    0 drwxr-xr-x    4 cltbld   cltbld        136 Nov 12 15:24 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
1687563286    0 drwxr-xr-x    2 cltbld   cltbld         68 Nov 12 15:24 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
1687563287    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov  2 15:59 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
1687563288    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov 12 15:24 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
1687563289    0 drwxr-xr-x    2 cltbld   cltbld         68 Nov 12 15:24 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib

Contents/MacOS/components/oji.xpt - email out to ted and smichaud
Contents/MacOS/libwidget.rsrc was removed by bug 464315 but not added to removed-files.in
Contents/Resources/firefox-bin.rsrc was removed by bug 363747 but not added to removed-files.in
Contents/Plug-Ins was removed by bug 520494, only empty dirs left
channel-prefs.js and old-homepage-default.properties as for linux

Win32 (no locale differences):

Only in source/bin: .autoreg
diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js
2c2
< pref("app.update.channel", "release");
---
> pref("app.update.channel", "beta");
Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Binary files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: old-homepage-default.properties
Only in source/bin: removed-files
Only in source/bin/res: broken-image.gif
Only in source/bin/res: loading-image.gif
Binary files source/bin/softokn3.chk and target/bin/softokn3.chk differ

.autoreg is normal on windows, as are the three chk'sand removed-files
everything else as for linux

3.5.5 to 3.6b4

Create these config files

CVS: mozilla/tools/patcher-configs/moz191-branch-major-update-patcher2.cfg
HG: build/tools/release/updates/moz191-firefox-{linux,mac,win32}-major.cfg

Snippet Generation

On fx-linux-1.9-slave2. Abbreviated from the 3.6b2 run because we're reusing the same working dir.

cd /builds/3.6b2-major-update-tests/

## get un-checked in configs 
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=414773 > patcher-configs/moz191-branch-major-update-patcher2.cfg
## end of configs get

# clean up 3.6b2 mar files
cd patcher
rm -rf temp/firefox/3.6b2/ftp/*

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz191-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
mkdir 3.5.5-3.6b4
ln -s ../3.6b4/ftp 3.5.5-3.6b4/ftp
# Replace the 3.5.5 mars that are using up space with 1 byte files,
# so we don't have to download them ever again (slow!!)
cd 3.5.5/ftp
for f in `ls *.*`; do echo > $f; done
cd ../../../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz191-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verify

Not really required when we're only going to push the test channels

Check that releasetest = beta = release. We know betatest should be different and point to ftp.m.o rather than the mirrors.

cd temp/firefox/3.5.5-3.6b4
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

Only pushing test snippets over here (for production also copy aus2.beta and aus2)

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091126-Firefox-3.5.5-3.6b4-MU-test

Enable test snippets

Leveraging an earlier backupsnip

# cltbld@aus2-staging
~/bin/pushsnip   20091126-Firefox-3.5.5-3.6b4-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-355-36b4-major/
cd /builds/verify/firefox-355-36b4-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
## get un-checked in configs
wget --no-check-certificate -O patch https://bug515421.bugzilla.mozilla.org/attachment.cgi?id=414770
patch -p3 < patch
## end of config get 
platform=linux   # or mac or win32
./verify.sh -c moz191-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Update verify analysis

No update found for es-MX, hr, ka, ku, mn, oc, sr because there's no a 3.6b4 build for those locales (only tested overlapping set of locales last time)

Linux output (all differences):

diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/linux_build/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/moz2_slave/linux_build/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");

which is expected

Mac output:

diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js
2c2
< pref("app.update.channel", "release");
---
> pref("app.update.channel", "beta");
Only in source/Firefox.app/Contents: Plug-Ins
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target
1700435169    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov  2 15:59 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
1700435170    0 drwxr-xr-x    4 cltbld   cltbld        136 Nov 26 15:18 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
1700435171    0 drwxr-xr-x    2 cltbld   cltbld         68 Nov 26 15:18 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
1700435172    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov  2 15:59 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
1700435173    0 drwxr-xr-x    3 cltbld   cltbld        102 Nov 26 15:18 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
1700435174    0 drwxr-xr-x    2 cltbld   cltbld         68 Nov 26 15:18 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib

channel-prefs.js and empty dirs

Win32 output:

Only in source/bin: .autoreg
diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js
2c2
< pref("app.update.channel", "release");
---
> pref("app.update.channel", "beta");
Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Binary files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Binary files source/bin/softokn3.chk and target/bin/softokn3.chk differ

.autoreg, removed/files and chk files all normal on Windows, channel-prefs.js as above

Locale specific differences (ml only, all platforms)

Only in source/Firefox.app/Contents/MacOS/searchplugins: yahoo-in.xml

approved change to search plugins for Fx3.6 (bug 475016), and should be in 3.5.6 too. We leave old searchplugins behind anyway.

3.0.z to 3.6.y

3.0.15 to 3.6b2

Patch these config files

CVS: mozilla/tools/patcher-configs/moz191-branch-major-update-patcher2.cfg
HG: build/tools/release/updates/moz191-firefox-{linux,mac,win32}-major.cfg

Snippet Generation

Reuse the setup we have from 3.5.5 to 3.6b2.

On fx-linux-1.9-slave2:

## get un-checked in configs 
cd /builds/3.6b2-major-update-tests/patcher-configs
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=413973 | patch -p0
## end of configs get

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
mkdir 3.0.15-3.6b2
ln -s ../3.6b2/ftp 3.0.15-3.6b2/ftp
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verify

Not really required when we're only going to push the test channels

Check that releasetest = beta = release. We know betatest should be different and point to ftp.m.o rather than the mirrors.

cd temp/firefox/3.0.15-3.6b2
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

Only pushing test snippets over here (for production also copy aus2.beta and aus2)

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091112-Firefox-3.0.15-3.6b2-MU-test

Enable test snippets

Leveraging an earlier backupsnip

# cltbld@aus2-staging
~/bin/pushsnip   20091112-Firefox-3.0.15-3.6b2-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3015-36b2-major/
cd /builds/verify/firefox-3015-36b2-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
## get un-checked in configs
wget --no-check-certificate -O patch https://bug515417.bugzilla.mozilla.org/attachment.cgi?id=413974
patch -p3 < patch
## end of config get 
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Update verify analysis

The simple combination of the 3.5.5 -> 3.6b2 results with the 3.0.15 -> 3.5.5 ones.

3.0.15 to 3.6b4

Patch these config files

CVS: mozilla/tools/patcher-configs/moz191-branch-major-update-patcher2.cfg
HG: build/tools/release/updates/moz191-firefox-{linux,mac,win32}-major.cfg

Snippet Generation

On fx-linux-1.9-slave2:

## get un-checked in configs 
cd /builds/3.6b2-major-update-tests/patcher-configs
cvs up -C moz19-branch-major-update-patcher2.cfg
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=414774 | patch -p0
## end of configs get

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd ../patcher/temp/firefox
mkdir 3.0.15-3.6b4
ln -s ../3.6b4/ftp 3.0.15-3.6b4/ftp
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verify

Not really required when we're only going to push the test channels

Check that releasetest = beta = release. We know betatest should be different and point to ftp.m.o rather than the mirrors.

cd temp/firefox/3.0.15-3.6b4
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

Only pushing test snippets over here (for production also copy aus2.beta and aus2)

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091126-Firefox-3.0.15-3.6b4-MU-test

Enable test snippets

Leveraging an earlier backupsnip

# cltbld@aus2-staging
~/bin/pushsnip   20091126-Firefox-3.0.15-3.6b4-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3015-36b4-major/
cd /builds/verify/firefox-3015-36b4-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
## get un-checked in configs
wget --no-check-certificate -O patch https://bug515417.bugzilla.mozilla.org/attachment.cgi?id=414776
patch -p3 < patch
## end of config get 
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Releases/Firefox 3.6 MU Tests/BuildNotes

Contents

Release Engineers

Bugs

3.5.x to 3.6.y

3.5.5 to 3.6b2

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.5.5 to 3.6b4

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.0.z to 3.6.y

3.0.15 to 3.6b2

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.0.15 to 3.6b4

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

Navigation menu

Releases/Firefox 3.6 MU Tests/BuildNotes

Release Engineers

Bugs

3.5.x to 3.6.y

3.5.5 to 3.6b2

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.5.5 to 3.6b4

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.0.z to 3.6.y

3.0.15 to 3.6b2

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

3.0.15 to 3.6b4

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Update verify analysis

Navigation menu

Search