User:Armenzg:L10n build roadmap

From MozillaWiki
Jump to: navigation, search

NOTE: Just look at sections "Goals" and "Patches" to have the latest info and to don't be overwhelmed

Goals

  • Run L10n repackages in buildbot in parallel which requires:
    • getting logic from tinderbox client code into make targets and buildbot
      • get en-US binary with make target
      • upload packages with correct naming
      • report to tinderbox pages
  • Make full switch from tinderbox to buildbot
  • Do l10n repackages in parallel:
    • in 1.9 (getting there)
    • in mozilla-central
  • Have L10n nightly updates which requires:
    • Script that will generate installer without missing entities (en-US entities added that are not yet added in a locale) - Gandalf might something in his Silme library
  • Integrate compare-locales.py as part of the steps instead of the old compare-locales.pl
    • requires checking out same source code cut off as the one used with en-US nightly

Patches/Status

This is to keep track of which bugs are open and what is the status of the patches. This might be the section out of this document that might me more up to date

description CVS hg status
456386 - tracking bug - repackages in 1.9 in *paralell* N/A delayed after 3.0.3 release
446038 - add more slaves to 1.9 automation N/A
398954 - use same source cut-off from en-US N/A no patches yet
451225 - master.cfg steps release N/A blocked in other bugs
456321 - "mozilla/other-licenses/7zstub/firefox" no patches yet
456351 - fix usage of variables no patches yet
453229 - TinderboxMailNotifier commited N/A TODO: File upstream bug

Patches done

description CVS hg status
451088 - master.cfg steps nightly commited N/A
451056 - buildbotcustom/l10n commited N/A
451461 - make target to prepare to upload to ftp commited commited
451466 - wget en-us binary commited commited

Roadmap + Status

All other milestones depend on M1. All others milestones do not block each other

Main fixes - porting to buildbot and make targets(M1)

  • Multiple builders pulling from same queue for locales
  • Do L10n nightly builds on staging 1.9
    • changes in master.cfg adding multiple build objects per locale'
    • buildbotcustom classes - NightlyL10n, L10nHelper and Build (it could be called BuildL10n or BuildQueueL10n)
    • be able to wget the right en-US binary (Bug 445254)
    • prepare repackages in the folder structure expected in ftp
    • The normal TinderboxMailNotifier just needs to allow WithProperties for the "tree" parameter
  • We will report to Firefox3.0-l10n and Mozilla-1.9.0-%locale%
    • with two TinderboxMailNotifier instances is enough since we can now use WithProperties with the "tree" parameter as mentioned before

Status

Best scenarios

In the next 4 scenarios I try to cover what is required so everything is working with buildbot and parallel builds

1.9 nightlies

The perfect world would be to have L10n repackages happening just after an en-US build, which we currently can't since en-US is being generated in dedicated machines with tinderbox code. It also requires to separate dependent/clobber builds from nightly builds

L10n nightlies in release automation (M2.a)

  • Use similar configuration as in staging-1.9
  • Turn off dedicated L10n slaves AND do L10n nightly builds in production-1.9

en-US nightly builds in release automation instead of dedicated machines (M2.b)

  • Run en-US nightly builds in production-1.9 automation instead of dedicated slaves using bootstrap and tinderbox code

Nightly & Periodic schedulers for en-US (M3 depends on M2.b)

  • Turn off $OfficialBuildMachinery to avoid generating nightly when generating dependent/clobber
  • Have a Nightly scheduler for en-US nightly builds and a Dependent scheduler for dependent/clobber builds

Trigger L10n repackages after en-US nightly (M4 depends on M3)

  • Use DependentL10n scheduler instead of NightlyL10n to trigger L10n nightly builds as soon as the Nightly scheduler finishes

1.9 releases

fake release in staging (M2)

  • Do a fake L10n release on staging 1.9
    • Make sure that ftp structure is proper and everything is uploaded
    • Download one of the fake releases and try to see if it would get an update
    • use shiped-locales instead of all-locales to determine which locales to repackages
    • what else to test for a fake release to be good?

production release (depends on M2)

  • Run production 1.9 releases

moz2 nightlies

To be added more

moz2 releases milestones

To be added more

Pollers milestones

To be added more

Open discussions

When is the right time to remove things

  • Once every 24 hours? How?

Reduce time before making installers

  • We have to check out the following code:
    • TBA
  • We have to configure

Using the configure file that comes by default checks for many more things than what we need when doing a repackage

  • Could we use another configure file?
  • Could we do "make -f client.mk configure-l10n" and check for less things that regular configure?
  • Could we move "make installers-foo" to a python script?

Checkout same source code

  • en-US builds should check out the 3AM source code - this way mac, linux and windows are exactly the same
  • l10n builds check out the en-US 3AM source code and the locale's tip (maybe the 3AM source code as well?? I think it makes more sense)

Compare-locales to be run from source in binary

  • We added the timestamp in application.ini to be able to checkout the same en-US source code that was used for a nightly - couldn't we compare with the entities that are in the installers?

Issues

  • "Force build" does not work with these changes - I think it is not a blocker, I know where it breaks but it might require changes in the Builder class that comes with buildbot which I do not want since we would be hacking buildbot


Bugs involved

Retrieved from "https://wiki.mozilla.org/index.php?title=User:Armenzg:L10n_build_roadmap&oldid=110032"