Media/WebRTC/libwebrtc Update Process

From MozillaWiki
< Media‎ | WebRTC
Jump to: navigation, search

This page describes the process for updating the in tree, vendored copy of libwebrtc using the fast-forward scripts.


  • NOTE: all steps below assume the user is in the bash shell.
  • Create a new bug for the fast-forward update. It should depend on the previous fast-forward bug. Bug 1800920 is an example.
  • Before each new merge cycle elm needs to be reset to Central.
    1. Ensure you have filed a bug for the next merge cycle, and further ensure that the next merge cycle bug has the previous merge cycle bug as a dependency.
    2. File a bug as a clone of Bug 1822294 and add your new bug as dependency to the next merge cycle bug.
    3. Wait for the repo admins to reset the repository.
    4. Celebrate
  • Builds are done after vendoring in each commit. A minimal, default .mozconfig is provided during the prep_repo step below.
  • Running the fast-forward process from macOS and Linux is supported. For more information on macOS, see here and here. TL;DR: If running from macOS, avoid Xcode and git while scripts are running, and you may also need to run this command ahead of time:
git config --global core.trustctime false
  • Starting with a fresh clone of elm is recommended for each new chromium milestone.
hg clone --stream ssh://
  • Verify the most recent commit is the .arcconfig update. If the most recent commit doesn't look like "Bug xxx - FLOAT - REPO-elm - update .arcconfig repo callsign", run the following commands:
 ./mach python dom/media/webrtc/third_party_build/
 hg push -r tip

Update steps

  • Start from the elm checkout
cd elm
  • Update the default config file for the next upstream milestone
bash dom/media/webrtc/third_party_build/
  • Prepare the git repository - Running will display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in our git repository. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.
bash dom/media/webrtc/third_party_build/
  • Prepare no-op tracking files for cherry-pick commits
bash dom/media/webrtc/third_party_build/
  • Save the newly updated patch-stack - run this in a bash shell
(source dom/media/webrtc/third_party_build/ ; \
 ./mach python dom/media/webrtc/third_party_build/ \
 --repo-path $MOZ_LIBWEBRTC_SRC \
 --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD \
 --separate-commit-bug-number $MOZ_FASTFORWARD_BUG
  • bootstrap and sanity build
./mach --no-interactive bootstrap --application-choice=browser && ./mach build
  • Run
bash dom/media/webrtc/third_party_build/
  • Push to elm - after the script completes with the message ===loop-ff=== Completed fast-foward to {short-sha}, run the following step to push the new version of libwebrtc to elm:
hg push -r tip

Operational Notes

Reacting to

Three main types of errors will cause to exit: the first is a git rebase conflict, the second is a failure to regenerate build files (most often due to changes or missing/added files), and the third is a build error due to API changes in upstream code.

  • When fixing rebase errors or making other changes to fix build issues in moz-libwebrtc (any changes under third_party/libwebrtc), mercurial commit messages follow the pattern:
Bug {current-fast-forward-bug-number} - (fix-{upstream-sha}) {description}

# after following all the other instructions from (running the
# generation script for files, committing possible 
# changes and doing a build), run the following command:
(source dom/media/webrtc/third_party_build/ ; 
 ./mach python dom/media/webrtc/third_party_build/ \
  --repo-path $MOZ_LIBWEBRTC_SRC \
  --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD )
  • If instructed to run, its auto-generated mercurial commit messages follow the pattern:
Bug {current-fast-forward-bug-number} - (fix-{upstream-sha}) {description} - file updates
  • When making changes to fix Mozilla code (code outside of third_party/libwebrtc) in response to changes upstream, mercurial commit messages follow the pattern:
Bug {current-fast-forward-bug-number} (MOZ) - {description}

If multiple kinds of changes are needed to fix build errors, commit hygiene is a must: (MOZ) patches contain changes to files outside of third_party/libwebrtc, (fix-xxx) commits contain changes to third_party/libwebrtc files, and finally "(fix-xxx) ... - file updates" contain only third_party/libwebrtc changes.

General tips

  • Stopping the manually is best done when in the Test build phase using Ctrl-c.
  • Rather than waiting until the end of the process update process, you may push to elm to confirm build fixes, or for others to follow along. It is best to push a commit that successfully builds, but this isn't a hard and fast rule. Sometimes, it may be helpful to push a commit that is broken to get help from others, or when the fixes are complicated enough that you feel a review is necessary.
  • If reveals an issue that is best dealt with in a followup, one can commit a temporary patch marked with REPO-elm in the first line of its description, i.e. the "title". The temporary commit allows to continue, whereas REPO-elm makes sure the commit cannot be pushed to any other mozilla repo than elm (and try).
  • While is running, in a separate terminal window it may be helpful to see the overall progress by running:
tail -f .moz-fast-forward/logs/log-loop-ff.txt | grep loop-ff
  • GYP generated Java files
  • If instructed to run, note that has more context than and offers information that is relevant: Specifically, the guidance about tip::tip:
   If you have previously run successfully, there may be a new
   change to third_party/libwebrtc that should be extracted and added to
   the patch stack in github.  It may be as easy as running:
       ./mach python dom/media/webrtc/third_party_build/ tip::tip
       mv mailbox.patch /Users/jan-ivar/moz/elm/.moz-fast-forward/moz-libwebrtc
       (cd /Users/jan-ivar/moz/elm/.moz-fast-forward/moz-libwebrtc && \
        git am mailbox.patch)
Replace tip here with your (fix-xxx) mercurial patch, if it is not at tip (e.g. if it is followed by (fix-xxx) - file updates).

Recovering from errors

  • If a particularly complicated rebase conflict is encountered, it is possible to start over the most recent vendoring step by resetting some state and restoring the patch stack. Run these commands in a bash shell:
rm .moz-fast-forward/*.resume # resets the resume logic in
hg revert third_party/libwebrtc && hg purge third_party/libwebrtc
./mach python dom/media/webrtc/third_party_build/ --repo-path .moz-fast-forward/moz-libwebrtc


  • Simple changes don't require individual review, but if you'd feel better about getting reviewed, push the vendored commit to elm (which may break all the builds). Then put the patch up for review. After r+, land it using Lando to elm.

Waiting to land

Daily rebase on elm (after completing loop-ff process)

Periodically, ideally daily to minimize surprises from mozilla-central, rebasing elm onto the latest mozilla-central is recommended after completing the update process. Rebases should be completed until the request for merging elm to moz-central is made.

Historical note: we use a script to rebase because 1) it regenerates the files on the commits where those are changed. This avoids the typically massive rebase conflict if anyone has touched the build files. 2) it exports/imports each commit individually to limit rebase conflicts on individual patches, making it easier to deal with issues.

bash dom/media/webrtc/third_party_build/

If one would like to make changes to the patch ordering during the rebase operation (say, for example, a fix-commit was necessary but is separated from the initial upstream patch), use the following command line:

STOP_FOR_REORDER=1 bash dom/media/webrtc/third_party_build/

Requesting merge from elm to mozilla-central

Typically, immediately after the code-freeze has lifted, a request is made for the sheriffs to merge elm to mozilla-central. This done via email, recently to Sebastian Hengst.

Post-merge Steps

  • After the merge from elm to mozilla-central is complete, the moz-libwebrtc github repo should be updated with the new branch of the commit stack for the new release. There is script that will make the proper branch name and do the push:
bash dom/media/webrtc/third_party_build/