Media/WebRTC/libwebrtc Update Process/automation plan

From MozillaWiki
Jump to: navigation, search

This describes the rough plan to document and automate the process to "fast-forward" through libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.

Plan Overview

  • repo tasks
- update moz-libwebrtc fork to latest
git checkout master
git remote add upstream
git fetch upstream
git merge upstream/master # fast forward merge
git push origin master
- update elm to latest moz-central
- check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86
- check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86
- check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86
- move one changeset forward in history
- verify local build in moz-libwebrtc
- apply moz-libwebrtc-to-upstream patches, verify local build
- vendor moz-libwebrtc
- apply moz-central patches
- full try run
  • write automation scripts "Manually identify how to fast-forward one changeset"
  • investigate updatebot


current third_party/libwebrtc directories and status

  • Additional vendoring
build (with some changes during vendoring)
  • Mozilla added
README.mozilla - generated during vendoring process - generated by
webrtc_gn - generated by
X11 - imported later by Byron in c91f12b557a1 D113830
tools/grit - imported later by Nico in 3cce5e6938f0 D114027
  • Mozilla vendored after a93c2d1df066
- Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.
sdk/ - imported later by Byron in 9314046d89eb D105015
sdk/objc - imported later by Byron in 9314046d89eb D105015
- We don't build the sdk directory (see here).
- Only sdk:helpers_objc is used (see here)
- From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:
  • Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc   (should have been vendored)
DEPS        (should have been vendored)
google_apis (is this referenced/used by current code?)
tools/clang (is this referenced/used by current code?)            

verify generation/build with extra, unused directories

  • Add missing directories
cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media
  • local build file generation produces no build modified build files
  • Full build results here using ./mach try fuzzy -q 'build-'
  • Full linux test results here using ./mach try fuzzy -q 'linux-'

verify current vendoring scripts cmd-line params

  • Vendor libwebrtc (a93c2d1df066)
- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit:
cd dom/media/webrtc/third_party_build
python3 \
        --from-github \
        --commit 149d693483e9055f574d9d65b01fe75a186b654b \
  • Vendor libwebrtc/third_party (d1df4970f4f0)
cd dom/media/webrtc/third_party_build
python3 \
        --from-googlesource \
        --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \
  • Vendor libwebrtc/build (587c40771588)
- The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit:
cd dom/media/webrtc/third_party_build
python3 \
        --from-googlesource \
        --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \

build instructions for moz-libwebrtc

Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.

  • for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:
sudo apt update
sudo apt install python2
curl --output
- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
git clone
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  export DEPOT_TOOLS_UPDATE=0 ; \
mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc # populates .gclient and .gclient_entries
  # on fresh Ubuntu install, do:
(cd src && git checkout edd7f9e94d)
gclient sync
cd src
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test
  • verify build for moz-libwebrtc
- Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.
git clone
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  export DEPOT_TOOLS_UPDATE=0 ; \
mkdir moz-libwebrtc-checkout
cd moz-libwebrtc-checkout
gclient config --name src --unmanaged
gclient sync -D --force --reset --with_branch_heads # ~30 min
cd src
git checkout edd7f9e94d
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

remove dead/unused files

  • Successfully removed:
- successful try builds
- successful try builds

patches after vendoring moz-libwebrtc into moz-central

hg diff -r 587c40771588 -r central \
   --exclude '**/' \
   --exclude 'third_party/libwebrtc/X11' \
   --exclude 'third_party/libwebrtc/sdk/' \
   --exclude 'third_party/libwebrtc/sdk/objc' \
   --exclude 'third_party/libwebrtc/tools/grit' \
   --exclude 'third_party/libwebrtc/build' \
   --exclude 'third_party/libwebrtc/third_party' \
- While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the changesets. It makes more sense to reuse the script to build a patch set of changes we can apply on top of our current patch stack residing in moz-libwebrtc. That command looks like:
cd dom/media/webrtc/third_party_build && \
python3 587c40771588::2b187d932ca1
- where 587c40771588 is the moz-central vendoring commit and 2b187d932ca1 is the current tip of moz-central
- extract-for-git produces a file named mailbox.patch which can be applied in moz-libwebrtc with:
git am mailbox.patch
- Note, fixes were required during the git am process. These were dealing with a couple places were we imported directories into moz-central from after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.

patches on top of upstream github moz-libwebrtc commit

  • easiest way
git diff -r edd7f9e94d -r mozilla-modifications-rel86
  • more useful for rebasing on top of upstream commits
git branch mozpatches edd7f9e94d
git checkout mozpatches
git format-patch -k edd7f9e94d..mozilla-modifications-rel86
git am *.patch # applies to branch mozpatches
  • in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central

curl -o
(cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)

Keeping transitive dependencies updated

  • we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
  • libyuv
  • libvpx
  • pipewire

Open Questions

  • How to see release branches in
 Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.
  • How to find versions of that ship with Chrome?
 If you have acquired using gclient sync (instructions), there are branches listed that look like 'branch-heads/nnnn'.  You can cross-reference the Chrome release and find the release here:
 example: Chrome's 98 release uses 4758, which is sha a6b138d6b4
  • Do we want to advance commit-by-commit?
 I think we may need to do this to avoid large-scale breakage and
 make it easier to fix issues.
  • Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?
 This allows a more finely tuned process to help us get caught up,
 but pushes out the integration with update-bot.
  • How/when do we start integrating's unittests into our try runs?
 This may need to happen in our tree (probably post-vendoring) because
 the process of getting the webrtc tree buildable is a long process
 requiring lots of network.
  • Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?

Operation Checklist


- Create a new bug for the fast-forward update. It should depend on the previous fast-forward bug. Bug 1800920 is an example.

- Builds are done after vendoring in each commit. Having a .mozconfig file that enables cache (either sccache or ccache) will improve the performance of the scripts.

hg clone
(cd elm && ./mach bootstrap --application-choice=browser && ./mach build)
git clone moz-libwebrtc

- On macOS, the python module requests should be installed.

pip install requests

Update steps

# Start from the elm checkout
cd elm

# Prepare the github repo
bash dom/media/webrtc/third_party_build/

# Run
bash dom/media/webrtc/third_party_build/

Operational notes

- Running with a fresh github clone of moz-libwebrtc will likely display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in github. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.

- Two main types of errors will cause to exit; the first is a rebase conflict, the second is a build error due to api changes in upstream code.

- When making changes to fix build issues in moz-libwebrtc, mercurial commit messages follow the pattern:

Bug xxx - (fix-{upstream-sha}) {description}

- When making changes to fix Mozilla code in response to changes upstream, mercurial commit messages follow the pattern:

Bug xxx (MOZ) - {description}

- While is running, in a separate terminal window it may be helpful to see the overall progress by running:

tail -f ~/log-loop-ff.txt | grep loop-ff

Moving third_party/libwebrtc/third_party/* to third_party/*


(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.

  • Reduce the number of libwebrtc patches, with an emphasis on build patches
  • Reduce the merge process complexity
  • Reduce our response time to security issues
  • Uplifting patches

Additionally, resolving these issues would assist with broader product goals.

  • Improve performance
  • Reduce build time

Patch Reduction

We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our build_with_mozilla flag. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.

Merge Complexity

Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.


We will need to employ several strategies in order to create a robust build environment.

  1. We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
  2. We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.

As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.


  1. (lib)webrtc dependency pinning
  2. Chromium Dependencies
  3. gclient setup
  4. Nika's bazel hack
  5. (lib)drm
  6. libgmb
  7. abseil-cpp
  8. libyuv
  9. libvpx
  10. libdav1d
  11. libaom
  12. Linux shared library intricacies
  13. dlopen
  14. Ubuntu 18.04 LTS support
  15. RedHat modifications
  16. Vendoring headers
  17. Finding versions of loaded libraries

Resetting Elm

Before each new merge cycle. Elm needs to be reset to Central, and the floating patches need to be reapplied.

  1. Ensure you have filed a bug for the next merge cycle, and further ensure that the next merge cycle bug has the previous merge cycle bug as a dependency.
  2. Make sure you have a local checkout of Elm which includes the floating patches.
  3. File a bug as a clone of Bug 1789675 and add your new bug as dependency to the next merge cycle bug.
  4. Wait for the repo admins to reset the repository.
  5. Mark the floating arc config patch with a bookmark. If the patch had id `5e8481a558fb` then one would run
    hg bookmark -r 5e8481a558fb ARC_CONFIG_FLOAT
  6. Pull the new head of Elm.
    hg pull
  7. Rebase the floating patches onto the new checkout of Elm, then checkout the new tip.
    hg rebase --keep -s ARC_CONFIG_FLOAT -d elm && \
    hg co tip && \
    hg bookmark ARC_CONFIG_FLOAT
  8. Push the floating patches to Elm.
    hg push --bookmark=ARC_CONFIG_FLOAT
  9. Celebrate