Media/WebRTC/WebRTC Debugging

Reporting WebRTC Call Issues

The best way to report an issue is through Bugzilla using this link. Describing the issue you've run into, and include a URL along, with the details of the call setup.

For simple issues, the first place to look is to check the web developer console for error messages related to media format issues. If you see messages here related to WebRTC, getUserMedia, or getDisplayMedia, please add this information to your bug.

Share Your about:support Text

In your Bugzilla report, please include support information about the current device you are seeing an issue on -

1. Open a tab and visit about:support
   
2. Click 'Copy Text to Clipboard'
   
3. Paste this text in your Bugzilla bug comment and post.
   

Share Your about:webrtc Contents

For issues with call quality, please share web conferencing related performance information by providing your about:webrtc information. Note this information should be collected while the call in question is still active.

1. While your call is still ongoing, open an additional tab and visit about:webrtc.
2. Click "Clear History" to clear the stats from other recent calls which are no longer ongoing.
3. At the bottom of the page click 'Save Page', and save this file.
4. Add this file as an attachment to your bug.

This data contains statistics about your call, the signalling that was used to setup your call, and information about the network transports.

Diagnosing Call Quality Issues

About Webrtc Overview

This section will contain screen shots of about:webrtc and description of each of the sections. This information needs to be referenced from multiple places later in the wikipage.

TODO

Audio/Video Delay

Delayed media is commonly caused by long physical paths between endpoints, though anything that slows down inter-hop delivery of packets can be at fault. Note that this is different than the bandwidth of the network path, and a high latency will not be fixed by reducing the video resolution or audio sample rate. Round trip time, or RTT, is the time it takes for a packet to get from the sender to the receiver and then for a packet to get from the receiver back to the sender. If the path is symmetric between the two endpoints one can assume that the one way delay is half the delay of the round trip.

The second common cause of A/V delay is jitter, the magnitude of variability in packet inter-arrival times. In order to smoothly play out of the incoming stream a receiver experiencing jitter will have to buffer (delay) incoming packets.

Using [about:webrtc] to Diagnose Delay

The key metrics in [about:webrtc] are RTT (round-trip-time) and jitter. They can be found in the RTP stats section of the PeerConnection. The PeerConnection informational blocks start out in a minimized state, and one will need to expand a block to find the RTP stats section:

Once expanded one can locate the RTP Stats section and find the key diagnostic stats:

In this image we can see that there are 0 milliseconds of jitter, and 32 milliseconds of round trip delay. This call should not be experiencing any noticeable delay. For all intents and purposes jitter and RTT are additive in nature. If there was 25ms of jitter reported and a RTT of 272ms, one could estimate the expected delay from transmission at the send side to play out on receive side to be 25ms + (272ms / 2) = 161ms If the perceived delay is larger than the estimated delay that could indicate a problem within Firefox that requires debugging. Under these circumstances it would be helpful to grab a JSON copy of the current stats by pressing the "Copy Report" button, pasting those stats into your Bugzilla bug report.

Performance Profiling and Logging

Capturing a Firefox Performance Profile

For basic performance issues, a performance profile can help engineers diagnose issues with video formats, performance, and rendering.

1. Visit https://profiler.firefox.com/ and enable the Profiler toolbar button.
2. Click the toolbar button down arrow and select 'Media' in the Settings drop down.
3. Open a tab and visit the page with the affected media content.
4. Click the Profiler toolbar main button to start recording.
5. Play media until the issue you are seeing manifests.
6. Click the Profiler toolbar button again to stop recording.
7. When a new Profile tab opens, click the upload profile button on the upper right.
8. Copy the resulting profile URL and post this to your Bugzilla report.

Additionally, detailed logging can be collected within performance profiles to help aid in debugging complicated issues. To enable the collection of a profile with low level debugging -

1. Visit https://profiler.firefox.com/ and enable the Profiler toolbar button.
2. In a new tab, visit about:webrtc. Click the 'Enable WebRTC Log Preset' button, which will open a tab for about:logging with prepopulated information.
3. In about:logging, click the "Start Logging" button. (You are now recording a profile, the profiler toolbar toggle button should be selected automatically.)
4. Open a new tab for testing and view the media you are having an issue with. (After reproducing, DO NOT close this test tab yet.)
5. Switch to the about:logging tab, click 'Stop logging', and then close the test tab.
6. Wait approximately 10 - 20 seconds for a new tab to automatically open containing the generated performance profile.
7. Within the upper-right side of the profiler tab click the 'upload local profile' button to initiate profile upload. Once the upload is complete, a drop down text edit will open displaying the URL of the profile. Select this text and copy it.
8. Share the URL of the profile for analysis with the engineer you are working with.

MOZ_LOG="jsep:5,sdp:5,signaling:5,mtransport:5,RTCRtpReceiver:5,RTCRtpSender:5,RTCDMTFSender:5,VideoFrameConverter:5,WebrtcTCPSocket:5,CamerasChild:5,CamerasParent:5,VideoEngine:5,ShmemPool:5,TabShare:5,MediaChild:5,MediaParent:5,MediaManager:5,MediaTrackGraph:5,cubeb:5,MediaStream:5,MediaStreamTrack:5,DriftCompensator:5,ForwardInputTrack:5,MediaRecorder:5,MediaEncoder:5,TrackEncoder:5,VP8TrackEncoder:5,Muxer:5,GetUserMedia:5,MediaPipeline:5,PeerConnectionImpl:5,WebAudioAPI:5,webrtc_trace:5,RTCRtpTransceiver:5,ForwardedInputTrack:5,HTMLMediaElement:5,HTMLMediaElementEvents:5"

Standard Logging Modules

Non-standard Logging Modules

Examining Call Performance Issues

Enabling Call Stats History

Call stats history is enabled by default in Nightly. To enable in release builds open [about:config], and change "media.aboutwebrtc.hist.enabled" to true. This will keep a history windows of stats for a number of recent calls, allowing for inspection in [about:webrtc] after a call has completed.

Dumping Call Stats

One can dump a JSON blob of call stats for an active call, or a recent call if call stats history is enabled. There are two buttons in [about:webrtc] to do this, "Copy Report" and "Copy Report History". The former will create a copy of the most recent stats for the PeerConnection. The later will copy all the history of stats reports that [about:webrtc] has accumulated for that PeerConnection, this can be up to several minutes of stats.

Debugging Encrypted Packets

TODO

Running WebRTC Tests

Mochitests

./mach try fuzzy --query 'mochitest-media'

Web Platform Tests

Web Platform Tests can be run locally from mach.

./mach wpt testing/web-platform/tests/webrtc

./mach try fuzzy --query 'wpt'

One can run those same tests in Chromium if one needs to compare behavior between browsers.

Debugging Using 3rd Party Websites

Using RR And/Or Pernosco