Media/WebRTC Audio Perf

Note: A lot of things described in here is still Work in Progress and is expected to change over time.

Introduction

This efforts aims at building tools and frameworks for analyzing audio and video quality for the Firefox WebRTC implementation. WebRTC involves peer-to-peer rich multimedia communications across variety of end-user devices and network conditions. The framework must be able to provide necessary means to simulate/emulate these usage configurations in order to analyze the behavior of WebRTC implementation in the firefox.

Background

Typical WebRTC Media Pipeline

Below picture captures various components involved in the flow of media captured from mic/camera till it gets transported. The reverse direction follows a similar path back till the RTP packets gets delivered as raw media for rendering.

With several moving components in the pipeline, it becomes necessary to analyze the impact these might have on the overall quality of the media being transmitted or rendered. For instance, the parts of the pipeline highlighted ( marked star) has potential to induce latency and impact quality of the encoded media. Thus, having possibilities to measure, analyze and account these impact has potential to improve the performance of the Firefox WebRTC implementation.

Not to forget, the pipeline doesn't capture impacts of latency induced due to network bandwidth, latency and congestion scenarios.

Scope

Following is wishlist of functionalities that the framework must be able to support eventually.

1. Audio and Video Quality Analysis
   1. Quantitative measurements
   2. Qualitative measurements
2. Latency Impact Analysis
   1. End to End Latency
   2. Latency impacts due to local processing
   3. Latency impacts under simulated constrained network conditions
3. RTCP based analysis
4. Codec Configuration Variability Analysis
5. Hardware and Platform Variability Analysis

Audio Performance Framework.

bug 901831 is ongoing effort to provide minimal components that serves as a good starting point for carrying out Peer Connection audio quality analysis as part of Mozilla's Talos framework

Below picture captures the test setup for reporting Perceptual Evaluation of Speech Quality (PESQ) scores for audio played through Peer Connection.

The idea here is to compute PESQ scores between an input audio file fed into the local Peer Connection and output audio audio file recorded at the play-out of the remote Peer Connection in a fully automated fashion. Following sub-sections explain in details the various components for this setup.

Talos

<a_fcknotitle="true" href="https://wiki.mozilla.org/Buildbot/Talos"> Talos </a>

Open Issues

Analysis of tools and techniques for measuring WebRTC Audio Performance. This page is under construction. please come back later for more complete information

Background

Chrome Audio Perf

Following are the specific tests in Chrome that attempt to measure audio performance.

Audio Processing Per 10ms Analysis

This test instruments WebRTC's AudioProcessing Module under various configurations to measure mic to render audio processing for a 10ms audio frames. APM can be configured on the following

Sample Rate, Input and Output Channels, Reverse Channels, Echo Cancellation,   Gain Control, Noise Suppression, Voice Activity Detection,  Level Metrics, Delay, Drift compensation, Echo Metrics

Logic:

  For every Input AudioFrame
    time ProcessStream()
    also apply component configuration
  For every Output AudioFrame
     time AnalyzeReverseStream()

Calculate Execution Time as average for all the 10ms frames processed and analyzed.
 

Audio Quality Voice Engine - E2E Code:run_audio_test.py

 third_party/webrtc/tools/e2e_quality

This uses PulseAudio to setup virtual devices followed by comparison tool to measure the quality. This is based on VoiceEngine loopback call

WebRTC Recording Time Code: webrtc_audio_device_unittest.cc This uses VoEMediaProcess::Process() callback to act as interceptor to audio frames at the recording path to time the recoding setup time

WebRTC Playout Setup Time This uses VoEMediaProcess::Process() callback to act as interceptor to audio frames before playback to time the setup time

WebRTC Loopback With Signal Processing WebRTC Loopback Without Signal Processing Both the tests uses loopback call with/without APM enabled. this loopback runs for 100 AudioFrames.

Proposal