Performance/Telemetry Regression Alerts: Difference between revisions

Performance/Telemetry Regression Alerts (view source)

46 bytes removed , 9 July 2014

39

edits

@@ Line 6: / Line 6: @@
 The current prototype uses telemetry.js to fetch the histograms for the build-ids of the past couple of months. The histograms are passed to a python job that for each metric runs a regression algorithm and aggregates the histograms by platform and channel. The Bhattacharyya distance is computed between the histograms of the current build-id and the past N build-ids. If the variance of the distance between the histogram of the current build-id and the histograms of the past N build-ids is small enough and the distance between the histograms of the current build-id and the previous build-id is above a cutoff value K, a regression is reported. Furthermore, Histograms that don't have enough data are filtered out. Cut-off values are determined empirically from the data and past known regressions.
-The Bhattacharyya distance has proven to perform significantly better on our dataset than using the correlation coefficient of the QQ plots, the Mahalanobis distance, a Chi-Square test, a Mann-Whitney test or a one class Support Vector Machine.
+The Bhattacharyya distance has proven to perform significantly better on our dataset than using the Pearson correlation, a Chi-Square test, a Mann-Whitney test or a one class Support Vector Machine.
 ==People==