Performance/Telemetry Regression Alerts: Difference between revisions

Performance/Telemetry Regression Alerts (view source)

2 bytes added , 10 July 2014

39

edits

@@ Line 6: / Line 6: @@
 The current prototype uses telemetry.js to fetch the histograms for the build-ids of the past couple of months. The histograms are passed to a python job that for each metric runs a regression algorithm and aggregates the histograms by platform and channel. The Bhattacharyya distance is computed between the histograms of the current build-id and the past N build-ids. If the variance of the distance between the histogram of the current build-id and the histograms of the past N build-ids is small enough and the distance between the histograms of the current build-id and the previous build-id is above a cutoff value K, a regression is reported. Furthermore, Histograms that don't have enough data are filtered out. Cut-off values are determined empirically from the data and past known regressions.
-The Bhattacharyya distance has proven to perform significantly better (in terms of false positives) on our dataset than using the Pearson correlation, a Chi-Square test, a Mann-Whitney test, a Kolmogorov-Smirnov test of the estimated densities or a one class Support Vector Machine.
+The Bhattacharyya distance has proven to perform significantly better (in terms of false positives) on our dataset than using a correlation coefficient, a Chi-Square test, a Mann-Whitney test, a Kolmogorov-Smirnov test of the estimated densities or a one class Support Vector Machine.
 ==People==