QA/Mozmill Test Automation/Dashboard/Documentation: Difference between revisions

QA/Mozmill Test Automation/Dashboard/Documentation (view source)

Revision as of 18:43, 1 December 2010

2,195 bytes added , 1 December 2010

no edit summary

Whimboo

canmove, Confirmed users, Bureaucrats and Sysops emeriti

4,747

edits

@@ Line 1: / Line 1: @@
-=Specs=
+With the continuous efforts to raise the popularity of automated ui testing with Mozmill, a central shared storage is needed to collect and visualize results of executed test-runs. The software to implement has to cover all the different types of possible Mozmill test types and should also be flexible enough to support other additions at a later stage. For the visualization of the test results, different views and sub-views will have to be added. Each view will cover a supported report type.
-The formerly specs can be found in the following [https://docs.google.com/document/edit?id=1YIxRwkqa62moVfrAcE3O9QDbU9W6eaow8T7kJOEWyw4# spreadsheet]
-=Data Source=
+The central storage mentioned above will be a CouchDB database located on one of our internal servers. The dashboard itself will be implemented as a Sammy application and is a part of the database itself.
-The data used for queries and visualizations is provided by the the [[QA/Mozmill_Test_Automation/Results_Data|Results Data]] project. Depending on the test-run different information will be send. The [https://spreadsheets.google.com/ccc?key=trPutwwkRQzWCrBtKMxNpUg&hl=en#gid=0 spreadsheet] lists the current list of values for each test-run.
-The following test-runs are supported:
+The target audience for the dashboard is the testing community around Firefox. Those people will run tests to check the basic functionality of Firefox. But also localizers, who want to verify their localized builds, and add-on authors, who want to check the compatibility of their add-ons, are on our focus. Beside that the Mozilla QA team will use a copy of the dashboard to track the results of release testing.
-* General (including non-restart and restart tests of the automated BFT tests)
-* Software Update (tests for the application update client)
-* Add-ons (tests provided by add-on authors)
-* L10n (tests for localizers, i.e. cut-off elements)
-* Accessibility tests (tests for accessibility)
-To correctly identify the results of different test-runs in the database, the [[QA/Mozmill_Test_Automation/Automated_Testruns|automation scripts]] have to be used. We do not support test results sent by Mozmill itself.
+=Objective=
+* Collect test results for all supported test-runs
+* Collect l10n related test results across different localized builds
+* Offer an easy way to analyze results of automated Mozmill tests
+* Improve the collaboration with the community in the automation perspective
+* Offering a free service to support localizers and add-on authors
-=Mockups=
+=Product Components=
-The following image shows a design mockup in an early state. There will be a discussion for requirements in the Q3/10 and we also plan to have version 0.1 ready by September.
-[[File:QA_Automation_Dashboard.png|650px|left|Design Mockup for Web Dashboard]]
+==Collecting test results==
+Whenever a test-run is executed and test results are available and are uploaded to the couchdb database, the collector script of the database has to make sure that the correct data is contained in the report.
+The following tasks listed below have to be run:
+* Check the type of the report. If the type is ‘mozmill’ or ‘mozmill-restart’ change it to ‘firefox-general’. This is needed as a fallback solution if the automation scripts cannot be used for a testrun. If such a transformation would not happen, those results wouldn’t be visible in the dashboard.
+* Please note that we will not update the current report version number, because the collector cannot add or update additionally information. That means that some additional features would not be available for those reports.
+* Update the entry for time_uploaded to the current time. This timestamp is not visible in the ui.
+* Convert the full path of the test file name into a canonical path. Therefore backslashes have to be replaced with slashes first, followed by the removal of all hierarchies above the firefox folder for general tests. For update, add-ons, l10n, and accessibility tests a slight modification is necessary because the root path will be different.
+==Visualization of Results==
+The visualization of test results will be divided into different views. Each view will cover a supported test-run. That allows an easier navigation between results of different test-runs. Each view will be directly accessible via a canonical URL.
+Information for reports of all types of testruns can be found in that [https://spreadsheets.google.com/ccc?key=trPutwwkRQzWCrBtKMxNpUg&hl=en#gid=0 spreadsheet].
+===General Test-run===
+===Update Test-run===
+===Add-ons Test-run===
+===L10n Test-run===
+===Accessibility Test-run===
+=todo=
+* Former specs: [https://docs.google.com/document/edit?id=1YIxRwkqa62moVfrAcE3O9QDbU9W6eaow8T7kJOEWyw4# spreadsheet]
+* [[File:QA_Automation_Dashboard.png|650px|left|Design Mockup for Web Dashboard]]