Automated Test Coverage

Project Description
Enter text here
Requirements:
Enter text here
Automated Test Coverage
| Milestone | Due Date | Status | Notes |
|---|---|---|---|
| Convert integration tests from GIP to GIJ | 12/31/15 | AT RISK | All but 14 have been converted. Engineers out on PTO. |
| Intermittently failing tests fixed | 01/31/15 | CHALLENGED | Engineers are disabling the tests that fail intermittently and will fix them before re-enabling. Need more engineering help with this. |
| Tests that work on mulet work on device | 01/31/15 | CHALLENGED | It's being reported that some of the flagged tests actually work on device. Need to reassess all. |
| Test Matrix created that defines platform(s) the tests are going to run on | 12/31/15 | AT RISK | Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January. |
| Automation of tests according to Test Matrix | 05/31/16 | ON TARGET | Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January. |
| Every single commit in gecko/gaia triggers automated tests | 05/31/16 | ON TARGET | There's still work that needs to be identified and scheduled. |
| MozTrap scenarios automated using GIJ when feasible | 05/31/16 | ON TARGET | |
| Working Mulet tests actively maintained with reporting on Treeherder | 05/31/16 | ON TARGET | |
| Code Coverage Tool developed | 12/31/15 | AT RISK | Originally targeted for end of Q4, but assistance at Mozlando didn't happen. |
Status Key
| Color | Status | Key |
|---|---|---|
| On Target | The project or deliverable is expected to meet its due date. | |
| Challenged | The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track. | |
| At Risk or Late | The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late. | |
| Done | The project or deliverable has been completed. | |
| On Hold | The project or deliverable has been placed on hold. |
Program Timeline
Project Scope
Bugs for GIP to GIJ conversions
| ID | Summary | Status | Target milestone | Resolution | Assigned to | Depends on | Blocks | Whiteboard | Feature-b2g |
|---|---|---|---|---|---|---|---|---|---|
| 1218401 | [META] Moving test scripts from GIP to GIJ | RESOLVED | --- | WONTFIX | 1219620, 1219627, 1219628, 1219629, 1219630, 1219631, 1219632, 1219633, 1219634, 1219635, 1219636, 1219640, 1219641, 1219643, 1219644, 1219645, 1219649, 1219650, 1219651, 1219653, 1219654, 1219655, 1219656, 1219657, 1219658, 1219659, 1219660, 1219661, 1219662, 1219663, 1219665, 1219666, 1219667, 1219668, 1219670, 1219671, 1219674, 1219675, 1219676, 1219677, 1219678, 1219679, 1219680, 1219681, 1219682, 1219683, 1219684, 1219685, 1219686, 1219687, 1219688, 1219689, 1219690, 1219691, 1219692, 1219693, 1219694, 1219695, 1219696, 1219697, 1219698, 1219699, 1219700, 1219701, 1219702, 1219703, 1219704, 1219705, 1219706, 1222975, 1232280, 1232281 | [gip-to-gij] | No feature-b2g | ||
| 1219620 | Implement *test_browser_bookmark.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | ||
| 1219627 | Implement *test_browser_play_video.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219628 | Implement *test_browser_save_image.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219629 | Implement *test_browser_clear_browsing_history.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Guillaume Marty [:gmarty] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219630 | Implement *test_browser_search.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219631 | Implement *test_browser_share_link.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219632 | Implement *test_browser_navigation.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219633 | Implement *test_calendar_flick_through_months.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | Justin Potts [:justinpotts] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219634 | Implement *test_cards_view_with_two_apps.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219635 | Implement *test_cards_view_kill_apps_with_two_apps.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Etienne Segonzac (:etienne) | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219636 | Implement *test_import_contacts_from_sdcard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219640 | Implement *test_default_contact_image.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219641 | Implement *test_delete_all_contacts.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401, 1219643, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219643 | Implement *test_delete_contact.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1219641 | 1218401 | [gip-to-gij] | No feature-b2g |
| 1219644 | Implement *test_sort_contacts.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219645 | Implement *test_export_contacts_to_sdcard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219649 | Implement *test_cost_control_reset_wifi.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219650 | Implement *test_call_log_groups.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1232680 | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219651 | Implement *test_delete_call_log.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1232680 | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219653 | Implement *test_dialer_add_contact.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Isabel Rios[:isabel_rios] | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219654 | Implement *test_dialer_find_contact.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219655 | Implement *test_ftu_status_bar.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Sam Foster [:sfoster] (he/him) | 1218401, 1228972 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219656 | Implement *test_gallery_cancel_delete_multiple_images.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Punam Dahiya [:pdahiya] | 1219657 | 1213991, 1218401 | [gip-to-gij] | No feature-b2g |
| 1219657 | Implement *test_gallery_delete_multiple_images.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1219656, 1223040 | [gip-to-gij] | No feature-b2g | |
| 1219658 | Implement *test_gallery_empty.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219659 | Implement *test_gallery_handle_invalid_file.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219660 | Implement *test_gallery_handle_valid_image_file.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219661 | Implement *test_gallery_share_picture.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219662 | Implement *test_gallery_view.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
| 1219663 | Implement *test_homescreen_change_wallpaper.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219665 | Implement *test_homescreen_column_layout.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Chris Lord [:cwiiis] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219666 | Implement *test_homescreen_delete_app_packaged.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Chris Lord [:cwiiis] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219667 | Implement *test_homescreen_divider.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219668 | Implement *test_homescreen_launch_app.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219670 | Implement *test_homescreen_launch_app_packaged.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219671 | Implement *test_homescreen_move_app.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219674 | Implement *test_email_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219675 | Implement *test_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
| 1219676 | Implement *test_keyboard_bug_1073870.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219677 | Implement *test_keyboard_predictive_key.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219678 | Implement *test_number_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219679 | Implement *test_lockscreen_time_check.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219680 | Implement *test_lockscreen_unlock_to_homescreen.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
| 1219681 | Implement *test_lockscreen_unlock_to_homescreen_with_passcode.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219682 | Implement *test_lockscreen_wake_with_notification.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219683 | Implement *test_music_{album,artist,song}_mp3.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Hubert Figuiere [:hub] | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | |
| 1219684 | Implement *test_music_artist_mp3.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219685 | Implement *test_music_change_rating.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219686 | Implement *test_music_share_ringtone.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219687 | Implement *test_music_songs_3gp.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | ||
| 1219688 | Implement *test_settings_change_keyboard_language.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219689 | Implement *test_settings_change_language_rtl.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219690 | Implement *test_settings_change_ring_tone.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219691 | Implement *test_settings_gps.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219692 | Implement *test_settings_passcode.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219693 | Implement *test_settings_usb_storage.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
| 1219694 | Implement *test_inter_app_comm.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219695 | Implement *test_privileged_app_contacts_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1014410, 1227011 | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g |
| 1219696 | Implement *test_privileged_app_device_music_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219697 | Implement *test_privileged_app_device_picture_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219698 | Implement *test_privileged_app_device_sdcard_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219699 | Implement *test_privileged_app_device_video_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219700 | Implement *test_privileged_app_geolocation_prompt.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | ||
| 1219701 | Implement *test_quick_settings.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1228972 | [gip-to-gij][systemsfe] | No feature-b2g | |
| 1219702 | Implement *test_system_message.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219703 | Implement *test_system_message_pending.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
| 1219704 | Implement *test_play_ogg_video.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
| 1219705 | Implement *test_play_ogv_video.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
| 1219706 | Implement *test_play_webm_video.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
| 1222975 | Delete the Python tests that are already running in Gij and not testing more than the integration between Gecko and Gaia | RESOLVED | --- | WONTFIX | 1223012, 1223055, 1228972, 1230099, 1233453 | 1218401 | [gip-to-gij] | No feature-b2g | |
| 1232280 | Implement test_home_button.py and test_home_button_in_edit_mode.py as an Integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Guillaume Marty [:gmarty] | 1195364 | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g |
| 1232281 | Implement test_home_button.py (in Lockscreen) as an Integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1195364 | 1218401, 1233453 | [gip-to-gij] | No feature-b2g |
73 Total; 0 Open (0%); 73 Resolved (100%); 0 Verified (0%);
Bugs for Tests that Run on Mulet but Fail on Device
10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);
Bugs for Tests that Fail Intermittently
8 Total; 0 Open (0%); 8 Resolved (100%); 0 Verified (0%);
Bugs for Code Coverage Tool
| ID | Summary | Status | Target milestone | Resolution | Assigned to | Depends on | Blocks | Whiteboard | Feature-b2g |
|---|---|---|---|---|---|---|---|---|---|
| 1176880 | Add Code Coverage interface to the Debugger. | RESOLVED | mozilla43 | FIXED | 1322663, 1190454, 1205880, 1226896, 1252111 | 1189360, 1179996, 1200154 | [CC] [CI] | No feature-b2g | |
| 1179996 | [fxos-automation] Code coverage tools for JavaScript tests | RESOLVED | --- | INVALID | Aus Lacroix [:aus] | 1176880, 1179997, 1179999, 1180003, 1180006, 1180007 | [CC] [CI] | No feature-b2g | |
| 1179997 | [fxos-automation][aus-q3] Turn on JavaScript engine code coverage support for B2G-Desktop builds (Linux x64 Opt Only) | RESOLVED | --- | INVALID | 1179996 | [CC] [CI] | No feature-b2g | ||
| 1179999 | [fxos-automation][aus-q3] Add code coverage log parsing support to Unit/Integration test tooling | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
| 1180003 | [fxos-automation][aus-q3] Add code coverage regression check | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
| 1180006 | [fxos-automation][aus-q3] Turn on Code Coverage for Gaia Unit Tests | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
| 1180007 | [fxos-automation][aus-q3] Turn on Code Coverage for Gaia Integration Tests | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g |
7 Total; 0 Open (0%); 7 Resolved (100%); 0 Verified (0%);
Dependency Tracking
Detailed Program Plan
| Action Item | Engineering Owner | QA Owner | UX Owner | Bugzilla ID | Planned Done | Actual Done |
|---|---|---|---|---|---|---|
Program Stakeholders
| Role | Name | IRC |
|---|---|---|
| EPM | ||
| EM | ||
| PM | ||
| TL | ||
| UX | ||
| QA |
- EPM = Engineering Program Manager
- EM = Engineering Manager
- PM = Product Manager
- TL = Tech Lead
- UX = User Experience
- QA = Quality Assurance