MozillaWiki

QA/DateTimeInputTypes: Difference between revisions

Page Discussion

QA/DateTimeInputTypes (view source)

Revision as of 15:00, 13 November 2017

4,202 bytes added ,  13 November 2017
-
(-)
 
(35 intermediate revisions by 4 users not shown)
Line 7: Line 7:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 11/23/2015 || 1.0 || Brindusa Tot || Created first draft
| 14/09/2016 || 1.0 || Brindusa Tot || Created first draft
|-
| 13/11/2017 || 1.1 || Emil Ghitta || Updates
|-
|-
| || || ||
|}
|}


Line 40: Line 41:
[mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) - PM for QA team <br />
[mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) - PM for QA team <br />
[mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts <br />
[mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts <br />
[mailto:simona.marcu@softvision.ro Simona Badau] (irc: simonab) - QA <br />
[mailto:roxana.leitan@softvision.ro Roxana Leitan] (irc: RoxanaLeitan) - QA
[mailto:liviu.cirdei@softvision.ro Liviu Cardei] (irc: liviucirdei) - QA <br />
[mailto:roxana.leitan@softvision.ro Roxana Leitan] (irc: RoxanaLeintan) - QA


= Testing summary =  
= Testing summary =  
Line 101: Line 100:
- Cursor + Keyboard<br />
- Cursor + Keyboard<br />
- Keyboard-only<br />
- Keyboard-only<br />
- testing using high Contract themes<br />
- testing using High Contrast themes<br />
- Screen reader ???
- Screen reader
|| Manual || Eng Team
|| Manual || Eng Team
|-
|-
Line 109: Line 108:
== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds
* Links for Nightly builds - available since 10.26.2016 - [http://archive.mozilla.org/pub/firefox/nightly/2016/10/2016-10-26-03-02-10-mozilla-central/ link]
* Links for Aurora builds
* Links for Beta builds
* Links for Beta builds


Line 125: Line 123:
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 2016.09.12 ||  
|style="text-align:center;" | 2016.09.12  
|style="text-align:center;" | 2016.09.15
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 2016.09.12 ||  
|style="text-align:center;" | 2016.09.12  
|style="text-align:center;" | 2016.09.15 
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 2016.09.12 ||  
|style="text-align:center;" | 2016.09.12  
|style="text-align:center;" | 2016.10.01 
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||
|style="text-align:center;" | 2016.11.10
|-
|style="text-align:center;" | 2017.09.13
| QA - Aurora Testing
|style="text-align:center;" | || 
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.09.14
|style="text-align:center;" | 2017.10.27
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.11.14 ||  
|}
|}


Line 154: Line 154:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/169&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
Line 163: Line 163:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
===Time===
  Track the dates and build number where feature was merged to Aurora
{| class="wikitable"
Track the dates and build number where feature was merged to Release/Beta
| <b>Basic input field  </b>||<small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 52: first landed, disabled by default
|-
|<b>Pop-up picker ||<small>'''<font color="green">[LANDED]</font>'''</small> </b>||Nightly 52: first landed, de-scoped
|-
|<b>Input field with STEP (displaying warning message when out of range) </b>||<small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed,disabled by default
|-
|<b>Input field with Min/Max (displaying warning message when out of range) </b>|| <small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed, disabled by default
|-
|<b>Input field with keyboard navigation </b>||  <small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 52: first landed, disabled by default
|-
|}
 
===Date===
{| class="wikitable"
| <b>Basic input field  </b>||<small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed, disabled by default
|-
|<b>Pop-up picker ||<small>'''<font color="green">[LANDED]</font>'''</small> </b>||Nightly 52: first landed, disabled by default
|-
|<b>Input field with STEP (displaying warning message when out of range) </b>||<small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed,disabled by default
|-
|<b>Input field with Min/Max (displaying warning message when out of range) </b>|| <small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed, disabled by default
|-
|<b>Input field with keyboard navigation </b>|| <small>'''<font color="green">[LANDED]</font>'''</small> || Nightly 53: first landed, disabled by default
|-
|}
 
== Risk Assessment and Coverage ==
 
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || User Experience || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-2 || High contrast || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-3 || Localization|| '''3-High''' || ''2-Possible'' || '''3-High'''|| '''18'''
|-
| RAC-4 || Screen reader for picker and input box  || '''3-High''' || ''2-Possible'' || '''3-High''' || '''18'''
|-
| RAC-5 || Input box in multiple frames  || '''3-High''' || ''2-Possible'' || ''2-Medium'' || '''12'''
|-
| RAC-6 || RTL  || 1- Low || ''2-Possible'' ||''2-Medium'' || 4
 
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* Meta Bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 Bug 888320 (datetime) meta Implement all time and date related input types]
* Meta Bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 Bug 888320 (datetime) meta Implement all time and date related input types]
* UX specs bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1069609 Bug 1069609]
* List and links for specs
* List and links for specs
   - UX specifications - [https://mozilla.invisionapp.com/share/237UTNHS8#/screens/171579739 link]
   - UX specifications - [https://mozilla.invisionapp.com/share/237UTNHS8#/screens/171579739 link]
   - Roadmap wiki page - [https://public.etherpad-mozilla.org/p/Date_Time_Input_Types link]
   - Roadmap wiki page - [https://wiki.mozilla.org/TPE_DOM/Date_time_input_types link]
   - demo presentation - [https://docs.google.com/presentation/d/1dfWRYg8tUnll8pXk7b9e78H-H90vKgrvUdoSOYvSeY8/edit#slide=id.p4 link]
   - demo presentation - [https://docs.google.com/presentation/d/1dfWRYg8tUnll8pXk7b9e78H-H90vKgrvUdoSOYvSeY8/edit#slide=id.p4 link]
   - webpage sample - [https://drive.google.com/file/d/0B1FW1kUbRTSuaEZJUlA2eld6cG8/viewn link]
   - webpage sample - [https://drive.google.com/file/d/0B1FW1kUbRTSuaEZJUlA2eld6cG8/viewn link]
Line 182: Line 235:


= Testcases =  
= Testcases =  
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 191: Line 242:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 206: Line 257:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | yes ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | yes || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | yes ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 230: Line 281:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
Line 240: Line 291:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | NA || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
Line 259: Line 310:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 275: Line 326:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 294: Line 345:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 310: Line 361:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 326: Line 377:
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 333: Line 384:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - can be found on TestRails at  [https://testrail.stage.mozaws.net/index.php?/suites/view/169&group_by=cases:section_id&group_order=asc link]
  Full Test suite - can be found on TestRails at  [https://testrail.stage.mozaws.net/index.php?/suites/view/169&group_by=cases:section_id&group_order=asc link] or in the [https://wiki.mozilla.org/File:Date_Time_Input_Types_-EngQA-_-_TestRail.pdf PDF doc]
Regression Test suite - if available/needed.
 


= Bug Work =
= Bug Work =
Tracking bug – [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 meta bug 888320]
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 Bug 888320 - [meta<nowiki>]</nowiki> (datetime) implement all time and date related input types]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 888320 ] ) ==
 
<bugzilla>
====== Bug fix verification ======
    {
<div class="mw-collapsible-content">
        "blocks":[888320],
{| class="wikitable" style="width:50%"
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
|-
    }
! Bug No !! Summary !! Status !! Firefox Verion
</bugzilla>
|-
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1323674 Bug 1323674 -[meta<nowiki>]</nowiki> defects in date time input types]
== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1323674 1323674 ] ) ==
<bugzilla>
    {
        "blocks":[1323674],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>


====== Logged bugs ======
== Bug Verification  (blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=888320 888320 ]) ==
<div class="mw-collapsible-content">
<bugzilla>
Bug 111111<br/>
    {
Bug 211111<br/>
        "blocks":[888320],
</div>
        "resolution":"FIXED",
</div>
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
== Bug Verification  (blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1323674 1323674 ]) ==
<bugzilla>
    {
        "blocks":[1323674],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>


= Sign off =
= Sign off =
Line 376: Line 438:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
== Results ==
'''Nightly testing'''<br />
'''Nightly testing'''<br />
''Time Input Box''


List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*[https://testrail.stage.mozaws.net/index.php?/plans/view/1400 Link for the tests run]
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
 
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
''Date Input Box and Date Picker''
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*[https://testrail.stage.mozaws.net/index.php?/plans/view/2295 Link for the tests run]
** Full Test suite
 
'''Beta Testing'''
*[https://testrail.stage.mozaws.net/index.php?/reports/view/664 Link for the tests run]


== Checklist ==
== Checklist ==
Line 399: Line 462:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | 09.12.2016
| style="text-align:center;" | 09.14.2016
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" |   
| style="text-align:center;" |   
|-
|  Time Input Box Functional Tests Execution
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |
|-
|  Date Input Box/Picker Functional Tests Execution
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |
|-
|-
|  Automation Coverage ||
|  Automation Coverage ||
Line 421: Line 494:
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified
|style="text-align:center;"| {{mdone|}}
| style="text-align:center;" |  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry||  
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" | Email sent on 2017.09.13
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | n/a
|style="text-align:center;" | n/a
|-
|-
|  QA Signoff  - Aurora Release ||
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent
|style="text-align:center;"| n/a
|-
|style="text-align:center;"| n/a
|  QA Beta - Full Testing
|style="text-align:center;" | ||
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" | Email sent on 2017.10.17
|}
|}
Eghitta
54

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1147547...1183983"