QA/Firefox3.5/Video And Audio TestPlan: Difference between revisions

From MozillaWiki
< QA‎ | Firefox3.5
Jump to navigation Jump to search
No edit summary
 
(20 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<small>[[QA/Firefox3.1/TestPlan|&laquo; QA/Firefox3.1/TestPlan]]</small>
<small>[[QA/Firefox3.1/TestPlan|&laquo; QA/Firefox3.1/TestPlan]]</small>


= NOTES: Things to help us once we start writing =
;Video and Audio Test Plan
* Something to include in Litmus/Gristmill: {{bug|448630}}
 
;<Video> and <Audio> Test Plan


= Overview =
= Overview =
Video and audio HTML 5 suport has landed on trunk for the 3.1 release.  It implements the [http://www.whatwg.org/specs/web-apps/current-work/#video WHATWG specification] for these tags.  So, far, only the ogg-theora codec support has landed.  However, the tests below must be tested with each codec supported unless otherwise noted.
Video and audio HTML 5 suport has landed on trunk for the 3.1 release.  It implements the [http://www.whatwg.org/specs/web-apps/current-work/#video WHATWG specification] for these tags.  So, far, only the ogg-theora codec support has landed.  However, the tests below must be tested with each codec supported unless otherwise noted.
Likewise, everywhere that "video" is used, it is also assumed that audio will be tested for the same test, except in those cases where it doesn't make sense for this to occur.


= Test Strategy =
= Test Strategy =
Line 13: Line 12:
* What must be done to ensure good coverage of the feature
* What must be done to ensure good coverage of the feature
* What should be done to test around the area of the feature and its interoperability with other HTML content
* What should be done to test around the area of the feature and its interoperability with other HTML content
* What might be done in a wonderful perfect world i.e. less important stuff that might get timeboxed)
* What might be done in a wonderful perfect world i.e. less important stuff that might get timeboxed
 
;Test Legend
For Test Type, we use:
* L - Litmus
* G - Gristmill
* M - Mochitest
* R - Reftest
* ? - TBD
 
For Test Status, we use:
* ? - TBD (also indicates that this test idea might be complete foolishness - has not been sized yet at all)
* D - Done - test written and checked in. If possible, "D" will be a link to the test in MXR or Litmus
* I - In Progress - test written, but not checked in (awaiting review, for instance).  The "I" should be a link to the bug where it is awaiting review.
* ' '  (blank) - Indicates that the test has not been written, but it was decided that this is a test we '''will''' be writing, i.e. it is not complete foolishness.


== What MUST be Tested ==
== Things that Must be Done ==
=== Functional Tests ===
{| class="standard-table"
{| class="standard-table"
|-
|-
| class="header"|Test Idea
| class="header"|Test Idea
| class="header"|Test Type
| class="header"|Status
|-
|-
| Analyze the unit tests to ensure they test all default values of the attributes
| Analyze the unit tests to ensure they test all default values of the attributes
| ?
| ?
|-
|-
| Test control toggle to determine if the controls appear and that the controls appear properly
| A(M)
| Done, test_controls.html
|-
| Test autoplay to ensure proper default and can be set to proper values
| A(M)
| Done, has bug (test_autoplay.html)
|-
| Change the controls to something of our own creation
| ?
| ?
|-
| Use Video to animate objects and place as a background to content
| ?
| ?
|-
| test dragging video to a new window and that it doesn't stop playing (via tab drag)
| ?
| ?
|-
| Test that it continues to play (or doesn't play - current behavior is to continue) when you switch to a new tab.
| ?
| ?
|-
| Test accessibility of the video/audio controls
| ?
| ?
|-
| Test that the controls work properly (litmus? gristmill?)
| ?
| ?
|-
| Test automatically removing controls from javascript - would this be a bug?
| A(M)
| Done(test_bug448534.html)
|-
| Use javascript events to override the volume control set by the user (should not be able to do this, I think, considering how much we hate the ads on tv that do this)
| ?
| ?
|-
| Test that playback of both formats stops on closing the tab
| ?
| ?
|-
| Test that reload of page causes both to restart
| ?
| ?
|-
| Test that back/forward work as expected (video back to beginning)
| ?
| ?
|-
| Test that Session restore should restore the original state of the video, not the mid-point where it was when you crashed. (I think).
| ?
| ?
|-
| Test attaching javascript event listeners to the controls themselves.  They should not really be able to take over a control, for example, you could move the control every time the mouse hovered over it making it impossible to stop the video
| ?
| ?
|-
| Test playing a large video that must be buffered.  Ensure the buffering works properly
| ?
| ?
|-
| Ensure the poster setting works properly, that seek offsets can be used well.
| ?
| ? (Not yet implemented - [https://bugzilla.mozilla.org/show_bug.cgi?id=449156 #449156])
|-
| Ensure that save page as functionality works and downloads the video/audio as well as the HTML
| ?
| ?
|-
| Test that Embedded Text in the video (and audio) tag is not displayed by Firefox
| ?
| ?
|-
| Test that currentTime returns currentTime inside the video
| A(M)
| Done, test_currentTime.html (only checks that it starts at 0)
|-
| Test duration of a video that is not buffered
| A(m)
| Done, tst_duration1.html
|-
| Test that ended event gets sent and currentTime set properly
| A(M)
| Done, test_ended1.html
|-
| Test that ended event gets sent and currentTime set properly for buffered video
| A(M)
| Done, test_ended1.html
|-
| Test that playback can be restarted
| A(M)
| Done, test_ended2.html
|-
| Should not be able to set networkState
| A(M)
| Done, test_networkState.html
|-
| Test paused
| A(M)
| Done, test_paused.html
|-
| Test cannot set readyState
| A(M)
| Done, test_readyState.html
|-
| Test that volume setting works
| A(M)
| Done, tst_volume.html
|}
|}
* attempt to ensure that you can use play and pause to cause video to get to a certain state and then figure out if it worked by comparing to a static image of the video at that time.
 
* use the above to test autoplay as well and whether or not the controls appear and that the controls appear properly
=== Mime and Lower level concerns ===
* can we style the page differently based on video states of play or pause or stop?  
{| class="standard-table"
* can we change the controls to something of our own creation
|-
* can we use video to animate objects and then insert them into a background of frame??
| class="header"|Test Idea
| class="header"|Test Type
| class="header"|Status
|-
| Test mimetypes, and test that we handle them by default when they are detected, if we want to do that
| ?
| ?
|-
| Try to test if we can unsync the video and audio
| ?
| ?
|-
| Change the framerate, ensure video playback changes
| ?
| ?
|-
| Give an invalid mime type for the provided file (when the file is a codec we support
| ?
| ?
|-
| Give a valid mime type when the file isn't a codec we support
| ?
| ?
|-
| Give an incorrect,but supported, mime type for a file that is a supported codec
| ?
| ?
|-
| Test for any relevant events - video stop/start/resume/reset etc
| ?
| ?
|}
 
 
=== CSS/Layout Considerations ===
{| class="standard-table"
|-
| class="header"|Test Idea
| class="header"|Test Type
| class="header"|Status
|-
| ensure videos obey their space directives for the content they are embedded within
* make video bigger than space (in div, in table, in frame/span)
* make video smaller than allotted space
* make video smaller than normal size (test aspect ratio is maintained)
* make video much larger than normal size -can you size video outside of content
* have the area surrounding a video be determined by an inherited CSS string that is changed after the page loads and autoplay begins such that the new size is too small for the video
* do above case and just keep the same size, and ensure the page changes and reflow happens but video doesn't restart
| ?
| ?
|-
| Can we style the page differently based on video states of play or pause or stop?
| ?
| ?
|-
| Test adherence to display:none - does an autoplay=1 video still play?
| ?
| ?
|-
| Test reflow if a presently display:none video is suddenly displayed
| ?
| ?
|-
| Test embedding video and audio in tables
| ?
| ?
|-
| Test embedding video and audio in iframes
| ?
| ?
|}
 
=== Leak/Thread/Crash Testing ===
{| class="standard-table"
|-
| class="header"|Test Idea
| class="header"|Test Type
| class="header"|Status
|-
| Test using a completely contained video (i.e. file local on server)
| ?
| ?
|-
| Test using a buffered file
| ?
| ?
|-
| Test ten videos all in their own tabs and/or windows, playing simultaneously
| ?
| ?
|-
| Test ten videos all on the same page, playing simultaneously in different codecs
| ?
| ?
|-
| Test ten videos all on the same page, playing simultaneously in the same codec
| ?
| ?
|-
| Test corrupted video file
| ?
| ?
|-
| Test non-existing video file
| ?
| ?
|-
| Test file being served from external https link on a non-SSL page.
| ?
| ?
|-
| Test file being served from an external http link on a SSL page
| ?
| ?
|-
| Test file being served from chrome link on a content page
| ?
| ?
|}
 
== Things that Should Be Done ==
{| class="standard-table"
|-
| class="header"|Test Idea
| class="header"|Test Type
| class="header"|Status
|-
| What happens if you put video inside content editable? can you drag the video around?
| ?
| ?
|-
| What about inside of an SVG section? (note: I don't think this is possible)
| ?
| ?
|-
|}
== Manual Testing ==
* Litmus tests will be created for the front end controls of audio and video. 
** [https://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=26&testgroup_id=105&subgroup_id=1141 FFT section]
** BFT section
 
== NEEDED for Automated testing ==
* Ensure that you can use play and pause to cause video to get to a certain state and then figure out if it worked by comparing to a static image of the video at that time.
* what tests should be done using the various backend frameworks?  How do you test that theora works on all platforms etc
* what tests should be done using the various backend frameworks?  How do you test that theora works on all platforms etc
* Test mimetypes, and test that we handle them by default when they are dtected, if we want to do that
* Ensure that we can automate interacting with the video by doing various JS things and then checking the states and attributes of the video both the constants and the actual attribute states. (in order to play to a certain time, for example)
* test that we can automate interacting with the video by doing various JS things and then checking the states and attributes of the video both the constants and the actual attribute states.
* Try to test if we can unsync the video and audio and hopefully that will not be possible.
* Try changing the framerate from the normal speed
* Try giving it a bad mime type for the provided file (when the file is something we support)
* Try the above by giving a good mime type when the file isn't something we support
* test that events are fired when a video finishes, starts and is paused and is resumed
* ensure proper observers catch those events and can do interesting things with them when they catch them.
* ensure videos obey their space directives for the content they are embedded within
** make video bigger than space (in div, in table, in frame/span)
** make video smaller than space
** make video smaller than normal size
** make video much larger than normal size -can you size video outside of content?
** have the area surrounding a video be determined by an inhereited CSS string that is changed after the page loads and autoplay begins such that the new size is too small for the video
** do above case and just keep the same size, and ensure the page changes and reflow happens but video doesn't restart.
** include a data URL in the video
** use a javascript source file in the src directive and see what happens (or a data URL encoded javascript alert that claims to be a video based on its mimetype, maybe even begins with OGG)
** Litmus or gristmill tests using the controls and verifying that the interaction works properly?
** automatically remove the controls after a moment of playing the video via javascript.
** create a shape or a object that sits over a video
** use javascript to turn the volume all the way up on the vdeo and blare it at the user. (shoudl we allow that -- I'm htinking about ads here that might exploit that)
** see if yo can find a way to allow video to block the user from the chrome controls by positioning the video over it or in a flotaed frame.
** Can you launch code from a video by doing soehting evil inside the video stream?
** do all the interactivity tests with javascirpt with just audio too. 
** audio - test turning controls on and thensizing the frame area such that there is no location for the controls to exist in.
** use all the ideas for video with audio especially the ones involved in embedding audio into the web page.
* test what happens if you have video with an embedded sound clip in it and you specify an audio clip and the javascript plays both at the same time, pehraps the audio doesn't match the video.
* test that playback of both formats stops on closing the tab
* test that reload of page causes both to restart
* test that back/forward work as expected (video back to beginning)
* session restore should restore the original state of the video, not the mid-point where it was when you crashed. (I think).
* test cross site video reference - should probably be disallowed
** or should it?  is there any harm in that? pehraps you want to rembed someone else's hosted video?
* test attaching javascript event listeners to the controls themselves.  They should not really be able to take over a control, for example, you could move the control every time the mouse hovered over it making it impossible to stop the video
* test playing a large video that must be buffered.  Ensure the buffering works properly
* Ensure the poster setting works properly, that seek offsets can be used well.
* Ensure that save page as funcitonality works and downloads the video/audio as well as the HTML
* what happens if you put video inside content editable? can you drag the video around?
** what about inside of SVG?
** test dragging video to a new window and that it doesn't stop playing (via tab drag)
** test that it continues to play (or doesn't play - current behavior is to continue) when you switch to a new tab.
* play with the access controls and see if you can force hieght to change based on a wacked out width setting and telling it to maintain aspect ratio - it is currently fixing height to a value
* Test accessibility of the video/audio controls


= Discovered Bugs =
= Discovered Bugs =
* chaning width crops controls and can no longer properly control the video any more
* chaning width crops controls and can no longer properly control the video any more
** should there be a context click menu for video control?
** should there be a context click menu for video control?


= Schedule Scoping =
= Schedule Scoping =
 
* Probably 3 weeks to get all these tests written unless we timebox a significant portion of them
  Estimate amount of time it will take to complete feature
  (Consider nightly builds, security reviews, bugs turnaround time)


= References =
= References =
 
* [http://www.whatwg.org/specs/web-apps/current-work/#video WHATWG specification]
  Include notes, links, specs, relevant bugs
* Something we should test in Litmus or Gristmill: {{bug|448630}}
* Primary bug (everything else linked from here): {{bug|382267}}
* [[Firefox3.1/Video_Security_Review|Security Review Notes]]

Latest revision as of 03:59, 24 March 2009

« QA/Firefox3.1/TestPlan

Video and Audio Test Plan

Overview

Video and audio HTML 5 suport has landed on trunk for the 3.1 release. It implements the WHATWG specification for these tags. So, far, only the ogg-theora codec support has landed. However, the tests below must be tested with each codec supported unless otherwise noted.

Likewise, everywhere that "video" is used, it is also assumed that audio will be tested for the same test, except in those cases where it doesn't make sense for this to occur.

Test Strategy

I am breaking these into three basic blocks of testing:

  • What must be done to ensure good coverage of the feature
  • What should be done to test around the area of the feature and its interoperability with other HTML content
  • What might be done in a wonderful perfect world i.e. less important stuff that might get timeboxed
Test Legend

For Test Type, we use:

  • L - Litmus
  • G - Gristmill
  • M - Mochitest
  • R - Reftest
  • ? - TBD

For Test Status, we use:

  • ? - TBD (also indicates that this test idea might be complete foolishness - has not been sized yet at all)
  • D - Done - test written and checked in. If possible, "D" will be a link to the test in MXR or Litmus
  • I - In Progress - test written, but not checked in (awaiting review, for instance). The "I" should be a link to the bug where it is awaiting review.
  • ' ' (blank) - Indicates that the test has not been written, but it was decided that this is a test we will be writing, i.e. it is not complete foolishness.

Things that Must be Done

Functional Tests

Test Idea Test Type Status
Analyze the unit tests to ensure they test all default values of the attributes ? ?
Test control toggle to determine if the controls appear and that the controls appear properly A(M) Done, test_controls.html
Test autoplay to ensure proper default and can be set to proper values A(M) Done, has bug (test_autoplay.html)
Change the controls to something of our own creation ? ?
Use Video to animate objects and place as a background to content ? ?
test dragging video to a new window and that it doesn't stop playing (via tab drag) ? ?
Test that it continues to play (or doesn't play - current behavior is to continue) when you switch to a new tab. ? ?
Test accessibility of the video/audio controls ? ?
Test that the controls work properly (litmus? gristmill?) ? ?
Test automatically removing controls from javascript - would this be a bug? A(M) Done(test_bug448534.html)
Use javascript events to override the volume control set by the user (should not be able to do this, I think, considering how much we hate the ads on tv that do this) ? ?
Test that playback of both formats stops on closing the tab ? ?
Test that reload of page causes both to restart ? ?
Test that back/forward work as expected (video back to beginning) ? ?
Test that Session restore should restore the original state of the video, not the mid-point where it was when you crashed. (I think). ? ?
Test attaching javascript event listeners to the controls themselves. They should not really be able to take over a control, for example, you could move the control every time the mouse hovered over it making it impossible to stop the video ? ?
Test playing a large video that must be buffered. Ensure the buffering works properly ? ?
Ensure the poster setting works properly, that seek offsets can be used well. ? ? (Not yet implemented - #449156)
Ensure that save page as functionality works and downloads the video/audio as well as the HTML ? ?
Test that Embedded Text in the video (and audio) tag is not displayed by Firefox ? ?
Test that currentTime returns currentTime inside the video A(M) Done, test_currentTime.html (only checks that it starts at 0)
Test duration of a video that is not buffered A(m) Done, tst_duration1.html
Test that ended event gets sent and currentTime set properly A(M) Done, test_ended1.html
Test that ended event gets sent and currentTime set properly for buffered video A(M) Done, test_ended1.html
Test that playback can be restarted A(M) Done, test_ended2.html
Should not be able to set networkState A(M) Done, test_networkState.html
Test paused A(M) Done, test_paused.html
Test cannot set readyState A(M) Done, test_readyState.html
Test that volume setting works A(M) Done, tst_volume.html

Mime and Lower level concerns

Test Idea Test Type Status
Test mimetypes, and test that we handle them by default when they are detected, if we want to do that ? ?
Try to test if we can unsync the video and audio ? ?
Change the framerate, ensure video playback changes ? ?
Give an invalid mime type for the provided file (when the file is a codec we support ? ?
Give a valid mime type when the file isn't a codec we support ? ?
Give an incorrect,but supported, mime type for a file that is a supported codec ? ?
Test for any relevant events - video stop/start/resume/reset etc ? ?


CSS/Layout Considerations

Test Idea Test Type Status
ensure videos obey their space directives for the content they are embedded within
  • make video bigger than space (in div, in table, in frame/span)
  • make video smaller than allotted space
  • make video smaller than normal size (test aspect ratio is maintained)
  • make video much larger than normal size -can you size video outside of content
  • have the area surrounding a video be determined by an inherited CSS string that is changed after the page loads and autoplay begins such that the new size is too small for the video
  • do above case and just keep the same size, and ensure the page changes and reflow happens but video doesn't restart
 ? ?
Can we style the page differently based on video states of play or pause or stop? ? ?
Test adherence to display:none - does an autoplay=1 video still play? ? ?
Test reflow if a presently display:none video is suddenly displayed ? ?
Test embedding video and audio in tables ? ?
Test embedding video and audio in iframes ? ?

Leak/Thread/Crash Testing

Test Idea Test Type Status
Test using a completely contained video (i.e. file local on server) ? ?
Test using a buffered file ? ?
Test ten videos all in their own tabs and/or windows, playing simultaneously ? ?
Test ten videos all on the same page, playing simultaneously in different codecs ? ?
Test ten videos all on the same page, playing simultaneously in the same codec ? ?
Test corrupted video file ? ?
Test non-existing video file ? ?
Test file being served from external https link on a non-SSL page. ? ?
Test file being served from an external http link on a SSL page ? ?
Test file being served from chrome link on a content page ? ?

Things that Should Be Done

Test Idea Test Type Status
What happens if you put video inside content editable? can you drag the video around? ? ?
What about inside of an SVG section? (note: I don't think this is possible) ? ?

Manual Testing

  • Litmus tests will be created for the front end controls of audio and video.

NEEDED for Automated testing

  • Ensure that you can use play and pause to cause video to get to a certain state and then figure out if it worked by comparing to a static image of the video at that time.
  • what tests should be done using the various backend frameworks? How do you test that theora works on all platforms etc
  • Ensure that we can automate interacting with the video by doing various JS things and then checking the states and attributes of the video both the constants and the actual attribute states. (in order to play to a certain time, for example)

Discovered Bugs

  • chaning width crops controls and can no longer properly control the video any more
    • should there be a context click menu for video control?

Schedule Scoping

  • Probably 3 weeks to get all these tests written unless we timebox a significant portion of them

References

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3.5/Video_And_Audio_TestPlan&oldid=135764"