QA/Firefox 4/Test Plan/WebM Support
- 1 WebM Support
- 2 Feature Release Readiness Assessment
- 3 Feature Documentation
- 4 Feature Bug Management
- 5 Feature Test Items
- 6 Feature Tests
- 7 Community Test Events
- 8 Feature Documentation Details
- 9 Feature Release Readiness Assessment Details
- 10 Feature Bug Management Details
- 11 Feature Test Items Details
- 12 Feature Tests Details
- 13 Community Test Events Details
- Development Status: - Completed (Alpha 5)
- Feature Testing: - On-going
- Team: Dev: cpearce, roc, QA: juanb
- Tracking Bugs:
WebM is a new open source video format. Firefox 4 will support this new video format similar to the way it supports Ogg.
The table below provides a top level go/no go assessment of whether the feature is release ready for the given milestone.
|#Alpha 5||In-house QA has not engaged yet fully in testing WebM. If any testing has been done manually it is from community.|
|#Project_Wiki||Wiki Links to all feature related entries|
|#Developer_Links (blogs)||Developer links to feature related sites|
|#Other_Docs||Web links to feature related sites|
|#Developer_QA_Review||Details from developer and qa discussions regarding feature test strategies and issues.|
|#Bug_Tracking||Top level bugs tracking feature|
|#Bug_Verification||Feature bugs that need verification|
|#Bug_Triage||Links triage bug tasks|
The table below provides a breakdown of all feature items that should be covered and how they will be tested. Not all items will be covered by internal QA team members. It is important to list what should be covered. If it is not covered, list it as not covered.
Note: not all items listed below will apply for a given feature
|Test Item||Description||Covered By||Status|
|Item 1||Item 1 Description||Developer Tests|
|Item 2||Item 2 Description||Beta tester exposure|
|#Topsites||Top internet sites compatibilities|
|#Developer_Tests||Links to automated developer tests||Information not yet gathered.|
|#Mozmill_Tests||Links to automated mozmill feature test cases||Will consult with Henrik to see how applicable these can be.|
|#Smoke_Tests||link to smoke tests||Not started.|
|#Regression_Tests||link to BFT and/or regression tests||Not started.|
|#Functional_Tests||link to FFT and/or complete functional tests||Not started.|
|#Testdays||Links to test day event results for feature|
|#Bugdays||Links to bug day event results for feature|
|#Meetups||Links to Meetup events for feature|
Project wiki not available. Refer to developer links with blogs on the subject.
- Mozillians Blogs on WebM:
- How to try WebM videos:
- YouTube samples:
- A non-YouTube example:
Preview builds of Firefox with WebM support are already available at http://nightly.mozilla.org/webm. Additionally, the Participatory Culture Foundation has updated their Miro Video Converter to be the first tool to produce WebM videos.
Developer QA Review
The QA person responsible for the feature should hold a formal interview with the lead developer or feature champion. Below are questions that should be asked in the interview:
- Do we have automated tests for the feature?
- Not yet answered.
- What do they cover?
- What do they not cover?
- How well do they cover the feature?
- What are the important areas we should focus on?
- Basically, we want to know if there are WebM videos out there which don't play/seek like you'd expect them to. For example: videos that play too slow or too fast. Videos that don't play at all. Seeking (clicking on the scrubber bar to change the playback position) which doesn't work or works funny/slow. Sound or video not working, or visual artefacts (like bug 566987) during playback or seeking. Playback with the audio not in sync with the video.
- What are the dependencies?
- You mean what libraries do we use? We're using Google's libvpx for VP8 decoding, Xiph.org's libvorbis for Vorbis decoding, and Matthew Gregan's (a MoCo employee) libnestegg for WebM demuxing.
- What is our comfort level with this feature in its current state?
- The WebM playback logic is shared with the Ogg playback logic, so all the HTML5-spec compliance/UI stuff should be the same as Ogg backend. Theoretically the only new bugs you find should be in the underlying decoding logic, not the playback logic. The logic surrounding the HTML5 controls/event architecture should be backend independent now, so...
- What feedback would you like from QA?
- It would be good to know that you guys have tried playing a variety of WebM videos on a variety of platforms and found no problems. :)
- It would be good to know that playback works ok on PPC MacOSX, we've not tried WebM playback on that platform, as we don't have a PPC Mac in our office.
WebM support was included in Alpha 5 without much in-house QA testing, and at the moment is covered by nightly usage.
By Beta 1 we will have litmus tests and an assessment of automated tested coverage, as well as a thorough pass by in-house QA. The similarities with Ogg support from a UI perspective should allow us to build upon existing tests and approaches.
- Top level bugs tracking feature. Include any relevant bug queries that are helpful for tracking feature status.
|bugzilla query url link||query description|
- Feature bugs that need verification
Ideally we will triage WebM bugs daily, confirming new unconfirmed bugs, and making sure we give them the right exposure and bubble them up accordingly. A quick search for new and unconfirmed bugs can be found here:
We should take a look at a list like this as frequently as daily.
Does this feature require localization testing?
- Context menu items for video, or video controls text should be localized.
- Must investigate RTL localization requirements for these elements of video.
- Video controls must have accessibility controls.
- Keyboard strokes invoking the video controls must be tested for functionality.
- Interactions between WebM video and other types of video formats such as Ogg and Flash must be tested for compatibility. Loading tabs with different types of video formats should work well.
- Check for top addons compatibility while loading and playing WebM videos
- Check for compatibility with major web sites, such as Facebook, Hulu, Yahoo, YouTube, as well as those sites which have content for WebM videos.
- Links to automated developer tests
If a particular feature needs manual tests which should also be covered by Mozmill tests please add the "[mozmill-test-needed]" whiteboard entry to the feature implementation or regression bug.
List of Mozmill Tests:
- Links to automated mozmill feature test cases
- links to litmus smoke tests or description
- links to litmus BFT and/or regression tests description
- links to litmus FFT and/or complete functional tests description
- No testday has been scheduled yet.
- No bugday has been scheduled yet.
- No meetup has been scheduled yet. For testing purposes testdays and bugdays seem to be of better value.