User:Ashughes/Test Plans/Firefox 3.next/Network Error Pages Test Plan
Contents
Overview
When a requested web page cannot be loaded, Firefox should display an error page that not only describes the problem, but which also presents the user with tools to resolve the issue. It also makes sense to extend this functionality to webserver 404 (File not found) responses as well.
Implementation is being tracked in bug 482874.
Potential Use Cases
404 Pages
- Override 404 pages containing less than 250 characters served by Apache (minimizes chances of overriding custom error pages)
- Do not override 404 pages containing greater than 250 characters
Suggestions
Alternate URLs i.e. "Did you mean...", drawn from Places DB using "edit distance" criteria, only appears if match is pretty close
- Appear when there is a very close match
- Does not appear when there is not a close match
- Suggested URLs are well-formed
- Suggested URLs work
Search
Pre-populate site-search box & button, primarily used on 404 error pages, terms derived from URL
- Appears for 404 error overrides
- Does not appear for DNS Not Found errors
- Search box is pre-populated when it appears
- Clicking the search button triggers a search
- Use the user's preferred search engine if it makes sense to do so
- Google, Yahoo, MSN make sense
- Wikipedia, Amazon, Ebay do not
Global Web-search
Similar to Search but this works for both 404 and DNS error pages, search terms derived from URL.
- Appears for 404 error pages
- Appears for DNS Not Found error pages
- Search box is pre-populated when it appears
- Clicking the search button performs a search
- Use the user's preferred search engine if it makes sense to do so
- Google, Yahoo, MSN make sense
- Wikipedia, Amazon, Ebay do not
View Original Error
Ability to view the original 404 error page
- UI appears (button, link, etc)
- UI is functional
Manually View 404 Page
Ability to view the Firefox 404 override page manually when it did not automatically override
- Display a conventional 404 page that will not be overridden
- 404-override page can be displayed
General
- Do the error pages validate?
- Is the spelling/grammar correct?
- Is the spelling/grammar correct for localizations?
- Does the layout look decent?
There are some other more general things to test. Some subjective, some not.
Test Strategy
Test Case URLs
- Develop a set of test URLs to trigger each error
- 404 Pages
- < 512 bytes:
- http://www.almostinfinite.com/qwertyuiop (283 bytes)
- > 512 bytes:
- http://www.mozilla.org/qwertyuiop (2168 bytes)
- Top Sites:
- < 512 bytes:
Litmus
- Layout and Content of error pages
- Suggestions
- Is the URL well-formed?
- Does the URL navigate correctly?
- Suggestions:
- Did you mean URL
- Other suggested URL
- Search
- Is the URL well-formed?
- Does the URL navigate correctly?
- Suggestions:
- Search domain for string
- Search Google for URL
- Try again
- Tools
- More Information
Caveats
- Browser error page vs Custom webserver error pages
- XSS, prevent by making page chromeless
- Google Link Doctor privacy
- One-click access to webserver error page?
Automated Tests
NOTE: Most of these may be best served as an automated and manual test
- If a 404 error page meets the constraint it should be overridden.
- If a 404 error page does not meet the constraint, it should *not* be overridden.
- Make sure Suggestion appears when there is a very close match.
- Make sure Suggestion does not appear when there is not remotely close match.
- Verify that the URL is well-formed, and that the anchor element displaying it works.
- Make sure Search appears for 404 error overrides.
- Make sure Search does *not* appear for DNS Not Found errors.
- Verify that the search box is pre-populated when it appears, and verify that the button actually triggers the search.
- Verify that the search uses the user's preferred search engine when it makes sense (e.g. Google or Yahoo, but not Wikipedia).
- Verify that the search is actually constrained to the site.
- Make sure that Global Websearch appears both for 404 error overrides and DNS Not Found errors.
- Make sure that the Global Websearch search box is pre-populated with search terms, and verify that the button actually triggers the search.
- Make sure the Global Websearch search uses the user's preferred search engine when it makes sense (e.g. Google or Yahoo, but not Wikipedia).
- Verify that the UI control (button, link, etc.) appears in the original error page, and verify that it works.
- [Manual Error Page] Trigger a conventional 404 page that will not be overridden (just too large, etc.). Verify that the UI control (button, link, etc.) is visible in the browser chrome, and verify that it will show the 404-override page.
- Do the error pages validate? (I don't know if this is critical, but if we can automate it, it won't hurt to test.)
Manual Tests
Potentially all of the above plus...
- Is the spelling and grammar correct? (Obviously localization dependent).
- Does the layout look decent? (Keep in mind that error pages may vary in layout from error to error since only some features will apply to each error page).
Schedule
Estimate amount of time it will take to complete feature (Consider nightly builds, security reviews, bugs turnaround time)
Community Contributors
Feel free to add yourself here if you wish to help QA out.
References
Sprint here
Relevant Bugs:
- bug 482874 Implementation tracking bug
- bug 479922 Feature suggestion bug
Boris' Blog Post here
Network Error Messages here and here
Competitive Comparison here
Mockups here