QA/TDAI/PlacesQueryTests

From MozillaWiki
< QA‎ | TDAI
Jump to: navigation, search

Testing Complex Places Queries and LiveUpdate

This is the automated test development project born from bug 384226

I propose tackling this in three phases of tests. Obviously, I can't combine every query type and query option in every way, but I think we can make some decently complex queries to understand whether or not the complex queries and live update features with those queries will continue to work. So, two phases of tests will concentrate on ensuring that liveupdate works for complex history and bookmark queries. The third phase will "stress test" doing more complex queries on history and bookmarks using a large database.

It remains to be seen whether the large places database will be generated on the fly or will be some canned database.

I have started a tracking matrix for the query attributes under test, and will keep it updated as I add tests and analyze what the existing tests test.

You can check it out here: http://spreadsheets.google.com/pub?key=pDgplBvniukmp-6lp8VuaYw

Phase 1

Testing: History, complex queries with live update

  • Verification strategy: Input known data into database, ensure that data is properly queried, modify data set, ensure that live update captures the modification.

Phase 2

Testing Bookmarks, complex queries with live update

  • Verification strategy: Input known data into database, ensure that data is properly queried, modify data set, ensure that live update captures the modification.

Phase 3

Testing Both History and Bookmarks queries on a generated or canned database. Ensure that proper result sets are generated by the query. Ensure that live update still works by adding/changing/deleting members from the query set.

This may require a canned database and so may need to do its verification against the database itself using the same SQL that the underlying API is generating. Live update tests here will be more sporadic - not on every test, and may need to have the ability to pick a random data member to modify and then search for that data member in the live-updated result set.

Notes And Questions

  • How similar should the tests in Phase 1 and Phase 2 be? There should probably be some degree of similarity since one tests history and the other bookmarks. However, there are also options that will be more likely to be used on bookmarks (ie folders) and those should be tested specifically in combination with other query attributes. So, to what degree should these be similar and different?
  • Do we expect much to change with large databases? I don't think XPCShell is a decent mechanism for checking performance, and so I don't want to get into performance questions here with it. But, are there other stress points that we should be aware of?
  • What about UTF-8 Data as search terms and folder names?
  • Do we need to test UTF-8 data inside the database as well? (i.e. have the bookmark title be in unicode).