Automated Regression Testing (mike/hallvord?)
Is it possible to collaborate on a tool between Web Compat and Rel Man for automated regression testing? What are the requirements that each of us have? Would this be better discussed in person in Whistler?
- miketaylr: we invited lawrence to join us. Web Compatibility and Release Management are both potentially interested in tracking a set of sites for regression. For WebCompat day to day, For Release Management maybe more specific releases. What would the requirements from RM.
- lawrence: Are we breaking any Web content? Have we introduced new regressions, we broke them or the site broke us. Could be changed in layout, security, graphics, etc. And then gather the information on who we should contact to fix it. Desktop and Mobile (Android). If we break something, the second question is then how bad do we break the Web. 1st one is Can I release 2. Should we fix it? 3. How do I convince them to fix it?
- miketaylr: One of our questions, which sites do we track? We need a system with a flexibility where we can grow the system.
- hallvord: We have two sorts of problems: Things we know which are broken. Things more explorative where we want to discover if it breaks. We need some kind of dictionary for equivalence of error messages across browsers. CSS Comparisons. Error tracking. HTTP redirects. These tests need to run in two engines simultaneously to avoid disparities. Change of content (widgets, ads), make it harder.
- miketaylr: One recent example is the ClojureScript bug, it would be good to know in advance how much we will break Web sites.
- hallvors: As a requirement we need to have something where we can modify the environment before testing.
- miketaylr: There's a potential for overlap. And it looks also like a huge undertaking.
- hallvors: Using marionette or slimerJS is not that complicated, but running two engines simultaneously and they have their own differences. PhantomJS/SlimerJS is a good start.
- miketaylr: 2 engines interesting for RM.
- lawrence: as a second priority, it is interesting. If we do a change, that is already broken in Chrome, it's kind of easier for not being so worried about breaking things.
- miketaylr: It would be ideal to build something small, that is modular so we could easily add to it.
- karl: There are parts of the infrastructure that are very similar. The things that are different, for Release Management it might be a different version of Firefox, for Web Compatibility it might also be a different Engine. If we are able to design a system in a way to isolate the task we should be able to create a tool that is useful for everyone.
- hallvors: The tool that could help us to isolate which gecko has been creating an issue.
- lawrence: We do not have the time to go to each site.
- hallvors: it would be good to be able to store the content of the page, but it starts to create a lot of storage. Like the tool that existed at Opera.
- miketaylr: As a tool, it was becoming unintelligible. Maybe we should write down our requirements. We also have compatipede 1. There is a version 2 which is not opensource, but too uncertain for now. I think we should maybe take a break to think about this and regroup later.
- karl: I think Telemetry has a sort of connection with all of this, in terms of discovering sites that will have issues with certain things.
- hallvord: Do you get the URL?
- miketaylr: you don't but there's progress for doing that.
- lawrence: There are two versions of data collections. There's a differenciation because of regulations in each countries, and they don't collect the exact same thing and the same platforms.
- miketaylr: there is a way. It's possible.
- lawrence: it's tricky. Tracking the features instead of the sites, we will know how many % will be impacted by a change.
- hallvord: The requirements for RM and WC are pretty much the same.
- lawrence: I would like us to try to pick up one thing and to grow it.
- karl: There's probably already pieces of the puzzle that exist in Mozilla, in terms of infrastructure.
- miketaylr: We should meet in Whistler about this.
- hallvord: it would be nice to have a decision before Whistler.
- lawrence: Maybe Telemetry could be a way to start it, and if we need URLs in reports, we should push for it.
- hallvord: The probe is useful and easy. but an example like the ClojureScript is a really small number of Web sites. I looked at the list of sites that were declared.
- karl: We have a list of candidate probes. URL: <WIKI-PROBE>
- miketaylr: ACTION ITEMS?
- karl: I like the idea of lawrence, to stay simple. We need a prototype of a small actionable thing, that we can come to Whistler to show a working tool. We have a big shopping list of things we would like to do, but here is working.
- hallvord: I think I'm a bit past that stage.
- miketaylr: You should explain and advertise the tools already done.
- lawrence: Data useful for other teams could be a first step. The earlier we find the bug the cheaper it will be to fix the things. Commit time would be good.
- ACTION: hallvord to extract Web Compatibilty data through a dashboard which could be useful for other teams (possibly via github.com/mozilla/corsica - office screens)
- ACTION: hallvord to explain the list of tools for Web Compatibility which are doing survey and data collection.
Agenda Web Compat in Whistler 2015 (karl)
The All hands work week meeting in Whistler Canada is very close we need to prepare an agenda so we are maximizing the time there.
- See: https://wiki.mozilla.org/Compatibility/Mobile/2015-06-work-week
- karl: Please look carefully at the agenda. The richness of the discussions are partly dependent on the topic bank and help us shape the meetings we have with other parties. The earlier the better. ACTION ITEM: add topics!!1111!1
- lawrence: It helped my team to book meetings well in advance.
Broken Voices of the Web
- 2015-04-06 - Intent to Implement: DEAA (Antialiasing) for CSS Transformed Layer Edges
- 2015-04-06 - Intent to Remove: InstallEvent Interface
- 2015-04-06 - Intent to Remove: ShadowRoot.getElementsBy*()
- 2015-04-03 - Intent to Remove: exemption for the video-specific prefixed fullscreen API
- 2015-04-02 - Gecko user agent string reference
- 2015-04-01 - (meta) "Intent to implement" and Security & Privacy concerns
- 2015-03-31 - Fwd: DevTools and Intent to Implement process
- 2015-03-31 - Intent to Ship: Permissions API
- 2015-03-31 - DevTools and Intent to Implement process
- 2015-03-31 - Last week in blink intents
- 2015-03-31 - Intent to Implement:
- 2015-03-31 - Intent to Ship: API to support web/native app install banners
- 2015-03-31 - Web Compatibility Summit Summary
- 2015-03-31 - Intent to Ship: Cache Storage API for Service Worker
- 2015-03-31 - (meta) "Intent to implement" - captured some docs on wikimo
Web Compatibility Progress
FIXED (no DUPLICATE)
- 2015-03-30 Desktop billingcp.us is RC4 and ECDHE-RSA-3DES only FIXED
- 2015-03-30 Desktop ebank.khb.hu is RC4 only FIXED
- 2015-03-30 Mobile CSS/HTML does not display properly on http://aprilzero.com WONTFIX
- 2015-03-30 Mobile disney.com.sg sends desktop site to Firefox OS WORKSFORME
- 2015-03-30 Mobile Layout issue on http://m.porsche.com/usa/models/cayenne/cayenne-s/ INVALID
- 2015-03-30 Mobile m.danner.com - Site completely unusable INVALID
- 2015-03-30 Mobile jalan.net sends desktop site to Firefox OS FIXED
- 2015-03-31 Desktop overclockers.co.uk is RC4-only FIXED
- 2015-03-31 Mobile m.tieyou.com sends desktop site to Firefox OS FIXED
- 2015-03-31 Mobile zhtao.cn sends simplified site to Firefox OS WONTFIX
- 2015-03-31 Mobile pixiv.net sends desktop site to Firefox OS FIXED
- 2015-04-01 Desktop https://www.electrabel.be and https://www.partenamut.be chain up to the GTE CyberTrust Global Root FIXED
- 2015-04-01 Desktop Connecting to corp.millenniumbcp.pt with RC4 disabled results in ssl_error_rx_unexpected_application_data FIXED
- 2015-04-01 Desktop Secure Connection Failed for https://acs.nccc.com.tw/ (VISA 3D auth page in Taiwan) FIXED
- 2015-04-01 Mobile ESPN sends Firefox OS a very simple mobile site, lacks video content among other features INVALID
- 2015-04-01 Mobile Espn.go.com doesn't appear correctly on Firefox for Android mobile INVALID
- 2015-04-01 Mobile pnet.co.za does not offer mobile site to Firefox OS FIXED
- 2015-04-02 Mobile www.primerahora.com does not redirect to m.primerahora.com for Firefox OS FIXED
- 2015-04-02 Mobile excite.co.jp sends desktop site to Firefox OS FIXED
- 2015-04-03 Desktop (Desktop)(Pocket) App's offline functionality does not work WONTFIX
- 2015-04-04 Mobile groupon.com serves desktop sites to Firefox OS FIXED
- 2015-04-05 Mobile Permission denied accessing document on voyages-sncf.mobi with B2G INVALID
- 2015-04-05 Mobile Operation not supported error resulting in blank on voyages-sncf.mobi with B2G INVALID
- 2015-04-07 Desktop Password is not saved for trello.com due to not using a form submission FIXED
- 2015-03-30 Desktop porsche.com initial menu is not working
- 2015-03-30 Mobile Investigate Marfeel compatibility
- 2015-03-31 Desktop Aptana download page fails to recognise OS or download software
- 2015-03-31 Mobile (RTL) Facebook: Special characters (like @ .) are put at the beginning of an LTR string when you input an email address on the log in page
- 2015-04-01 Mobile groupon.com intermediate page uses CSS Webkit prefixes
- 2015-04-03 Desktop extranet.ac-grenoble.fr is TLS 1.2 intolerant, RC4 only, and does send the full cert chain
- 2015-04-06 Desktop CB online (https://access.boekhuis.nl) is TLS 1.1/1.2 intolerant