WebDriver/Meetings/2018-11-05

From MozillaWiki
Jump to: navigation, search

Agenda

  • Deprecation plan for marionette_driver
  • Intermittents update
    • [wfm] Intermittent test_ev_certificate.py TestEVCertificate.test_ev_certificate | AssertionError: u'verifiedDomain' != 'verifiedIdentity' (Bug 1289344)
    • [fixed] Intermittent Mn/MnH Automation Error: mozprocess timed out after 1000 seconds running (Bug 1391545) - caused by unskipping start timeout test
    • [needs investigation] Intermittent testing/marionette/harness/marionette_harness/tests/unit/test_window_close_content.py TestCloseWindow.test_close_browserless_tab | IOError: Process killed because the connection to Marionette server is lost (Bug 1501130)
    • [beta sim - needs investigation] Intermittent Linux Mn testing/marionette/harness/marionette_harness/tests/unit/test_screenshot.py TestScreenCaptureChrome.test_formats (Bug 1504201)

Minutes

Deprecation plan for marionette_driver

ato
I think there is agreement that the Marionette Python client is not the greatest thing we offer.
I want to propose that we come up with a deprecation plan, where the end goal is not decommission it.
But because we don't have an alternative to using marionette_driver internally to mozilla-central, that is not a realistic goal.
It was a shame our intern project proposal was rejected, because it would've given us an alternate client.
We could have used this to easily replace marionette_driver in e.g. Firefox UI tests.
We have the Marionette unit tests and a couple of others in the tree using it.
The second complication is that we have out-of-tree usage marionette_driver in both webpagetest and wptrunner.
As I said, it would be good to replace it with something else but sadly we don't have it.
But we should we stop maintaining it, meaning no updates to PyPI, and make it only for internal usage.
Out of tree, people could use Selenium + geckodriver.
wptrunner also uses Marionette driver because James had reservations due to possible performance reasons using geckodriver.
AutomatedTester
I'm OK with a deprecation plan but we have to sort out WPT.
How much work would it be to switch WPT over to using wdclient + geckodriver?
ato
Or Selenium.
AutomatedTester
What scares me are the webpagetests (mozbase, marionette_driver) which have a lot of users: they have to be made on-board first
ato
I foresee a future where we only have marionette_driver as an internal tool in mozilla-central.
Hard to remember the release process, even harder to work with backwards compatibility.
We also have this complexity with geckodriver, but it is better maintained and tested.
However, it seems redundant to go through this pain twice.
Maybe we should make a list which projects make use of Marionette, and get them to port away from Marionette.
About a years time sounds like we would need.
whimboo
There's already tooling available to wdspec, but reftests have special windows and commands.
ato
Yes, there's lots of tooling around Marionette to bootstrap it.
But unfortuantely we don't have any better clients to offer them.
So as much as I'd like to, we can't realistically replace marionette_driver internally in m-c.
AutomatedTester
Let's get a document started to map out stakeholders and to collaborate on.
Then we can make an informed decision from then onwards.

ACTION ato: Start document

whimboo
Is there a more accurate way to tell who is using it than the dependency lists?
I don't think everyone subscribes to tool-marionette@.
AutomatedTester
We could put deprecation warnings in there.
whimboo
But that would spam our try logs, so we wouldn't want to do them for internal usage.
ato
We don't want to break people using webpagetest, so this is a major concern.
I care more about the user experience than the software in this case.

Intermittents update

whimboo
Four items for today.
Two highly occuring issues, one with Firefox UI tests related to certificates has been fixed.
There was a huge spike, but now it's not happening anymore.
I wonder what happened. Network problems? Perhaps some preference switches?
The second one is for mozprocess timeouts for Marionette jobs, which was actually caused by a test I unskipped lately.
But I have now provided a fix for the test, and it seems to work fine.
I think we can now close the meta bug for mozprocess timeouts.
But there are two new highly intermittent bugs.
TestCloseWindow.test_close_browserless_tab: started recently on central, so I will see if I can backfill.
It must be something in Firefox, because we haven't made any changes in Marionette dealing with closing tabs or browsers.
Second is about beta simulation and screenshots. Will need some investigation.
ato
How serious is it?
whimboo
Two screenshots in chrome, and those screenshots are different.
I can't check the differences because we truncate the logs.
ato
There's a marionette.log.truncate preference that turns off log truncation.
whimboo
Yes, I was proposing we enable that specifically for this test.
So when this is part of the next beta simulation we might have a better idea of what's going on.
ato
I don't think we've changed anything in our screen capturing code.
Although there was one change done by a contributor, because obviously beta goes quite a long way back.

Status updates

whimboo
I have some status updates, but I don't see ato's.
Two alerts in an Execute Script call, we actually have the same behaviour as IE.
I added a test which does a page load after an alert, so it means when the driver closes the alert a page load will happen.
Whatever a user puts in a script after a user prompt it will cause side-effects.
The specification is wrong in this case, so I will at some point submit a PR.
ato
If you test this with Chrome, that might provide us enough data to classify this as a bug.
In the case of a bug there's nothing for vendors to disagree about.
The specification must reflect the reality of implementations.
whimboo
[explains JavaScript documentation generated from XPCOM Marionette code]
AutomatedTester
I'm working on the shadow DOM changes to WebDriver when I have the chance.
But it's taking some time because everyone seems to disagree.

Status updates

(Spoken status updates in bold.)

PTO/travel (🍂)

  • whimboo - Nov 8th - PTO
  • ato away 9 November
Retrieved from "https://wiki.mozilla.org/index.php?title=WebDriver/Meetings/2018-11-05&oldid=1203345"