QA/BrowserID Automation
How It Works
Jenkins is the virtual host that runs the cmds that are sent to saucelabs which has webdriver running. So any scripts/run-all.js logs are on jenkins and webdriver logs are on saucelabs. Capice'? KICK OFF RUN https://ci.mozilla.org/
- login with ldap passwd
- jenkins kicks off runs based on project names (e.g. identity.browserid.dev.mac.firefox.14)
- if you 'configure' inside the project, you'll see the cmd line args etc...
- if you want to kick something off manually, get on IRC and PM jenkins with something like this:
jenkins: build identity.browserid.dev.mac.firefox.14
- NOTE: the 'build now' on the jenkin's UI doesn't work
RESULTS
- How this works: Jenkins looks at results/test.xml file with pass/fail which is generated when you specify output at --output xunit
- Run is complete - look at last fail or last success permalinks or in build history list
- Click on a failed run
- If there isn't any inteligent error reporting. Click on Console link
- find the failing test, and goto the saucelabs URL to view the webdriver output
- NOTE: sign in to saucelabs with your env variable user:passwd
Selenium
Known Issues
Safari: browser cache is not cleared between each test (except chrome, FF) http://code.google.com/p/selenium/issues/detail?id=40
Opera: goes fullscreen on mac opens in new tab instead of specific sized pop up (mac only)
Android: only supports stock browser with no chrome ( can't user Firefox or Chrome )
iOS: dunno must provision hooked up to xcode to install app
some browsers can't close alert dialog (safari, opera) selenium.common.exceptions.WebDriverException: Message: u"A modal dialog was opened. The SafariDriver does not support interacting with modal dialogs. To avoid hanging your test, the alert has been dismissed. For more information, see http://code.google.com/p/selenium/issues/detail?id=3862\nBuild info: version: '2.31.0', revision: '1bd294d', time: '2013-02-27 20:52:59'\nSystem info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.2', java.version: '1.6.0_41'\nSession ID: null\nDriver info: org.openqa.selenium.safari.SafariDriver\nCapabilities [{platform=MAC, javascriptEnabled=true, cssSelectorsEnabled=true, secureSsl=true, browserName=safari, takesScreenshot=true, version=6.0.2}]" ; Screenshot: available via screen ; Stacktrace: Method newInstance0 threw an error in NativeConstructorAccessorImpl.java
Jenkins and Saucelabs
Feature Requests
- need a way to embed saucelabs URL in results.xml to reduce hunting for logs
- link to github test to see src
General
- General build failure from fetching modules:
- npm http GET https://github.com/downloads/lloyd/node-statsd/0509f85.tgz
- npm ERR! browserid@1.0.0-b1 preinstall: `node ./scripts/lockdown`
I'm going to log a bug on this
- User email validation
- This seems flakey, often you'll see 'timeout' error as the browser sits on .watch() for validation
- Errors are intermittent and all over the map for some browsers
- osx_safari_5 example:
run 322
- change-password-test.js.12: back to the first browser: should be signed out of 123done on reload: succeeds
- health-check-tests.js.12: switch back to main window, look for the email in acct mgr, then log out: succeeds
- new-user-secondary-test.js.11: verify logged in automatically to 123done in first browser: succeeds
run 320
- primary-shuts-down-single-email.js.10: Authed user tries to log in: succeeds
run 319
- health-check-tests.js.12: switch back to main window, look for the email in acct mgr, then log out: succeeds
Issues
Approach: need to get all builds to show green on Jenkins dashboard. Starting with are more stable platforms, here's a breakdown of why we have failures with Jenkins/Saucelabs.
- dev.win7_chrome
- primary-shuts-down-single-email - testidp.org doesn't get confirmed
- dev.osx_chrome_14
- webdriver went stale and didn't kill existing browser
- dev.linux_firefox_16
- sign in button just spins
Saucelabs
- Currently using out of date browsers and configs. Need to update.