Automated Testing for Web applications and Web sites
Keep adding stuff
- Think about testing in a multi-browser context with different capabilities.
- do not promote your own tools
- Talk about real testing experience in a business context
When doing automated testing for Web applications/Web sites, what do you use? (selenium, PhantomJS, etc?) and why?
- Qunit & Chai for TDD
- CasperJS: fast execution of functional/integration tests on top of PhantomJS (Webkit) + now compatible with Gecko using SlimerJS
- Mechanize + RSpec: We used to use Selenium, but reliability sucked.
- Intern: Uses WebDriver, so works to test all browsers; async support using promises, not callbacks and test interfaces; interfaces for unit testing and integration/function or polling; easy one-command install with npm; built-in code coverage, built-in Grunt support, built-in SauceLabs support; enforces test case modularity; extensible reporteal testing; well-maintained
- Watai: writing integration tests is one thing, writing them in a reliable & maintainable way is another. WebDriver is still unavoidable unless you want smoke tests (PhantomJS-based runtimes forget the web is more than 1 engine).
- CasperJS : easy to use and to setup. (I agree; disclaimer: I'm n1k0)
- PHPUnit: I'm awesome
- CasperJs: the same, just a simple and powerfull tool
- PHPUnit or Atoum http://docs.atoum.org/ for unit tests (PHP applications)
- CasperJS (with PhantomJS or SlimerJS) for functional/UI tests
- Selenium in some projects. One of its advantage is to allowing to run tests on real browsers. But it is slow and a bit bloatware.
- Unit testing: Tytrle & Myrtle + Karma-runner / We want to unit test our code ;)
- Performance testing: Telemetry / It's an experiment right now + Jenkins for Continuous Integration
- pa11y for accessibility testing https://github.com/nature/pa11y
- Jasmine + Karma: Clean syntax, fast runner
- Unit testing: Mocha
- Functional/UI testing: CasperJS (with PhantomJS, and SlimerJS in a near future) + 1
- - integration tests: Watai
- - components tests: Watai
- - unit tests: Mocha on its own, Konacha when in a RoR app.
- UI testing: Webdriver (watir-webdriver)
- RSpec for unit testing for Ruby apps
- Mocha for unit testing of node.js apps
- Zombie.js for headless testing
- Functional: Phantom.js and Zombie.js (disclosure: I maintain the HTML parser it uses), Robot Framework, testling
- Unit: lime_test.php, tap.js and tape. PHP Makes me sad. PHP :(
- Geb (gebish.org)
- Automated testing in the cloud via BrowserStack / Saucelabs
- Unit Testing: Doesn't really matter which tool you use. I prefer mocha (bdd) + bondjs (probably because I wrote it) + simple assert methods (node assert is fine)
- Integration Testing: Zombie and CasperJS have too many issues to be usable. (now that's a bold statement. references? Blog post is coming. Keep an eye on the Groupon Engineering Blog.) PhantomJS is OK for faster dev (but also has too many issues to be comfortable), but you really have to run you tests in real browsers. Therefore, Selenium is mostly unavoidable. WebDriver is ok, but existing Node.js bindings aren't that great.
- QUnit for unit testing, using Grunt to run the QUnit tests through PhantomJS for running tests on continuous integration servers (like Jenkins and CircleCI). Also run QUnit tests on Saucelabs to test every browser that should work with my app/library (this is done automatically through the grunt-saucelabs plugin). Then, I use selenium for UI based testing.
- Automated testing of Web Application using SilkTest.
- Mink: Abstract test code from which web driver is used; Supports both headless and non-headless drivers. http://mink.behat.org
What are the things you would like to see in automated testing tools that do not exist yet?
- PhantomJS no headless-IE. We need a way to run tests on headless versions of all browser platformsJs & SlimerJS offer a way of running tests on headless browsers, but the Webkit version of PhantomJS is obsolete, and there
- Better interop with various IEs.
- Easier setup for real device testing +1
- CSS performance regression testing. yell at me when I commit something that regresses our render performance significantly. let me see a chart of key render perf statistics over project history. +2
- More headless browsers, and those ones as close as possible to real browsers.+1
- Mechanize http://mechanize.rubyforge.org/
- RSpec http://rspec.info/
- SlimerJS http://slimerjs.org/
- Selenium http://docs.seleniumhq.org/
- Webdriver Specification https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html
- Tyrtle https://github.com/soundcloud/tyrtle
- Myrtle https://github.com/spadgos/myrtle
- Karma-runner http://karma-runner.github.io/0.8/index.html
- WebDriver.js https://github.com/camme/webdriverjs
- Telemetry http://www.chromium.org/developers/telemetry
- Jenkins http://jenkins-ci.org/
- CSS Testing tools http://csste.st/tools/
- Mocha https://github.com/visionmedia/mocha
- Konacha https://github.com/jfirebaugh/konacha
- Watai https://github.com/MattiSG/Watai/
- Jasmine http://pivotal.github.io/jasmine/
Some anonymized comments from elsewhere
These are direct comments. It doesn't mean they represent a fact, but they are at least feelings, opinions, perceptions of what testing, tools are.
- "Running integration tests on headless browsers is not acceptable. You have to make your tests on all latest versions of Firefox, Chrome, Safari, Opera et IE"
- "From memory (a couple of years), selenium is very hard to install"
- "Easy to install selenium with pip install selenium. no need to download the jar for the server mode."
- "don't use Selenium™, thank me later".
- Why not? Selenium Webdriver has been much improved over the last two years. This is a great project.
- because http://ljouanneau.com/blog/post/2013/07/15/Tada-Here-is-SlimerJS#c7076 (you asked me):
- «"for me Selenium appears to be complicated to use and too slow."
- +1, that's pretty much why I've started casperjs actually.
- But I hear concerns about multiplatform support and standardization, though I'd personally think there would be room for a more-phantomjs-like API standardization effort, or at least a more "close to the metal" one than what the WebDriver spec tries to provide.»
- I don't see why a BDD setup using Jasmine or Mocha with a decent runner such as Testem or Karma isn't used much more for functional/UI testing of single-page applications in real browsers (see https://github.com/webpro/Automated-SPA-Testing#functional-testing or https://github.com/webpro/baseplate/blob/master/test/mocha-behavior/specs/moduleB.behavior.js for an example). The only real downside I can see so far is not being able to navigate across pages (which is usually not a big issue here).