How To: Getting a project Automated

Web QA has a large number of automated projects. Tests which are created and maintained by the Web QA team. This document will outline the process to get a project in the queue for Web QA automation.

Eligibility:

Automation can be a great way to increase coverage for a project. If you have a project, this document will outline the process of submitting it for consideration. It is important to note that not all projects are candidates for automation. There are certain qualities of a project, outlined below, that increase the likelihood of QA implementing it.

• Urgency: What is the urgency of your project? If your project is a high priority for the company, or on the QA goals list, then it will be more likely to get automated.
  • If it is a low priority, it may be some time before your project is eligible for automation.
  • If there is a short deadline for an automation project, this reduces the chance of it being automated fully or on schedule. Ideally projects are submitted for automation with plenty of leeway for deadlines. If your project has a very short and tight deadline, it may not be a good candidate for automation.
• Lifespan: Is your project permanent? Permanent projects are more likely to get automated.
  • Is it going to be a maintained project that will last a year or more? Long term projects are better automation candidates.
  • Or is this a short term project? Short term projects are not eliminated from automation, but the scope of what can or should be done is limited. The lifespan of a project does impact the eligibility of it for automation.
• UI Stability: Is the project in a stable phase, or is it new to development? More stable projects are better candidates for automation. A changing UI makes for a higher intensity of development to keep up with the evolution of the site. Revamping a site's UI may be a good time to create or edit automation also.
• Coverage Complexity: Are you requesting to have a short smoke test automated, or in-depth coverage of a full product? How much needs to be done, and how quickly? These are important elements for automation. Team resources need to be scoped out, as far in advance as is possible.
• Discussion: The bottom line is that there cannot be any cut-and-dry formula to determine if a project is a good candidate for automation. There is a lot that depends on the project, its urgency, and the resources the team has available. Web QA wants to be available as a resource, however it's just not possible to always say 'Yes', which is why there is a need for discussion. Even if it is not possible to do right now, there is a possibility that eligibility may change in the future. For example we initially said no to marketing projects, however since then we have created a short template for those projects that actually carved off time for manual testing. So please talk to us!

Submission: Submitting your project to the Web QA team involves submitting a bug to Bugzilla.

Let us know the scope of your project, and the timeline. The more information we have, the easier it is to work with your project. Also, the sooner you let us know a project will need Automation the better.

After contacting us, we will review the project. We will incorporate all of the considerations listed above and let you know the status as soon as possible, within 3 working days. This will include whether or not the project is a good candidate for automation, and also a time frame that we believe it could be done in.

If your project is not eligible for automation we will give you the reason[s]. This information will include what might be changed in order to be eligible, if that is a possibility.

What next?

After your project has been nominated for automation, and resources have been scheduled these are the next steps towards automation. There is no set division of labor for these tasks, and we invite every project team to get involved. Specifically we invite devs to help set up and maintain jobs for their projects in our CI. This is an opportunity to work more closely with QA, and to help your own project along!

We realize that these tasks may be new or unfamiliar to people outside of QA. Unfortunately we cannot provide exact set-up details as each project has unique needs and constraints. However, QA will be available to provide help. You may confer with the QA person[s] assigned to your project. Referring to previous projects and tests often give a good starting point for beginning automation. See the Reference section below to get more information.

• Project creation on GitHub: Mozilla Admins have this permission. If it is unclear who should be ask, request project submission from StephenD.
• Jenkins: Access is based on LDAP. Any team member can get access to view the Jenkins project dashboard. Currently Jenkins can be set to pull SCM (source code management), and given how many minutes for the interval. It is given the developer's GitHub repo and then goes to the dev site every 15-35 minutes to look for changes. Due to firewall issues, it is not possible at this time to do a pull when there is a change rather than a timed interval.
• Test creation: Project requirements, including test coverage requests will be discussed by the team. Resources will be based on availability and need.
• Test maintenance: Test maintenance can be done by QA or team members. This can be determined on a per-project basis.
• IRC notifications: IRC can be set up to notify the project team in any channel when the project status changes. For example, if it's been failing and suddenly passes it will generate a notification. Alternately it will notify the channel if tests which have been passing suddenly fail. You can check the status of any project by this IRC command, "qatestbot: status name.of.job [ex: qmo.prod]"
• QA Automation Reference:
  • Automation Home page that includes tool documentation and resource information.
  • Automation Style Guide
  • WebQA Team page with info on Automation, Python, Selenium, Getting started and more.
  • Test Templates on Github.