Phabricator/TestPlan

Getting Started

The following steps are required to get QA up and running for Phabricator testing on Mozilla's staging server.

NOTE: At this time we are using the dev server for QA test runs rather than staging, as the latter is on the VPN, which has been causing some difficulties.

1. Clone the following repositories to the same directory on your machine:

• https://github.com/phacility/libphutil
• https://github.com/phacility/arcanist

2. Add arc to your path. On OS X and Linux (and the Windows 10 Linux Subsystem), you can add the following to your `~/.bash_profile` (modifying the path appropriately):

   alias arc='/Users/youruser/path/to/arcanist/bin/arc'

Doing so will allow you to use the arc command from anywhere on your machine.

3. Clone the repo(s) you plan to test on:

• Staging: https://hg.mozilla.org/automation/phabricator-qa-stage/
• Dev: https://hg.mozilla.org/automation/phabricator-qa-dev/

4. In the local repository checkout, run:

   arc install-certificate 

Follow the instructions presented in the command line to associate your local machine with the Phabricator instance via an API key.

5. Test away! You can create branches within your local repository checkout, add commits, and send them to Phabricator via arc diff HEAD^. See the Mozilla Phabricator User Documentation for more.

This should be all you need to get going with arc and our staging and dev servers!

Test Plan

These tests will help ensure that our extensions continue to work as expected when Phabricator is upgraded. They are important as Phacility make no guarantees that internal modules/objects/APIs won't change. See the upgrade process for more.

Please update this doc with new tests as features are added/changed and as holes in the plan are discovered.

Signing up is successful

Test Plan

• Create a new BMO account.
• Make sure to include an irc nick in the Real Name field.
  • E.g. "John Doe [:jdoe]"
• Go to Phabricator and login as the new user.
• Finish account creation.

Results

• After logging into Phabricator, on the "Create a New Account" page,
  • Username should be prefilled with the irc nick,
  • Real Name is correct and does not contain [] or any other extraneous characters.
• Clicking register approval completes account creation without error.
• The account works as expected.

[EDGE CASE] Currently when you revoke the "MozPhab" API key on BMO, there is no way to create a new one for Phabricator to use.

Creating a revision is successful

Test Plan

• Make sure you have arc properly setup on your machine and have run arc install-certificate using the Phabricator user you wish to test with as documented above.
• Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug).
  • To create bugs directly and bypass triaging, go to:

   http://bmo.test/enter_bug.cgi?product=Firefox&format=__default__

• Using the repo listed in the "Getting Started" section above that matches the environment you are testing, make some change to a file.
• Run hg commit -A -m 'commit1: New changes
• Run arc diff .^
• Input something for the title, summary, test plan, and the correct bug id.
• Save and exit the file.

Results

• arc diff only submitted the 1 newly created commit.
• Visit Phabricator and there should be a new revision with the title.
• The revision should contain the correct diff of the changes that were made.
• The "Bugzilla Bug ID" is correct.
  • The Bug number is correct.
  • The link is correct based on the environment (bugzilla.allizom.org for staging, bugzilla-dev.allizom.org for dev).
  • Note: If the bug id is short (like 3, 92, etc, which is common on local test environments) the link will not work but that is fine because it only works on very long bug ids which match what new bugs on production bmo will be using.
• Visiting the bug on bugzilla shows an x-phabricator-request attachment.

Updating a revision is successful

Test Plan

• `hg update` to a commit which you previously submitted with arc
• Make a change and `hg commit`

 - If you make a new commit run `arc diff .^^`
 - If you amend the existing commit run `arc diff .^`

Results

• arc should automatically detect a revision and ask you if you want to update it.
• The revision is updated with the new diff on Phabricator.
• The bug id and other information remains unchanged.

Creating a secure revision is successful

Test Plan

• Go to bugzilla and create a security bug.
• Create a new hg commit.
• Run `arc diff .^`.
• Enter the title, summary, test plan, and the bug id of the security bug.
• Submit the revision.

Results

• The diff and information of the revision are as expected.
• The revision has a "Custom Policy" attached to it.
• The revision added the creator as a subscriber.
• The revision is visible to the user who made it.
• The revision is visible to users belonging to the security groups of the bug.
• The revision is NOT visible to the public without logging in.
• The revision is NOT visible to logged in members without the correct permission.
• There is an x-phabricator-request attachment on the bug in Bugzilla.

Creating a revision checks the bug id

Test Plan

• Create a new hg commit.
• Run `arc diff .^`
• Enter information for the title, summary, and test plan.
• Enter the bug id, repeat for each expected result.

• • Result**
• Entering no bug id fails.
• Entering an invalid bug id like "abcd efg", or "$1000", fails.
• Entering the id of a bug that does not exist fails.
• Entering the id of a bug of a secure revision that the user does not have

 access to fails.

• Entering a valid bug id is successful.

Creating multiple revisions with the same bug id is successful

Test Plan

• Create a new hg commit.
• Run `arc diff .^`
• Enter information for the title, summary, and test plan.
• Enter the bug id.
• Repeat and create another revision with the same bug id.

• • Result**
• Both revisions are created successfully.
• There are 2 corresponding x-phabricator-request attachments on the bug in bugzilla.- Entering no bug id fails.

Requesting a reviewer on a revision is successful

Test Plan

• Ensure that you have 2 phabricator accounts that login via BMO ready to go.

 - If not, create a new bugzilla user and the phabricator account for it.

• Create a commit, run arc diff.
• Input the title, summary, test plan, and bug id of a public bug.
• For the reviewers field enter the Phabricator user name of the other account.

Results

• The revision is created as normal.
• The phabricator attachment on bugzilla is present.
• Phabricator shows the reviewer on the Revision.

Review Status is propagated to Bugzilla

Test Plan

• Login to Phabricator as the Reviewer on a Revision
• Accept the Revision

Results

• The attachment on the bug in Bugzilla should have an r+ flag from the Reviewer.

Please contact a developer within the #conduit IRC channel with further questions.