Phabricator/TestPlan

Getting Started

NOTE: We are using the dev server for QA test runs.

The following manual steps are required to get QA up and running for Phabricator testing on Mozilla's staging server.

Alternatively, you can try out the a preset environment using Docker from this Github repo. Once checked out run:

   docker-compose build
   docker-compose run qa su - phab

You will still need to perform step 8 using your own Conduit API keys.

Manual Setup

1. Clone the following repositories to the same directory on your machine:

   git clone https://github.com/phacility/libphutil $HOME/libphutil
   git clone https://github.com/phacility/arcanist $HOME/arcanist
   git clone https://github.com/mozilla-conduit/arcanist --branch master --single-branch $HOME/cinnabarc

2. Add vanilla arcanist ($HOME/arcanist/bin from your phacility repo checkout) to your path. On OS X and Linux (and the Windows 10 Linux Subsystem), you can add the following to your `~/.bash_profile`:

   export PATH="$PATH:$HOME/arcanist/bin"

Doing so will allow you to use the arc command from anywhere on your machine.

3. Install git-cinnabar:

   git clone https://github.com/glandium/git-cinnabar.git --branch master --single-branch
   export PATH="$PATH:$HOME//git-cinnabar:$PATH
   cd git-cinnabar && git cinnabar download

4. Alias Arcanist modified for git-cinnabar use to cinnabarc. This will allow to use both vanilla and modified Arcanist interchangeably. You can add the following to your `~/.bash_profile:

   alias cinnabarc="$HOME/cinnabarc/bin/arc"

5. Install the moz-phab arcanist wrapper used for submitting stacked revisions to Phabricator. You can see instructions on how to install from the Github repo. The moz-phab command needs to be in your PATH such as in $HOME/bin.

6. Reload your .bash_profile:

   source $HOME/.bash_profile

7. Clone the repo(s) you plan to test on:

• hg clone https://hg.mozilla.org/automation/phabricator-qa-dev/
• git clone hg::https://hg.mozilla.org/automation/phabricator-qa-dev/ phabricator-qa-dev-cinnabar

8. In the local phabricator-qa-dev repository checkout, run:

   arc install-certificate 

Follow the instructions presented in the command line to associate your local machine with the Phabricator instance via an API key.

Repeat the process in phabricator-qa-dev-cinnabar. Use cinnabarc instead:

   cinnabarc install-certificate

9. Test away! You can create branches within your local phabricator-qa-dev repository checkout, add commits, and send them to Phabricator via mozphab. See the Mozilla Phabricator User Documentation for more.

This should be all you need to get going with moz-phab, arc, and our staging and dev servers!

Test Plan

These tests will help ensure that our extensions continue to work as expected when Phabricator is upgraded. They are important as Phacility make no guarantees that internal modules/objects/APIs won't change. See the upgrade process for more.

Please update this doc with new tests as features are added/changed and as holes in the plan are discovered.

T1 - Signing up is successful

Test Plan

1. Create a new BMO account.
2. Make sure to include an irc nick in the Real Name field.
   • E.g. "John Doe [:jdoe]"
3. Go to Phabricator and login as the new user.
4. Finish account creation.

Results

1. After logging into Phabricator, on the "Create a New Account" page,
   1. Username should be prefilled with the irc nick,
   2. Real Name is correct and does not contain [] or any other extraneous characters.
2. Clicking register approval completes account creation without error.
3. The account works as expected.

T2 - Creating a revision is successful

Test Plan

1. Make sure you have moz-phab properly setup on your machine and have run arc install-certificate using the Phabricator user you wish to test with as documented above.
2. Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug).
   • To create bugs directly and bypass triaging, go to https://bugzilla.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (staging) or https://bugzilla-dev.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (dev)
3. Using the repo listed in the "Getting Started" section above that matches the environment you are testing, make some change to a file.
4. Run hg commit -A -m 'Bug <bugid>: New changes
5. Run moz-phab

Results

1. moz-phab only submitted the 1 newly created commit.
2. Visit Phabricator and there should be a new revision with the title.
3. The revision should contain the correct diff of the changes that were made.
4. The "Bugzilla Bug ID" is correct.
   1. The Bug number is correct.
   2. The link is correct based on the environment (bugzilla.allizom.org for staging, bugzilla-dev.allizom.org for dev).
   3. If the bug in Bugzilla is a public bug, the revision should also be public.
5. Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section.
6. Author received an email from mozphab-dev@mozilla.com. It has a title formatted as "[Differential] [Changed Policy] D{revision number}: {first line}." and a body with a text "phab-bot changed the visibility from "Administrators" to "Public (No Login Required)"."

T3 - Updating a revision with an amended commit is successful

Test Plan

1. hg update to a commit which you previously submitted with moz-phab
2. Make a change and run hg commit --amend
3. Run moz-phab

Results

1. moz-phab should automatically detect a revision and ask you if you want to update it.
2. The revision is updated with the new diff on Phabricator:
   1. The History tab should have two entries, Diff 1 and Diff 2.
   2. The Commits tab should have a single entry.
3. The bug id and other information remains unchanged.

T4 - Creating a secure revision is successful

Your Bugzilla user must belong to a security group, e.g. core-security.

Test Plan

1. Go to bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
2. Run hg commit -A -m 'Bug <bugid>: Private changes
3. Run moz-phab.

Results

1. The diff and information of the revision are as expected.
2. The revision has a "Custom Policy" attached to it.
3. Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
   • It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
4. The revision has a "secure-revision" project tag added.
5. The revision has a warning titled "This is a secure revision.".
6. The revision added the creator as a subscriber.
7. The revision is visible to the user who made it.
8. The revision is visible to users belonging to the security groups of the bug.
9. Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section.
10. The revision is NOT visible to the public without logging in.
11. The revision is NOT visible to logged in members without the correct permission.

T5 - Adding a secure bug to an existing revision locks it down

Your Bugzilla user must belong to a security group, e.g. core-security. You will also need a second, unprivileged user.

Test Plan

1. Go to bugzilla and create a public bug:
2. Run hg commit -A -m 'Bug <bugid>: Public changes
3. Run moz-phab.
4. Check if revision is available for public.
5. Go to bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
6. Edit the revision created above, and set the Bugzilla Bug ID field to the ID of the newly created bug.
7. Add the second, unprivileged Bugzilla user to the bug's CC list.

Results

1. The revision has a "Custom Policy" attached to it.
2. Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
   • It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
3. The revision has a "secure-revision" project tag added.
4. The revision has the creator and the second Bugzilla user as subscribers.
5. The revision is visible to the user who made it.
6. The revision is visible to users belonging to the security groups of the bug.
7. The revision is NOT visible to the public without logging in.
8. The revision is NOT visible to logged in members without the correct permission.
9. The revision IS visible to the second Bugzilla user.
10. Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section.
11. The reviewer received an email from the author with a text "The content for this message can only be transmitted over a secure channel." and a link to Phabricator.

T6 - Creating a revision checks the bug id

Test Plan

1. Enter the bug id as specified for each expected result below.
2. Run hg commit -A -m 'Bug <bugid>: Public changes
3. Run moz-phab.

Results

1. Entering an invalid bug id, e.g "abcd efg" or "$1000", fails.
2. Entering the id of a bug that does not exist fails.
3. Entering the id of a bug of a secure revision that the user does not have access to fails.
4. Entering a valid bug id is successful.

T7 - Creating multiple revisions with the same bug id is successful

Test Plan

1. Run hg commit -A -m 'Bug <bugid>: Public changes
2. Run moz-phab.
3. Create a different comment using same bug id but different title.

Result

1. Both revisions are created successfully.
2. Visiting the bug on Bugzilla shows 2 corresponding entries for the revisions in the Phabricator Revisions section.

T8 - Requesting and leaving a review on a revision is successful

Test Plan

1. Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
2. Create a commit, run moz-phab.
3. Input the title, summary, test plan, and bug id of a public bug.
4. For the "Reviewers" field enter the Phabricator username of the other account.
5. Log into Phabricator as the reviewer account.
6. Add the "Accept Revision" action at the bottom. Submit.
7. Add the "Request Changes" action at the bottom. Submit.

Results

1. The Phabricator attachment on Bugzilla is present.
2. Reviewer received an email from author with a text "{author} added a reviewer: {reviewer}"
3. Author received an email from reviewer with a text "{reviewer} accepted this revision."
4. Author received an email from reviewer with a text "{reviewer} requested changes to this revision."

T9 - Abandoning a revision obsoletes the attachment

Test Plan

1. Create a new bug.
2. Create a commit and run moz-phab.
3. On a previously created revision, perform the "Abandon Revision" action from the "Add Action" dropdown.

Results

1. The bug shows the revision as 'Abandoned' in the Phabricator Revisions list of the bug.
2. The bug history shows the attachment is being set to obsolete.

T10 - Reclaiming a revision unobsoletes the attachment

Test Plan

1. In the revision created as part of the "Abandoning a revision obsoletes the attachment" test, perform the "Reclaim revision" action.

Results

1. The bug's Phabricator attachment is unobsoleted and the revisions is no abandoned in the Phabricator revisions list.

T11 - Creating a Diff from local git repository to remote Mercurial repository is not allowed

Test Plan

1. Change directory to the repository phabricator-qa-dev-cinnabar.
2. Create a commit with git and run arc diff.
3. Fill the needed data (Test Plan) and confirm.

Results

1. Creating a diff fails with :

   ERR-CONDUIT-CORE: Local VCS (git) is different from the remote repository VCS (hg).

T12 - Patching Diff created from git repository

Test Plan

1. Using a commit created above run cinnabarc diff HEAD~.
2. Fill the needed data (Test Plan) and confirm.
3. Change directory to phabricator-qa-dev.
4. Run moz-phab patch D{number of the revision created above}.

Results

1. Code is sucessfully patched using the Diff.

T13 - Verify the private revisions deliver emails that does not contain any sensitive content

Your Bugzilla user must belong to a security group, e.g. core-security.

Test Plan

1. Login to Phabricator (after creating account in Bugzilla) using an account that can have email delivered to it, such as your own email address.
2. At the top right of Phabricator, click on your initial or gravatar image to open the drop down menu and select "Settings".
3. Click on "Email Delivery".
4. Select "Enable Self Action Mail" for the "Self Action" drop down.
5. Click "Save Changes".
6. Go to Bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
7. Create a new hg commit.
8. Run moz-phab.

Results

1. The diff and information of the revision are as expected.
2. The revision has a "Custom Policy" attached to it.
3. The revision has a "secure-revision" project tag added.
4. The revision has a warning titled "This is a secure revision".
5. Check to see if you received an email about the new object (Revision) that was just created.
6. The email should not contain any information about the revision other than a link to Phabricator.
7. Clicking on the link in the email should take you to the Phabricator page that displays the full unfiltered email contents.
8. The email contents should contain the title, summary, test plan, reviewers, etc. of the new revision.
9. Submitting a public revision should instead show the full contents in the email similar to what was displayed on the Phabricator mail page.