CIDuty/How To/PyUp Dependency PR Updates

Overview

PyUp is an automated service, that at it's most basic level, helps developers keep their Python Dependencies always Up-To-Date. CiDuty will be helping RelOps/RelEng with this weekly actionable (every Wednesday) and making sure that the PR is all green, before merging. This is an easy process that requires attention and knowledge on how pip is working.

The 4 pillars of the Task

1. Review and Fix possible conflicts in the PR.
2. Make a list of critical dependencies and/or servers that will be affected.
3. Communicate with the repository maintainer(s)
4. Merge the PyUp PR and Monitor incoming Puppet mails for issues.

Important notes to take into account

While working on a PyUp PR you always have to take notes and/or ask yourself the following questions:

   - Did any signing-server (NOT scriptworkers!) requirements file(s) got changed?
       = Yes? ANNOUNCE bhearsum because the signing servers might need to be restarted!
       = We don't do merges till Ben gives the green light.
   - Did any important deps. got updated?
       = Example: Crypthograpy, certifi, anything to do with OAuth, etc. 
   - Did any MAC machines got requirements file(s) changed?
       = Yes? ANNOUNCE bhearsum because the deps might need to be manually downloaded!
   - Did any deps upgrades ask for lower version packages?
       = Dependiong on which repo, you will have to talk with jlorenzo or bhearsum and see which packages get pinned (locked version)
   - When everything is ready to be merged, ALWAYS "Squash and Merge".

Repositories which we need to activate on

1. Mozilla Version
2. Build Puppet

How To: Mozilla-Version

We always start with this repository first because the repository itself is a dependency of Build-Puppet!

1. Head to https://github.com/mozilla-releng/mozilla-version/pulls and check if any "Scheduled weekly dependency update for week X" is currently open.
2. Verify if the tests are green. Keep in mind that all three checks need to be green:
   1. Travis CI PR
   2. Travis CI Push
   3. Coverage/Coveralls
3. If tests are RED head to the build/test and look in the logs for problems.
4. Implement the needed changes + commit them (you can do it via github webui). Wait for the test results to come (about 10-15 minutes)
5. Squash and Merge the PR.

Common Issues

This repository usually has 2 distinct problems which will stop you from merging the PR:

1. Requirements of a dependency. Example: flake8 3.5.0 has requirement pycodestyle<2.4.0,>=2.0.0, but you have pycodestyle 2.4.0.

With this issue, you have to keep which deps you have to either manually revert back at the older version (keep in mind you need the old sha512!) or if you pin the version of the dependency.

1. Wrong Sha

When a error like this comes:

ERROR: could not install deps [-rrequirements-test.txt]; v = InvocationError('/home/travis/build/mozilla-releng/mozilla-version/.tox/py27/bin/python -m pip install -rrequirements-test.txt (see /home/travis/build/mozilla-releng/mozilla-version/.tox/py27/log/py27-1.log)', 1)

it means the hash=sha512 is wrong. Please contact JLorenzo for a fix/sha string. A quick fix would be cloning the repository, take the old version of that package (copy it from the cloned repository) and use it to revert the changes by reverting the package.

How To: Build-Puppet

After you are done with Mozilla-Version, its time for Build-Puppet!

1. Head to https://github.com/mozilla-releng/build-puppet/pulls and check if any "Scheduled weekly dependency update for week X" is currently open.
2. Verify if the tests are green. Keep in mind that all three checks need to be green:
   1. Travis CI PR
   2. Travis CI Push
   3. TaskCluster (pull_request)
3. If tests are RED head to the build/test and look in the logs for problems. In taskcluster, usually the errors are at the bottom.
4. Implement the needed changes + commit them (you can do it via github webui). Wait for the test results to come (about 40-60 minutes)
5. Squash and Merge the PR.
6. Announce Bhearsum that the weekly PR is complete.

Common Issues

1.Requirements of a dependency. Example: flake8 3.5.0 has requirement pycodestyle<2.4.0,>=2.0.0, but you have pycodestyle 2.4.0. With this issue, you have to keep which deps you have to either manually revert back at the older version (keep in mind you need the old sha512!) or if you pin the version of the dependency.

2.Another common issue regarding package versions, is that the bot will try and update a package several times if it's not pinned, thus braking things every week if that package cannot be updated because has another unmet dependency . In order to correct that we need to pin it, and that can be done providing a "\" after the hash of the package, example. This situation can appear on both repositories and would be a good thing to ask around what can be pinned and what not.

See also : Update Python dependencies