Firefox/Python 3 Migration
This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
- 1 Why This is Important
- 2 What Will Happen If We Don't Upgrade?
- 3 How We Get There
- 4 Getting Involved
- 5 How to Port the Code
- 6 References
Why This is Important
In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.
What Will Happen If We Don't Upgrade?
The PSF said it best:
"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.
Some of these problems will start on January 1. Other problems will grow over time."
How We Get There
As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.
- Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
- Make key support libraries run in both Python 2 and Python 3. (mach-core)
- Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
- Port Python 2-only components to Python 3 piecemeal.
- Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)
Find the others
Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.
What you'll need
You will need a working Firefox build environment.
Where to start:
Remove more of the "excluded" paths from the py2 and py3 linters
See this Etherpad for instructions. (Etherpad is blank, needs to be fixed)
- Pick a directory from the list, and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
- Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
- Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
- To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
- To install `futurize`:
pip install --user future
- To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
- Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.
When your patch is ready file:
- Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
- Use bug 1559975
- Set one of callek, ahal, or catlee as the code reviewer
Enable Python 3 on more Python unittests
Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:
- Find a test that is skipped with Python 3, you can use this query to find these tests.
- Edit the manifest and remove the Python 3 skip-if annotation.
- Run the test with `./mach python-test --python 3 path/to/test`.
- Fix any errors and repeat the previous step.
- Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in python.yml.
- Push the task to try on all platforms it is configured for.
Port a Mach Command
The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2.
The whitelist contained in the mach command is py2commands
py2commands=" addtest analyze android android-emulator artifact awsy-test bootstrap..."
Pick the command that you'd like to convert and then:
- Remove it from the whitelist
- Run: ./mach <command>
- See what breaks and fix it!
- Repeat with a variety of different flags and arguments
- Make sure any Python unittests running in CI have Python 3 enabled (see above)
How to Port the Code
If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.
Porting modules in the Firefox source tree
- We've already standardized on Python 2.7 across the tree.
- Make sure you can test the code you're working on with python3!
- If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
- If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
- `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
- The `six` module is available in-tree, feel free to use it as a dependency.
- Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).
six when possible. Make use of
six.integer_types, etc. This make it easy to find the py2 code again so we can remove it.
import __builtin__ __builtin__.__import__ = myimporthook()
from six.moves import builtins builtins.__import__ = myimporthook()
import six if six.PY2: # py2 stuff else: # py3 stuff
If six isn't available or you are in the imports section of the module then this is fine too:
if sys.version_info < (3,): # py2 stuff else: # py3 stuff
six.binary_type, etc. Also see the
six docs for "Binary and text data":
# using unicode def write(self, buf): if isinstance(buf, unicode): buf = buf.encode('utf-8') BytesIO.write(self, buf) # using types.StringTypes if isinstance(value, types.StringTypes): ...
import six # fixed unicode def write(self, buf): if isinstance(buf, six.text_type): buf = buf.encode('utf-8') BytesIO.write(self, buf) # fixed types.StringTypes if isinstance(value, six.string_types): ...
Python 2.7 code will often call
subprocess.check_output() and then do string manipulation on the output, such as
str.splitlines(). The code will fail under Python 3 because
subprocess.check_output() now returns a binary data stream. However we can use the
universal_newlines=True keyword argument to get back a text type object in both Python 2 and Python 3.
output = subprocess.check_output([callargs]) lines = output.splitlines() # Broken in Python 3
output = subprocess.check_output([callargs], universal_newlines=True) lines = output.splitlines()
See the subprocess documentation for details.
Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.
Bug 1496527 tracks the migration