Firefox/Python 3 Migration
- 1 Why This is Important
- 2 How We Get There
- 3 Getting Involved
- 4 How to Port the Code
- 5 References
Why This is Important
In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.
Python 2 will reach it's End Of Life on January 1st 2020. This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
How We Get There
As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.
- Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
- Make key support libraries run in both Python 2 and Python 3. (mach-core)
- Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
- Port Python 2-only components to Python 3 piecemeal.
- Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)
Find the others
Most discussion happens in the #python channel on Slack (NDA'd Mozillians only).
What you'll need
You will need a working Firefox build environment.
Where to start:
Remove more of the "excluded" paths from the py2 and py3 linters
See this Etherpad for instructions.
- Pick a directory from the list, and put your name beside it so we know it's ported
- Edit py2.yml and/or py3.yml and remove that directory from the exclusion list
- Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
- To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
- To install `futurize`:
pip install --user future
- To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
- Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.
When your patch is ready file:
- Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
- Use bug 1559975
- Set one of callek, ahal, or catlee as the code reviewer
Run all python source tests with Python 3 and see what breaks
Make sure each of the tasks listed in https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml has Python 3 specified, and do a try push, enabling all the python source tests.
See what breaks, then fix it!
Help port mach-core to Python 3 (bug 1473498)
- Edit python/mach/mach/test/python.ini to remove the skip-if
- Run: ./mach python-test --python 3 python/mach
- Fix the errors until all tests pass
See the #References for links to other ideas and tracking bugs.
How to Port the Code
If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.
Porting modules in the Firefox source tree
- We've already standardized on Python 2.7 across the tree.
- Make sure you can test the code you're working on with python3!
- If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
- If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
- `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
- The `six` module is available in-tree, feel free to use it as a dependency.
- Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).
six when possible. Make use of
six.integer_types, etc. This make it easy to find the py2 code again so we can remove it.
import __builtin__ __builtin__.__import__ = myimporthook()
from six.moves import builtins builtins.__import__ = myimporthook()
import six if six.PY2: # py2 stuff else: # py3 stuff
If six isn't available or you are in the imports section of the module then this is fine too:
if sys.version_info < (3,): # py2 stuff else: # py3 stuff
six.binary_type, etc. Also see the
six docs for "Binary and text data":
# using unicode def write(self, buf): if isinstance(buf, unicode): buf = buf.encode('utf-8') BytesIO.write(self, buf) # using types.StringTypes if isinstance(value, types.StringTypes): ...
import six # fixed unicode def write(self, buf): if isinstance(buf, six.text_type): buf = buf.encode('utf-8') BytesIO.write(self, buf) # fixed types.StringTypes if isinstance(value, six.string_types): ...
Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.
Bug 1496527 tracks the migration