MozillaWiki

Firefox/Python 3 Migration: Difference between revisions

Page Discussion

Firefox/Python 3 Migration (view source)

Revision as of 03:23, 24 December 2019

5,333 bytes added ,  24 December 2019
m
→‎Port a Mach Command
(→‎Getting Involved: simplify instructions)
 
(11 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
== Why This is Important ==
== Why This is Important ==


In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]
In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]


Python 2 will reach it's End Of Life on January 1st 2020. This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
Python 2 will reach it's End Of Life on '''[https://www.python.org/doc/sunset-python-2/ January 1st 2020]''' (here's [https://pythonclock.org/ a countdown!]).
 
== What Will Happen If We Don't Upgrade? ==
 
The [https://www.python.org/doc/sunset-python-2/ PSF said it best]:
 
<blockquote style="background-color: #F0F0FF; padding: 1em">"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.
 
Some of these problems will start on January 1. Other problems will grow over time."
</blockquote>


== How We Get There ==
== How We Get There ==
Line 21: Line 32:
=== Find the others ===
=== Find the others ===


Most discussion happens in the #python channel on Slack (NDA'd Mozillians only).
Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.


=== What you'll need ===
=== What you'll need ===
Line 29: Line 40:
=== Where to start: ===
=== Where to start: ===


==== Help port mach-core to Python 3 ([https://bugzilla.mozilla.org/show_bug.cgi?id=1473498 bug 1473498]) ====
==== Remove more of the "excluded" paths from the py2 and py3 linters ====
 
See [https://pad.mozilla.org/p/py3 this Etherpad] for instructions. (Etherpad is blank, needs to be fixed)
 
Roughly:
 
* Pick a directory [https://pad.mozilla.org/p/py3 from the list], and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
* Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
* Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
* To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
* To install `futurize`:
pip install --user future
* To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
* Visually inspect the patch, especially around `except:` clauses.  Watch out for "xxx_todo_changeme" in the patch.  These are places where futurize couldn't port the code and you need to fix it manually.
 
When your patch is ready file:
 
# Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
# Use [https://bugzilla.mozilla.org/show_bug.cgi?id=1559975 bug 1559975]
# Set one of callek, ahal, or catlee as the code reviewer
 
==== Enable Python 3 on more Python unittests ====


# Edit python/mach/mach/test/python.ini to remove the skip-if
Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:
# Run: ./mach python-test --python 3 python/mach
# Fix the errors until all tests pass


==== Remove more of the "excluded" paths from the py2 and py3 linters ====
# Find a test that is skipped with Python 3, you can use [https://searchfox.org/mozilla-central/search?q=.*(skip|fail)-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=.ini%24 this query] to find these tests.
# Edit the manifest and remove the Python 3 skip-if annotation.
# Run the test with `./mach python-test --python 3 path/to/test`.
# Fix any errors and repeat the previous step.
# Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in [https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml python.yml].
# Push the task to try on all platforms it is configured for.
 
==== Port a Mach Command ====


Remove a line from https://searchfox.org/mozilla-central/source/tools/lint/py3.yml or https://searchfox.org/mozilla-central/source/tools/lint/py2.yml. Run the linter, see if the module passes.
The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2.  


==== Run all python source tests with Python 3 and see what breaks ====


Make sure each of the tasks listed in https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml has Python 3 specified, and do a try push, enabling all the python source tests.
The whitelist contained in the mach command is py2commands


See what breaks, then fix it!
py2commands="
    addtest
    analyze
    android
    android-emulator
    artifact
    awsy-test
    bootstrap..."


See the [[#References]] for links to other ideas and tracking bugs.
Pick the command that you'd like to convert and then:
# Remove it from the whitelist
# Run: ./mach <command>
# See what breaks and fix it!
# Repeat with a variety of different flags and arguments
# Make sure any Python unittests running in CI have Python 3 enabled (see above)


== How to Port the Code ==
== How to Port the Code ==
Line 62: Line 112:
* The `six` module is available in-tree, feel free to use it as a dependency.
* The `six` module is available in-tree, feel free to use it as a dependency.
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).
=== Porting examples ===
Prefer <code>[https://six.readthedocs.io/ six]</code> when possible.  Make use of <code>six.PY2</code>, <code>six.PY3</code>, <code>six.moves</code>, <code>six.string_types</code>, <code>six.integer_types</code>, etc.  This make it easy to find the py2 code again so we can remove it.
==== <code>import</code> statements ====
Use the <code>[https://six.readthedocs.io/#module-six.moves six.moves]</code> module:
Before:
<pre>
import __builtin__
__builtin__.__import__ = myimporthook()
</pre>
After:
<pre>
from six.moves import builtins
builtins.__import__ = myimporthook()
</pre>
==== <code>if</code> statements ====
Use <code>six.PY2</code> and <code>six.PY3</code>:
<pre>
import six
if six.PY2:
    # py2 stuff
else:
    # py3 stuff
</pre>
If six isn't available or you are in the imports section of the module then this is fine too:
<pre>
if sys.version_info < (3,):
  # py2 stuff
else:
  # py3 stuff
</pre>
==== String data ====
Use <code>six.string_types</code>, <code>six.text_type</code>, <code>six.binary_type</code>, etc. Also see the <code>six</code> docs for "[https://six.readthedocs.io/#binary-and-text-data Binary and text data]":
Before:
<pre>
# using unicode
def write(self, buf):
    if isinstance(buf, unicode):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)
# using types.StringTypes
if isinstance(value, types.StringTypes):
    ...
</pre>
After:
<pre>
import six
# fixed unicode
def write(self, buf):
    if isinstance(buf, six.text_type):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)
# fixed types.StringTypes
if isinstance(value, six.string_types):
    ...
</pre>
==== <code>subprocess.check_output()</code> ====
Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>.  The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream.  However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3.
Before:
<pre>
output = subprocess.check_output([callargs])
lines = output.splitlines()  # Broken in Python 3
</pre>
After:
<pre>
output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()
</pre>
See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details.


=== Porting Resources ===
=== Porting Resources ===
Kmoir
Confirmed users
1,989

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1213273...1221627"