Services/Sync/Getting Started: Difference between revisions

From MozillaWiki
< Services‎ | Sync
Jump to navigation Jump to search
(replaced Source Code section links with direct paths to searchfox)
 
(15 intermediate revisions by 6 users not shown)
Line 3: Line 3:
== Source Code ==
== Source Code ==


Once you have a copy of the Mozilla source tree (read below for how to obtain a copy), you can find the core Sync source code in:
Once you have a copy of the Mozilla source tree (read below for how to obtain a copy), you can find the core Sync source code in https://searchfox.org/firefox-main/source/services/sync


/services/sync/
This is just the core JavaScript APIs and logic. Additional Sync code for integration with Firefox lives in https://searchfox.org/firefox-main/source/browser/base/content


You can view this online at https://hg.mozilla.org/services/services-central/file/tip/services/sync
Look for files with ''sync'' in their name. Files here define the integration between Firefox and Sync. This includes all the Sync UI.
 
This is just the core JavaScript APIs and logic. Additional Sync code for integration with Firefox lives in:
 
/browser/base/content


Look for files with ''sync'' in their name. Files here define the integration between Firefox and Sync. This includes all the Sync UI.
Additionally, Firefox Accounts specific code lives in https://searchfox.org/firefox-main/source/services/fxaccounts


== Debugging Sync ==
== Debugging Sync ==
Line 20: Line 16:


  # Enable logging to Error Console
  # Enable logging to Error Console
  services.sync.log.appender.console Debug
  services.sync.log.appender.console Trace


  # Enable logging to STDOUT
  # Enable logging to STDOUT
  services.sync.log.appender.dump Debug
  services.sync.log.appender.dump Trace


  # Log full sync logs on sync success (typically they are only saved on failure)
  # Log full sync logs on sync success (typically they are only saved on failure)
  services.sync.log.appender.file.logOnSuccess true
  services.sync.log.appender.file.logOnSuccess true
# Enable trace logging for engines
services.sync.log.logger.engine Trace
# (optional) Enable trace logging for all of sync (this is very verbose and will include all request bodies)
services.sync.log.logger Trace


Once you've updated those settings, restart Firefox for them to take effect.
Once you've updated those settings, restart Firefox for them to take effect.
Line 34: Line 36:
  export MOZ_QUIET=1
  export MOZ_QUIET=1


You can view full sync logs by opening '''about:sync-log''' in Firefox.
You can view full sync logs by opening '''about:sync-log''' in Firefox. Read more about [[Firefox Sync extended logging]] for fetching successful sync event logfiles.


== Building Sync ==
== Building Sync ==


The party line to rebuild the tree after making changes is to run:
You probably should build using an artifact build, as described [https://firefox-source-docs.mozilla.org/contributing/contribution_quickref.html#bootstrap-a-copy-of-the-firefox-source-code here].


$ make -f client.mk
After setting this up, you'll do `mach build` to perform a build, `mach run` to run Firefox, and `mach xpcshell-test /services/sync/tests/unit` to run the sync unit test suite. There's also a separate end-to-end test suite, TPS, which is documented [https://firefox-source-docs.mozilla.org/testing/tps/index.html here]. (For most changes it's sufficient that you run the unit tests.)
or from the object directory:
$ make -sj8


However, this is slow. If you have simply changed a JS file in ''services/sync/'', it is sufficient to run the following from your object directory:
Depending on your OS, most changes will not require a rebuild to take effect, only restarting and rerunning via `mach run`.
 
$ make -C services/sync
 
The xpcshell tests will pick up changes immediately. However, if you launch Firefox, your changes '''won't''' be reflected. For that, you'll also need to rebuild the browser component:
 
$ make -C services/sync && make -C browser
 
A handy alias to have around is:
 
$ alias sync-launch make -C services/sync && make -C browser && dist/bin/firefox -no-remote -ProfileManager
 
Or, on OS X:
 
$ alias sync-launch make -C services/sync && make -C browser && dist/NightlyDebug.app/Contents/MacOS/ -no-remote -ProfileManager


= Using Bugzilla =
= Using Bugzilla =


Every code change starts out as a bug in [https://bugzilla.mozilla.org Bugzilla] where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see [[Services/Code_Review|code review policy]] for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into [https://hg.mozilla.org/services/services-central services-central], which is the Mercurial repository used for Services client work.
Every code change starts out as a bug in [https://bugzilla.mozilla.org Bugzilla] where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see [[Services/Process/Code_Review|code review policy]] for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into [https://github.com/mozilla-firefox/firefox mozilla-central].
 
On rare occasions it will make more sense to land work on [http://hg.mozilla.org/mozilla-central mozilla-central] instead — for example, if a change needs to propagate upstream before the regular [[Services/Process/MergingBetweenBranches|merge from services-central to mozilla-central]]. This might involve additional procedures (e.g., you need to request approval in addition to review.)
 
For server code written in Python, Tarek has provided a useful tool called [https://bitbucket.org/tarek/flake8 flake8] which does Python style checks against your code.  


You probably also want to get up to speed on [http://bugzilla.mozilla.org Bugzilla] if you're not already.
You probably also want to get up to speed on [http://bugzilla.mozilla.org Bugzilla] if you're not already.
* A good start is to go watch [http://vimeo.com/9205730 Jonath's "Bugzilla for Humans" video].
* A good start is to go watch [http://vimeo.com/9205730 Jonath's "Bugzilla for Humans" video].
* Our bugs are currently under "Other Products", "Mozilla Services" or just bookmark "[https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services Add a bug]" and "[https://bugzilla.mozilla.org/describecomponents.cgi?product=Mozilla%20Services Browse bugs]".
* Sync bugs are found in different places depending on which part of the system they affect. For the service side (syncstorage or tokenserver), bugs will be in [https://github.com/mozilla-services/syncstorage-rs|GitHub]. For the client side, you can find a list of components [CloudServices/Sync#Report_Bugs here].
* Don't forget to add a Bugzilla ID shortcut string to your Bugzilla [https://bugzilla.mozilla.org/userprefs.cgi?tab=account account preferences]. The current standard practice is to append "<whitespace>[:irc_nick]" to your Real Name field.
* Don't forget to add a Bugzilla ID shortcut string to your Bugzilla [https://bugzilla.mozilla.org/userprefs.cgi?tab=account account preferences]. The current standard practice is to append "<whitespace>[:irc_nick]" to your Real Name field.
* You also want to watch the components relevant to your work so you automatically get emails for all bug changes in those components. The easiest way to do this is to add the default QA contact for each component to the list of users your are watching in Bugzilla
* You also want to watch the components relevant to your work so you automatically get emails for all bug changes in those components. The expected components where bugs will be filed are listed on [[CloudServices/Sync#Report_Bugs]].
** Open the [https://bugzilla.mozilla.org/describecomponents.cgi?product=Mozilla%20Services list of Services components]
* A useful Firefox add-on that makes life in Bugzilla a bit easier is [https://addons.mozilla.org/en-US/firefox/addon/bugzillajs/ Bugzilla Tweaks]. Highly recommended!
** In another tab go to your Bugzilla [https://bugzilla.mozilla.org/userprefs.cgi?tab=email email preferences]
** At the bottom of the page under the <b>User Watching</b> section there is an input form that takes a comma separated list of email addresses
** If you're working on server side components, you will want to add at least:
*** core-server@services.bugs
*** sync-server@services.bugs
* A useful Firefox add-on that makes life in Bugzilla a bit easier is [https://addons.mozilla.org/en-Us/firefox/addon/bugzilla-tweaks/ Bugzilla Tweaks]. Highly recommended!




<!--
= Using Mercurial =
= Using Mercurial =
Note: This section may be outdated.


Mozilla's development process is very much patch driven. [https://developer.mozilla.org/En/Developer_Guide/Source_Code Mozilla uses Mercurial] (as well as git for some projects) as the version control system. [https://developer.mozilla.org/en/Mercurial_queues Mercurial Patch Queues], although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.
Mozilla's development process is very much patch driven. [https://developer.mozilla.org/En/Developer_Guide/Source_Code Mozilla uses Mercurial] (as well as git for some projects) as the version control system. [https://developer.mozilla.org/en/Mercurial_queues Mercurial Patch Queues], although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.
You can find the Services team's code repository at [http://hg.mozilla.org/services].


== Configuration ==
== Configuration ==
Line 126: Line 103:
== Landing patches ==
== Landing patches ==


You have items in a patch queue that you'd like to push to s-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course).
You have items in a patch queue that you'd like to push to m-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course). It's recommended that you use autoland via mozreview, or flag bugs as having checkin-needed instead.  


=== Make sure services-central is open and green ===
=== Make sure mozilla-central is open and green ===


Check [https://tbpl.mozilla.org/?tree=Services-Central TBPL].
Check [https://treeherder.mozilla.org treeherder].


=== Make sure your tree is up to date ===
=== Make sure your tree is up to date ===
Line 168: Line 145:
=== Run tests ===
=== Run tests ===


Now is a great time to run tests, because this is the state in which services-central will be after you push. If you apply a <tt>tests</tt> patch to re-enable disabled tests, ''make sure you pop it when you're done''.
Now is a great time to run tests, because this is the state in which mozilla-central will be after you push. If you apply a <tt>tests</tt> patch to re-enable disabled tests, ''make sure you pop it when you're done''.


=== Finish the applied patches ===
=== Finish the applied patches ===
Line 185: Line 162:


   hg push
   hg push
-->


= Using Git =
= Using Git =


Various projects exist as Git repositories instead of Mercurial repositories. Even when projects are officially hosted in Mercurial, Git mirrors are maintained.
Various projects exist as Git repositories.
 
The Services team maintains a GitHub organization at https://github.com/mozilla-services. A mirror of services-central can be found at https://github.com/mozilla-services/services-central.


== Configuring Git ==
== Configuring Git ==
Line 199: Line 175:
  $ git config --global user.name "John Doe"
  $ git config --global user.name "John Doe"
  $ git config --global user.email "johndoe@mozilla.com"
  $ git config --global user.email "johndoe@mozilla.com"
 
  # Enable color output for all commands
  # Enable color output for all commands
  $ git config --global color.branch auto
  $ git config --global color.branch auto
Line 207: Line 183:
  $ git config --global color.status auto
  $ git config --global color.status auto
  $ git config --global color.ui auto
  $ git config --global color.ui auto
 
  # Highlight whitespace
  # Highlight whitespace
  $ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"
  $ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"
 
  # Define your editor for commits, other tools
  # Define your editor for commits, other tools
  $ git config --global core.editor vim
  $ git config --global core.editor vim


If you find you have a slow git prompt, you may find the instructions [https://gist.github.com/thomcc/4c23eeed3b248394bffa0bff04f5c00b here] beneficial as well.
<!--
= Repositories =
= Repositories =


Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is '''bundles'''. [https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29 Follow the directions]. The services-central bundle is [http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/services-central.hg here].
Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is '''bundles'''. [https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29 Follow the directions]. More information on bundles is available [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial/Bundles here].
 
== services-central ==
 
[https://hg.mozilla.org/services/services-central services-central] is the clone of mozilla-central in which the Sync client (and other stable Services client-side projects) are developed.
 
Things that live there:
 
* The Services client libraries in services/crypto/ and services/sync.
 
* Built-in UI in browser/base/content/.
 
== fx-sync ==
 
Sync was primarily developed in the [https://hg.mozilla.org/services/fx-sync/ fx-sync] repository. The code there includes support for the Sync add-on; it's deprecated since Firefox 4.
 
Things that live there:
 
* The Services client libraries in services/crypto/ and services/sync.
 
* The Firefox Sync add-on in addon/*
 
* The Firefox 3.x and Fennec 1.x UIs for the Sync add-on in ui/*


== mozilla-central ==
== mozilla-central ==


mozilla-central is the repository for Firefox and the base repository for other Mozilla-based applications. How mozilla-central is relevant to Sync:
mozilla-central is the repository for Firefox.
 
* services-central is [[Services/Process/MergingBetweenBranches|periodically merged into mozilla-central]].
 
* The integrated UI for Firefox Sync lives in browser/base/*, along with the rest of the Firefox UI.
 
* Firefox releases are spun from mozilla-central. Changes made to services-central don't directly reach users.


== Try server ==
== Try server ==
Because Sync is developed in a clone of mozilla-central, you can use ordinary "try build" procedures to test your changes. (You don't always need to, though; one of the advantages of having an integration branch like services-central is that we're at liberty to break it!)


To push to try, teave your changes as applied mq patches. Then run:
To push to try, teave your changes as applied mq patches. Then run:
Line 258: Line 207:
   hg push -f ssh://hg.mozilla.org/try/
   hg push -f ssh://hg.mozilla.org/try/


Alternatively, you can push to try using `mach try`, which works for both `git` and `hg` users (assuming git users are using git cinnabar), and is recommended for both.
-->


== Testing ==
= Testing =


[https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests Writing xpcshell-based unit testing].
* [https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests Writing xpcshell-based unit testing].  
* Sync tests are located at [https://searchfox.org/mozilla-central/source/services/sync/tests/unit/ /services/sync/tests/unit]
* [https://firefox-source-docs.mozilla.org/testing/tps/index.html TPS tests]


<!--
= Collaboration =
= Collaboration =


Line 287: Line 241:
   }
   }
   {
   {
     comment = "Public channel for Identity";
     comment = "Public channel for FxA";
     name = "#identity";
     name = "#fxa";
     chatnet = "mozilla";
     chatnet = "mozilla";
   }
   }
  }
  }
 
-->
== Team Meetings ==
 
* Services Weekly Status: Every Tuesday at 9:15 PT in North Bridge (4th floor), [https://intranet.mozilla.org/Teleconferencing conference dial] in number is #8616. Meeting notes live under [https://wiki.mozilla.org/Services/Meetings/ Services/Meetings].
* [[WeeklyUpdates|Mozilla Weekly Updates]]: Every Monday at 11:00 PT in 10-Forward (you'll need to attend the first one in person since we want to show you off.)
* Out-of-office, PTO, and other kinds of availability is managed in a shared "Services Team Availability" calendar. See [https://intranet.mozilla.org/Calendar#Shared_Company_Calendars Shared Company Calenders] for details on how to set it up.


= Miscellany =
= Miscellany =

Latest revision as of 19:30, 29 June 2026

Hacking

Source Code

Once you have a copy of the Mozilla source tree (read below for how to obtain a copy), you can find the core Sync source code in https://searchfox.org/firefox-main/source/services/sync

This is just the core JavaScript APIs and logic. Additional Sync code for integration with Firefox lives in https://searchfox.org/firefox-main/source/browser/base/content

Look for files with sync in their name. Files here define the integration between Firefox and Sync. This includes all the Sync UI.

Additionally, Firefox Accounts specific code lives in https://searchfox.org/firefox-main/source/services/fxaccounts

Debugging Sync

If you are debugging Sync in your browser, the first thing you want to do is crank up the logging. In about:config, set the following preferences:

# Enable logging to Error Console
services.sync.log.appender.console Trace
# Enable logging to STDOUT
services.sync.log.appender.dump Trace
# Log full sync logs on sync success (typically they are only saved on failure)
services.sync.log.appender.file.logOnSuccess true
# Enable trace logging for engines
services.sync.log.logger.engine Trace

# (optional) Enable trace logging for all of sync (this is very verbose and will include all request bodies)
services.sync.log.logger Trace

Once you've updated those settings, restart Firefox for them to take effect.

If you are logging to STDOUT, you may want to set the environment variable MOZ_QUIET to silence a lot of spew from other parts of Firefox:

export MOZ_QUIET=1

You can view full sync logs by opening about:sync-log in Firefox. Read more about Firefox Sync extended logging for fetching successful sync event logfiles.

Building Sync

You probably should build using an artifact build, as described here.

After setting this up, you'll do `mach build` to perform a build, `mach run` to run Firefox, and `mach xpcshell-test /services/sync/tests/unit` to run the sync unit test suite. There's also a separate end-to-end test suite, TPS, which is documented here. (For most changes it's sufficient that you run the unit tests.)

Depending on your OS, most changes will not require a rebuild to take effect, only restarting and rerunning via `mach run`.

Using Bugzilla

Every code change starts out as a bug in Bugzilla where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see code review policy for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into mozilla-central.

You probably also want to get up to speed on Bugzilla if you're not already.

  • A good start is to go watch Jonath's "Bugzilla for Humans" video.
  • Sync bugs are found in different places depending on which part of the system they affect. For the service side (syncstorage or tokenserver), bugs will be in [1]. For the client side, you can find a list of components [CloudServices/Sync#Report_Bugs here].
  • Don't forget to add a Bugzilla ID shortcut string to your Bugzilla account preferences. The current standard practice is to append "<whitespace>[:irc_nick]" to your Real Name field.
  • You also want to watch the components relevant to your work so you automatically get emails for all bug changes in those components. The expected components where bugs will be filed are listed on CloudServices/Sync#Report_Bugs.
  • A useful Firefox add-on that makes life in Bugzilla a bit easier is Bugzilla Tweaks. Highly recommended!


Using Git

Various projects exist as Git repositories.

Configuring Git

If this is your first time using Git, you'll need for perform some one-time configuration:

# Configure your global name and email. These will be used for commit info.
$ git config --global user.name "John Doe"
$ git config --global user.email "johndoe@mozilla.com"

# Enable color output for all commands
$ git config --global color.branch auto
$ git config --global color.diff auto
$ git config --global color.grep auto
$ git config --global color.interactive auto
$ git config --global color.status auto
$ git config --global color.ui auto

# Highlight whitespace
$ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"

# Define your editor for commits, other tools
$ git config --global core.editor vim

If you find you have a slow git prompt, you may find the instructions here beneficial as well.


Testing


Miscellany

  • You also want to grab a Nightly build of Firefox. Everybody in the company should be running a nightly build: it's a great way to find bugs before the code goes out to the general public. If you want to run against different profiles (which is useful for development), you will be interested in two extra command flags: "-P and --no-remote"
    • -P forces the Profile Picker (You probably want to have more than one.)
    • --no-remote allows multiple versions to run on one machine at the same time.