MozillaWiki - User contributions [en]

Performance/Glossary

2020-08-07T18:17:15Z

Kmoir: /* Backfill */

Classification of user actions/flows that will be perceived as task with discrete performance expectations and settings/configurations that affect the raw performance of those tasks.

==Pageload==

=== Metrics ===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| || || One or two sentences ||
* Full definition
* Related tests/coverage
* Results
* User research
|-
| SpeedIndex || Visual || (SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better. ||[https://developer.mozilla.org/en-US/docs/Glossary/Speed_index MDN Documentation]
|-
| Contentful SpeedIndex || Visual || This metric uses edge detection to calculate the amount of “content” that is visible on each frame. It was designed to have a good metric to measure the amount of text that is visible and ignore the impact of popup or login screens at the end of a page load. || [https://www.sitespeed.io/documentation/sitespeed.io/metrics/#contentful-speed-index sitespeed.io documentation]
|-
| Perceptual SpeedIndex || Visual || A metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability. || * [http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages Perceptual Speed Index (PSI) for measuring above-fold visual performance of webpages]
* [https://github.com/WPO-Foundation/visualmetrics Visual metrics github repo]
|-
| Visual Completeness || Visual || The point in time when a web browser has finished rendering viewable content in the viewport, but not the entire page. ||
|-
| First Paint event || Standard || The time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation. || [https://developer.mozilla.org/en-US/docs/Glossary/First_paint MDN documentation]
|-
| First Contentful Paint || Standard || How long it takes the browser to render the first bit of content from the DOM providing the first feedback to the user the page is actually loading. || [https://developer.mozilla.org/en-US/docs/Glossary/First_contentful_paint MDN documentation]
|-
| DOMContentLoaded event || Standard || A DOM event that is fired once the underlying document has finished being parsed and loaded, but before any of its additional resources (stylesheets, images, subframes) have finished loading. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContent MDN documentation]
|-
| Onload event || Standard || A DOM event that is fired once the underlying document has finished loading, including all of its stylesheets, images and subframes. This does not include additional resources that start to load due to script. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event MDN Documentation]
|-
| Time to Interactive || Proposed standard || (TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity.

# The page has displayed useful content
# Event handlers are registered for most visible page elements
# When a user interacts with the page, the page consistently responds within 50ms - the user does not experience jank.
||
* [https://developer.mozilla.org/en-US/docs/Glossary/Time_to_interactive MDN Documentation]
* [https://github.com/WICG/time-to-interactive Proposed web standard]
|-
| Cold page load || || browser started and stabilized/idle, same profile dir as previous, site not loaded or cache cleared ||
|-
| Warm page load || || browser started and stabilized/idle, same profile dir as previous, site already loaded once, any click-throughs clicked ||
|-
| Reload || || To re-request the rendering of a document. This may mean re-retrieving the entire document and any of its resources off of the network, or might mean reloading the resources out of a local cache. ||
|-
| Backwards/Forwards navigation || || ||
|-
| Level 1 cookie blocking || Tracking Protection || ||
|-
| Level 2 cookie blocking || Tracking Protection || ||
|-
| Level 1 content blocking || Tracking Protection|| ||
|-
| Level 2 content blocking || Tracking Protection || ||
|-
| || || ||
|-
|}

=== Responsiveness (mobile vs Desktop) ===

====Scroll====

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Checkerboarding || Metrics || Checkerboarding occurs when Gecko is unable to paint the viewable portion of a webpage while we're scrolling. Instead, we just paint a color instead of the content. ||
|-
| Velocity || Metrics || ||
|-
| Touch|| Variations || ||
|-
| Keyboard || Variations || ||
|-
| Mouse/UI || Variations || ||
|-
|}

====Browser====
{| class="wikitable sortable"
|-
! Term !! Metadata !! Desktop Definition !! Mobile Definition !! Relevant Links
|-
| Pageload Jank || || || ||
|-
| Time to first blank || Open || The time from process start to display the very first painted frame. On Windows, this is currently a white rectangle, which we consider "first blank" || N/A ||
|-
| Time to first chrome paint || Open || The time from process start to displaying the first frame of the browser UI (not including the content area) || N/A ||
|-
| Time to settled about:home || Open || The time from process start to displaying a fully loaded about:home || N/A ||
|-
| Fresh profile || Open || new profile dir, user.js, first time network requests || Application process starts for the very first time. This includes onboarding, but no migration. ||
|-
| Cold start || Open || The browser is started when it hasn't already been started in this operating system session, so the disk cache is empty. For example, the first time that Firefox starts up after the operating system boots is a cold start. || Application process is not running. Not a first run, and no onboarding. The application is started either via the homescreen or applink. In GPlay terminology, there's no stored state (i.e. no `onSaveInstanceState` bundles) When the Fenix team measures cold start times, the generally measure from process start to visual completeness of the homescreen. ||
|-
| Warm start || Open || The browser starts when a previous instance of the browser was already started and shutdown within the same operating system session. || Application process is running, but the HomeActivity is not. The application is started either via the homescreen or applink. In GPlay terminology, "warm start" includes many situations and is not well defined. For example, it also includes if the app process is killed but there are `onSaveInstance` state bundles (which seems closer to Cold start)
'''This is not currently measured by the Fenix team.'''
||
|-
| Hot start || Open || Depending on user settings this will either open new window or new tab || Application process is running, HomeActivity is also running. The application is started either via the homescreen or applink, which brings the HomeActivity to the foreground.

In GPlay terminology, `onTrimMemory` may or may not have been called)

'''This is not currently measured by the Fenix team.'''
||
|-
| Delete history on close || Close || A non-default configuration that has Firefox forget all user browsing history on shut down. || N/A ||
|-
| From crash || Session Restore || || N/A ||
|-
| User initiated || Session Restore || || N/A ||
|-
| App link || Variations || N/A || ||
|-
| With WebExtension->Theme || Variations || At least one or more WebExtension themes is installed, and one is enabled. || ||
|-
| With WebExtension->Addon || Variations || At least one or more WebExtension addon is installed and enabled. || ||
|-
| Secure Proxy || Variations || The "Firefox Private Network" WebExtension. || ||
|-
| Time to interface change complete || UI || Click on a "share" button and the delay to when the display is shown. || ||
|-
| Open || Window || To open a new browser window.|| N/A ||
|-
| Close || Window || To close a new browser window. || N/A ||
|-
| Switch|| Window || To switch focus from one browser window to another. || N/A ||
|-
| Resize || Window || To resize a browser window. || N/A ||
|-
| Open || Tab || To open a new browser tab. || ||
|-
| Close || Tab || To close a new browser tab. || ||
|-
| Switch || Tab || To switch from one browser tab to another within the same window. || ||
|-
| Open || Menu || To open one of Firefox's built in toolbar menus or context menu. || ||
|-
| Close || Menu || To close one of Firefox's built in toolbar menus or context menu. || ||
|-
| Touch || Input || Input from the user via a touchscreen. || ||
|-
| Keyboard|| Input || Input from the user via the keyboard. || ||
|-
| Mouse || Input || Input from the user via a mouse. || ||
|-
|}

===Resource===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Memory || || ||
|-
| Power || || ||
|-
| CPU || || ||
|-
| GPU || || ||
|-
| Disk || || ||
|-
| Network || || ||
|-
| App idle in foreground || Network variations || ||
|-
| App idle in background || Network variations || ||
|-
| Audio || Media || ||
|-
| Dropped Frame || Video|| ||
|-
| Resolution Change || Media || ||
|-
| Stall length || Media || ||
|-
| AV sync || Media || ||
|-
| Number of stalls || Media || ||
|-
| AV sync || Media || ||
|-
| Time to payback state change || Media || ||
|-
| Real Time Communication (RTC) || Media || ||
|-
| Frame per second (FPS) || Virtual Reality (VR) || ||
|-
| Visual Distortion || Virtual Reality (VR) || ||
|-
| Z-fighting || Virtual Reality (VR) || ||
|-
| Judder || Virtual Reality (VR) || ||
|-
|}

===Components===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Graphics || || ||
|-
| Layout || || ||
|-
| Parsing || Javascript || || [[JavaScript]
|-
| Compilation || Javascript || || [[JavaScript]]
|-
| Network || || || [[Networking]]
|-
| DOM || || || [[DOM]]
|-
| Storage || || || [[Storage]]
|-
| Media || || || [[Media]]
|-
|}

=Mozilla specific terms=

== Backfill ==
Trigger skipped jobs (Treeherder terminology).
Also refers to filling in missing data points from Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graphs view].

== Data point ==
Performance metric value visible in Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graph view].
[[File:Data point.png|700px|none|left|Selected data point]]

== DOM Content Flushed (dcf) ==
This is a Mozilla proprietary extension and not part of the performance/navigation timing specification. It marks the completion of the first presentation flush after DOMContentLoaded.
Internal Firefox metric activated by setting the preference dom.performance.time_to_dom_content_flushed.enabled to true.

== Downstream ==
Alert triggered by a commit that originally happened on another repo and was merged into the current one after.

== First Non-Blank Paint (fnbpaint) ==
The timestamp of a shipped frame that contains the first paint to draw more than the document background.
Has low value to a user, as this paint may be completely meaningless or useless. The first non-blank paint is often represented as "Start Render" in performance products.

== Gecko profile ==
The Firefox Profiler is a profiler that is built into Firefox and is available at profiler.firefox.com. It has tighter integration with Firefox than external profilers, and has more of a platform focus that the DevTools performance panel. It can be used a variety of situations where external profilers are not available, and can provide more information and insight into what the browser is doing.

== mitmproxy ==
[https://mitmproxy.org/ Mitmproxy] is a third-party open-source tool ([https://github.com/mitmproxy/mitmproxy found here in github]) that allows the recording of a live web page and saving it as a playback archive. Using Mitmproxy's 'mitmdump' tool, the recorded page can be played back through a local proxy browser proxy; then browsing to the URL loads the page from the mitmproxy recording instead of the live site.

The Raptor performance testing framework uses Mitmproxy for Raptor page-load performance tests. Mitmproxy is available as a python [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozbase/mozproxy] module for import in any python project.

== Noise (regarding performance tests) ==
Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have.
Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify.
The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points.

== Perf Sheriff ==
Using perfherder, a perf-sheriff will perform these tasks:
# Look at the graph and determine the original branch, date, revision where the alert occurred
# Look at Treeherder and determine if we have all the data.
# Retrigger jobs if needed (more noise, more retriggers)
# Once you have more data, look at the data in compare view to see if other tests/platforms have changed
# Add all related alerts you see to the summary with the reassign button

== Perfherder ==
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against '''Mozilla''' products (currently Firefox and Firefox for Android).

== Raptor ==
Raptor is a performance testing framework used for running browser page-load and browser benchmark tests. Raptor can run performance tests on Firefox desktop, Firefox android (Fennec, Geckoview example app, and the android components Reference Browser), and Google Chrome/Chromium. For more information please refer to [https://wiki.mozilla.org/Performance_sheriffing/Raptor the Raptor wiki page.]

==RelMan glossary==
[https://wiki.mozilla.org/Release_Management/Glossary These terms] are of use, especially when doing Performance sheriffing.

== Retrigger ==
Any run of the same job on the same commit for the same repo triggered after the first one.
Retriggers usually happen when the available datapoint in the graph are not enough for the sheriff to determine the culprit.

== SETA ==
SETA finds the minimum set of jobs to run in order to catch all failures that our automation has found in the recent past on Firefox development repositories.
SETA creates job priorities for all jobs found in the runnable-jobs API for that repository.

== Talos ==
Talos is our versatile performance testing framework we use at '''Mozilla'''. It was created to serve as a test runner for the existing performance tests that '''Mozilla''' was running back in 2007 as well as providing an extensible framework for new tests as they were created.

== Treeherder ==
Treeherder is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.

== Time To First Interactive (ttfi) ==
This is a newer metric and reports when the page is first expected to be usable and will respond to input quickly (with the possibility of slow responses as more content loads).
# Start looking for First Interactive at first contentful paint
# Look for the first interactive window (with no regard to in-flight requests)
# First Interactive is the start of the interactive window from step 2, first contentful paint or DOM Content Loaded, whichever is later

== R/b ==
Short name for retrigger/backfill.

== WebPageTest ==
Accurately measure, track, and report on key Firefox-performance metrics, over time.

Performance/Glossary

2020-08-07T18:10:25Z

Kmoir: /* First Contentful Paint (fcp) */

Classification of user actions/flows that will be perceived as task with discrete performance expectations and settings/configurations that affect the raw performance of those tasks.

==Pageload==

=== Metrics ===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| || || One or two sentences ||
* Full definition
* Related tests/coverage
* Results
* User research
|-
| SpeedIndex || Visual || (SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better. ||[https://developer.mozilla.org/en-US/docs/Glossary/Speed_index MDN Documentation]
|-
| Contentful SpeedIndex || Visual || This metric uses edge detection to calculate the amount of “content” that is visible on each frame. It was designed to have a good metric to measure the amount of text that is visible and ignore the impact of popup or login screens at the end of a page load. || [https://www.sitespeed.io/documentation/sitespeed.io/metrics/#contentful-speed-index sitespeed.io documentation]
|-
| Perceptual SpeedIndex || Visual || A metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability. || * [http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages Perceptual Speed Index (PSI) for measuring above-fold visual performance of webpages]
* [https://github.com/WPO-Foundation/visualmetrics Visual metrics github repo]
|-
| Visual Completeness || Visual || The point in time when a web browser has finished rendering viewable content in the viewport, but not the entire page. ||
|-
| First Paint event || Standard || The time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation. || [https://developer.mozilla.org/en-US/docs/Glossary/First_paint MDN documentation]
|-
| First Contentful Paint || Standard || How long it takes the browser to render the first bit of content from the DOM providing the first feedback to the user the page is actually loading. || [https://developer.mozilla.org/en-US/docs/Glossary/First_contentful_paint MDN documentation]
|-
| DOMContentLoaded event || Standard || A DOM event that is fired once the underlying document has finished being parsed and loaded, but before any of its additional resources (stylesheets, images, subframes) have finished loading. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContent MDN documentation]
|-
| Onload event || Standard || A DOM event that is fired once the underlying document has finished loading, including all of its stylesheets, images and subframes. This does not include additional resources that start to load due to script. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event MDN Documentation]
|-
| Time to Interactive || Proposed standard || (TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity.

# The page has displayed useful content
# Event handlers are registered for most visible page elements
# When a user interacts with the page, the page consistently responds within 50ms - the user does not experience jank.
||
* [https://developer.mozilla.org/en-US/docs/Glossary/Time_to_interactive MDN Documentation]
* [https://github.com/WICG/time-to-interactive Proposed web standard]
|-
| Cold page load || || browser started and stabilized/idle, same profile dir as previous, site not loaded or cache cleared ||
|-
| Warm page load || || browser started and stabilized/idle, same profile dir as previous, site already loaded once, any click-throughs clicked ||
|-
| Reload || || To re-request the rendering of a document. This may mean re-retrieving the entire document and any of its resources off of the network, or might mean reloading the resources out of a local cache. ||
|-
| Backwards/Forwards navigation || || ||
|-
| Level 1 cookie blocking || Tracking Protection || ||
|-
| Level 2 cookie blocking || Tracking Protection || ||
|-
| Level 1 content blocking || Tracking Protection|| ||
|-
| Level 2 content blocking || Tracking Protection || ||
|-
| || || ||
|-
|}

=== Responsiveness (mobile vs Desktop) ===

====Scroll====

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Checkerboarding || Metrics || Checkerboarding occurs when Gecko is unable to paint the viewable portion of a webpage while we're scrolling. Instead, we just paint a color instead of the content. ||
|-
| Velocity || Metrics || ||
|-
| Touch|| Variations || ||
|-
| Keyboard || Variations || ||
|-
| Mouse/UI || Variations || ||
|-
|}

====Browser====
{| class="wikitable sortable"
|-
! Term !! Metadata !! Desktop Definition !! Mobile Definition !! Relevant Links
|-
| Pageload Jank || || || ||
|-
| Time to first blank || Open || The time from process start to display the very first painted frame. On Windows, this is currently a white rectangle, which we consider "first blank" || N/A ||
|-
| Time to first chrome paint || Open || The time from process start to displaying the first frame of the browser UI (not including the content area) || N/A ||
|-
| Time to settled about:home || Open || The time from process start to displaying a fully loaded about:home || N/A ||
|-
| Fresh profile || Open || new profile dir, user.js, first time network requests || Application process starts for the very first time. This includes onboarding, but no migration. ||
|-
| Cold start || Open || The browser is started when it hasn't already been started in this operating system session, so the disk cache is empty. For example, the first time that Firefox starts up after the operating system boots is a cold start. || Application process is not running. Not a first run, and no onboarding. The application is started either via the homescreen or applink. In GPlay terminology, there's no stored state (i.e. no `onSaveInstanceState` bundles) When the Fenix team measures cold start times, the generally measure from process start to visual completeness of the homescreen. ||
|-
| Warm start || Open || The browser starts when a previous instance of the browser was already started and shutdown within the same operating system session. || Application process is running, but the HomeActivity is not. The application is started either via the homescreen or applink. In GPlay terminology, "warm start" includes many situations and is not well defined. For example, it also includes if the app process is killed but there are `onSaveInstance` state bundles (which seems closer to Cold start)
'''This is not currently measured by the Fenix team.'''
||
|-
| Hot start || Open || Depending on user settings this will either open new window or new tab || Application process is running, HomeActivity is also running. The application is started either via the homescreen or applink, which brings the HomeActivity to the foreground.

In GPlay terminology, `onTrimMemory` may or may not have been called)

'''This is not currently measured by the Fenix team.'''
||
|-
| Delete history on close || Close || A non-default configuration that has Firefox forget all user browsing history on shut down. || N/A ||
|-
| From crash || Session Restore || || N/A ||
|-
| User initiated || Session Restore || || N/A ||
|-
| App link || Variations || N/A || ||
|-
| With WebExtension->Theme || Variations || At least one or more WebExtension themes is installed, and one is enabled. || ||
|-
| With WebExtension->Addon || Variations || At least one or more WebExtension addon is installed and enabled. || ||
|-
| Secure Proxy || Variations || The "Firefox Private Network" WebExtension. || ||
|-
| Time to interface change complete || UI || Click on a "share" button and the delay to when the display is shown. || ||
|-
| Open || Window || To open a new browser window.|| N/A ||
|-
| Close || Window || To close a new browser window. || N/A ||
|-
| Switch|| Window || To switch focus from one browser window to another. || N/A ||
|-
| Resize || Window || To resize a browser window. || N/A ||
|-
| Open || Tab || To open a new browser tab. || ||
|-
| Close || Tab || To close a new browser tab. || ||
|-
| Switch || Tab || To switch from one browser tab to another within the same window. || ||
|-
| Open || Menu || To open one of Firefox's built in toolbar menus or context menu. || ||
|-
| Close || Menu || To close one of Firefox's built in toolbar menus or context menu. || ||
|-
| Touch || Input || Input from the user via a touchscreen. || ||
|-
| Keyboard|| Input || Input from the user via the keyboard. || ||
|-
| Mouse || Input || Input from the user via a mouse. || ||
|-
|}

===Resource===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Memory || || ||
|-
| Power || || ||
|-
| CPU || || ||
|-
| GPU || || ||
|-
| Disk || || ||
|-
| Network || || ||
|-
| App idle in foreground || Network variations || ||
|-
| App idle in background || Network variations || ||
|-
| Audio || Media || ||
|-
| Dropped Frame || Video|| ||
|-
| Resolution Change || Media || ||
|-
| Stall length || Media || ||
|-
| AV sync || Media || ||
|-
| Number of stalls || Media || ||
|-
| AV sync || Media || ||
|-
| Time to payback state change || Media || ||
|-
| Real Time Communication (RTC) || Media || ||
|-
| Frame per second (FPS) || Virtual Reality (VR) || ||
|-
| Visual Distortion || Virtual Reality (VR) || ||
|-
| Z-fighting || Virtual Reality (VR) || ||
|-
| Judder || Virtual Reality (VR) || ||
|-
|}

===Components===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Graphics || || ||
|-
| Layout || || ||
|-
| Parsing || Javascript || || [[JavaScript]
|-
| Compilation || Javascript || || [[JavaScript]]
|-
| Network || || || [[Networking]]
|-
| DOM || || || [[DOM]]
|-
| Storage || || || [[Storage]]
|-
| Media || || || [[Media]]
|-
|}

== Backfill ==
Trigger skipped jobs (Treeherder terminology).
Also refers to filling in missing data points from Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graphs view].

== Data point ==
Performance metric value visible in Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graph view].
[[File:Data point.png|700px|none|left|Selected data point]]

== DOM Content Flushed (dcf) ==
This is a Mozilla proprietary extension and not part of the performance/navigation timing specification. It marks the completion of the first presentation flush after DOMContentLoaded.
Internal Firefox metric activated by setting the preference dom.performance.time_to_dom_content_flushed.enabled to true.

== Downstream ==
Alert triggered by a commit that originally happened on another repo and was merged into the current one after.

== First Non-Blank Paint (fnbpaint) ==
The timestamp of a shipped frame that contains the first paint to draw more than the document background.
Has low value to a user, as this paint may be completely meaningless or useless. The first non-blank paint is often represented as "Start Render" in performance products.

== Gecko profile ==
The Firefox Profiler is a profiler that is built into Firefox and is available at profiler.firefox.com. It has tighter integration with Firefox than external profilers, and has more of a platform focus that the DevTools performance panel. It can be used a variety of situations where external profilers are not available, and can provide more information and insight into what the browser is doing.

== mitmproxy ==
[https://mitmproxy.org/ Mitmproxy] is a third-party open-source tool ([https://github.com/mitmproxy/mitmproxy found here in github]) that allows the recording of a live web page and saving it as a playback archive. Using Mitmproxy's 'mitmdump' tool, the recorded page can be played back through a local proxy browser proxy; then browsing to the URL loads the page from the mitmproxy recording instead of the live site.

The Raptor performance testing framework uses Mitmproxy for Raptor page-load performance tests. Mitmproxy is available as a python [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozbase/mozproxy] module for import in any python project.

== Noise (regarding performance tests) ==
Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have.
Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify.
The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points.

== Perf Sheriff ==
Using perfherder, a perf-sheriff will perform these tasks:
# Look at the graph and determine the original branch, date, revision where the alert occurred
# Look at Treeherder and determine if we have all the data.
# Retrigger jobs if needed (more noise, more retriggers)
# Once you have more data, look at the data in compare view to see if other tests/platforms have changed
# Add all related alerts you see to the summary with the reassign button

== Perfherder ==
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against '''Mozilla''' products (currently Firefox and Firefox for Android).

== Raptor ==
Raptor is a performance testing framework used for running browser page-load and browser benchmark tests. Raptor can run performance tests on Firefox desktop, Firefox android (Fennec, Geckoview example app, and the android components Reference Browser), and Google Chrome/Chromium. For more information please refer to [https://wiki.mozilla.org/Performance_sheriffing/Raptor the Raptor wiki page.]

==RelMan glossary==
[https://wiki.mozilla.org/Release_Management/Glossary These terms] are of use, especially when doing Performance sheriffing.

== Retrigger ==
Any run of the same job on the same commit for the same repo triggered after the first one.
Retriggers usually happen when the available datapoint in the graph are not enough for the sheriff to determine the culprit.

== SETA ==
SETA finds the minimum set of jobs to run in order to catch all failures that our automation has found in the recent past on Firefox development repositories.
SETA creates job priorities for all jobs found in the runnable-jobs API for that repository.

== Talos ==
Talos is our versatile performance testing framework we use at '''Mozilla'''. It was created to serve as a test runner for the existing performance tests that '''Mozilla''' was running back in 2007 as well as providing an extensible framework for new tests as they were created.

== Treeherder ==
Treeherder is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.

== Time To First Interactive (ttfi) ==
This is a newer metric and reports when the page is first expected to be usable and will respond to input quickly (with the possibility of slow responses as more content loads).
# Start looking for First Interactive at first contentful paint
# Look for the first interactive window (with no regard to in-flight requests)
# First Interactive is the start of the interactive window from step 2, first contentful paint or DOM Content Loaded, whichever is later

== R/b ==
Short name for retrigger/backfill.

== WebPageTest ==
Accurately measure, track, and report on key Firefox-performance metrics, over time.

Performance/Glossary

2020-08-05T21:07:21Z

Kmoir: merge content from TestEngineering/Performance/Glossary in

Classification of user actions/flows that will be perceived as task with discrete performance expectations and settings/configurations that affect the raw performance of those tasks.

==Pageload==

=== Metrics ===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| || || One or two sentences ||
* Full definition
* Related tests/coverage
* Results
* User research
|-
| SpeedIndex || Visual || (SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better. ||[https://developer.mozilla.org/en-US/docs/Glossary/Speed_index MDN Documentation]
|-
| Contentful SpeedIndex || Visual || This metric uses edge detection to calculate the amount of “content” that is visible on each frame. It was designed to have a good metric to measure the amount of text that is visible and ignore the impact of popup or login screens at the end of a page load. || [https://www.sitespeed.io/documentation/sitespeed.io/metrics/#contentful-speed-index sitespeed.io documentation]
|-
| Perceptual SpeedIndex || Visual || A metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability. || * [http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages Perceptual Speed Index (PSI) for measuring above-fold visual performance of webpages]
* [https://github.com/WPO-Foundation/visualmetrics Visual metrics github repo]
|-
| Visual Completeness || Visual || The point in time when a web browser has finished rendering viewable content in the viewport, but not the entire page. ||
|-
| First Paint event || Standard || The time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation. || [https://developer.mozilla.org/en-US/docs/Glossary/First_paint MDN documentation]
|-
| First Contentful Paint || Standard || How long it takes the browser to render the first bit of content from the DOM providing the first feedback to the user the page is actually loading. || [https://developer.mozilla.org/en-US/docs/Glossary/First_contentful_paint MDN documentation]
|-
| DOMContentLoaded event || Standard || A DOM event that is fired once the underlying document has finished being parsed and loaded, but before any of its additional resources (stylesheets, images, subframes) have finished loading. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContent MDN documentation]
|-
| Onload event || Standard || A DOM event that is fired once the underlying document has finished loading, including all of its stylesheets, images and subframes. This does not include additional resources that start to load due to script. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event MDN Documentation]
|-
| Time to Interactive || Proposed standard || (TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity.

# The page has displayed useful content
# Event handlers are registered for most visible page elements
# When a user interacts with the page, the page consistently responds within 50ms - the user does not experience jank.
||
* [https://developer.mozilla.org/en-US/docs/Glossary/Time_to_interactive MDN Documentation]
* [https://github.com/WICG/time-to-interactive Proposed web standard]
|-
| Cold page load || || browser started and stabilized/idle, same profile dir as previous, site not loaded or cache cleared ||
|-
| Warm page load || || browser started and stabilized/idle, same profile dir as previous, site already loaded once, any click-throughs clicked ||
|-
| Reload || || To re-request the rendering of a document. This may mean re-retrieving the entire document and any of its resources off of the network, or might mean reloading the resources out of a local cache. ||
|-
| Backwards/Forwards navigation || || ||
|-
| Level 1 cookie blocking || Tracking Protection || ||
|-
| Level 2 cookie blocking || Tracking Protection || ||
|-
| Level 1 content blocking || Tracking Protection|| ||
|-
| Level 2 content blocking || Tracking Protection || ||
|-
| || || ||
|-
|}

=== Responsiveness (mobile vs Desktop) ===

====Scroll====

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Checkerboarding || Metrics || Checkerboarding occurs when Gecko is unable to paint the viewable portion of a webpage while we're scrolling. Instead, we just paint a color instead of the content. ||
|-
| Velocity || Metrics || ||
|-
| Touch|| Variations || ||
|-
| Keyboard || Variations || ||
|-
| Mouse/UI || Variations || ||
|-
|}

====Browser====
{| class="wikitable sortable"
|-
! Term !! Metadata !! Desktop Definition !! Mobile Definition !! Relevant Links
|-
| Pageload Jank || || || ||
|-
| Time to first blank || Open || The time from process start to display the very first painted frame. On Windows, this is currently a white rectangle, which we consider "first blank" || N/A ||
|-
| Time to first chrome paint || Open || The time from process start to displaying the first frame of the browser UI (not including the content area) || N/A ||
|-
| Time to settled about:home || Open || The time from process start to displaying a fully loaded about:home || N/A ||
|-
| Fresh profile || Open || new profile dir, user.js, first time network requests || Application process starts for the very first time. This includes onboarding, but no migration. ||
|-
| Cold start || Open || The browser is started when it hasn't already been started in this operating system session, so the disk cache is empty. For example, the first time that Firefox starts up after the operating system boots is a cold start. || Application process is not running. Not a first run, and no onboarding. The application is started either via the homescreen or applink. In GPlay terminology, there's no stored state (i.e. no `onSaveInstanceState` bundles) When the Fenix team measures cold start times, the generally measure from process start to visual completeness of the homescreen. ||
|-
| Warm start || Open || The browser starts when a previous instance of the browser was already started and shutdown within the same operating system session. || Application process is running, but the HomeActivity is not. The application is started either via the homescreen or applink. In GPlay terminology, "warm start" includes many situations and is not well defined. For example, it also includes if the app process is killed but there are `onSaveInstance` state bundles (which seems closer to Cold start)
'''This is not currently measured by the Fenix team.'''
||
|-
| Hot start || Open || Depending on user settings this will either open new window or new tab || Application process is running, HomeActivity is also running. The application is started either via the homescreen or applink, which brings the HomeActivity to the foreground.

In GPlay terminology, `onTrimMemory` may or may not have been called)

'''This is not currently measured by the Fenix team.'''
||
|-
| Delete history on close || Close || A non-default configuration that has Firefox forget all user browsing history on shut down. || N/A ||
|-
| From crash || Session Restore || || N/A ||
|-
| User initiated || Session Restore || || N/A ||
|-
| App link || Variations || N/A || ||
|-
| With WebExtension->Theme || Variations || At least one or more WebExtension themes is installed, and one is enabled. || ||
|-
| With WebExtension->Addon || Variations || At least one or more WebExtension addon is installed and enabled. || ||
|-
| Secure Proxy || Variations || The "Firefox Private Network" WebExtension. || ||
|-
| Time to interface change complete || UI || Click on a "share" button and the delay to when the display is shown. || ||
|-
| Open || Window || To open a new browser window.|| N/A ||
|-
| Close || Window || To close a new browser window. || N/A ||
|-
| Switch|| Window || To switch focus from one browser window to another. || N/A ||
|-
| Resize || Window || To resize a browser window. || N/A ||
|-
| Open || Tab || To open a new browser tab. || ||
|-
| Close || Tab || To close a new browser tab. || ||
|-
| Switch || Tab || To switch from one browser tab to another within the same window. || ||
|-
| Open || Menu || To open one of Firefox's built in toolbar menus or context menu. || ||
|-
| Close || Menu || To close one of Firefox's built in toolbar menus or context menu. || ||
|-
| Touch || Input || Input from the user via a touchscreen. || ||
|-
| Keyboard|| Input || Input from the user via the keyboard. || ||
|-
| Mouse || Input || Input from the user via a mouse. || ||
|-
|}

===Resource===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Memory || || ||
|-
| Power || || ||
|-
| CPU || || ||
|-
| GPU || || ||
|-
| Disk || || ||
|-
| Network || || ||
|-
| App idle in foreground || Network variations || ||
|-
| App idle in background || Network variations || ||
|-
| Audio || Media || ||
|-
| Dropped Frame || Video|| ||
|-
| Resolution Change || Media || ||
|-
| Stall length || Media || ||
|-
| AV sync || Media || ||
|-
| Number of stalls || Media || ||
|-
| AV sync || Media || ||
|-
| Time to payback state change || Media || ||
|-
| Real Time Communication (RTC) || Media || ||
|-
| Frame per second (FPS) || Virtual Reality (VR) || ||
|-
| Visual Distortion || Virtual Reality (VR) || ||
|-
| Z-fighting || Virtual Reality (VR) || ||
|-
| Judder || Virtual Reality (VR) || ||
|-
|}

===Components===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Graphics || || ||
|-
| Layout || || ||
|-
| Parsing || Javascript || || [[JavaScript]
|-
| Compilation || Javascript || || [[JavaScript]]
|-
| Network || || || [[Networking]]
|-
| DOM || || || [[DOM]]
|-
| Storage || || || [[Storage]]
|-
| Media || || || [[Media]]
|-
|}

== Backfill ==
Trigger skipped jobs (Treeherder terminology).
Also refers to filling in missing data points from Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graphs view].

== Data point ==
Performance metric value visible in Perfherder's [https://treeherder.mozilla.org/perf.html#/graphs Graph view].
[[File:Data point.png|700px|none|left|Selected data point]]

== DOM Content Flushed (dcf) ==
This is a Mozilla proprietary extension and not part of the performance/navigation timing specification. It marks the completion of the first presentation flush after DOMContentLoaded.
Internal Firefox metric activated by setting the preference dom.performance.time_to_dom_content_flushed.enabled to true.

== Downstream ==
Alert triggered by a commit that originally happened on another repo and was merged into the current one after.

== First Contentful Paint (fcp) ==
First Contentful Paint (FCP) measures the time from navigation to the time when the browser renders the first bit of content from the DOM.

== First Non-Blank Paint (fnbpaint) ==
The timestamp of a shipped frame that contains the first paint to draw more than the document background.
Has low value to a user, as this paint may be completely meaningless or useless. The first non-blank paint is often represented as "Start Render" in performance products.

== Gecko profile ==
The Firefox Profiler is a profiler that is built into Firefox and is available at profiler.firefox.com. It has tighter integration with Firefox than external profilers, and has more of a platform focus that the DevTools performance panel. It can be used a variety of situations where external profilers are not available, and can provide more information and insight into what the browser is doing.

== mitmproxy ==
[https://mitmproxy.org/ Mitmproxy] is a third-party open-source tool ([https://github.com/mitmproxy/mitmproxy found here in github]) that allows the recording of a live web page and saving it as a playback archive. Using Mitmproxy's 'mitmdump' tool, the recorded page can be played back through a local proxy browser proxy; then browsing to the URL loads the page from the mitmproxy recording instead of the live site.

The Raptor performance testing framework uses Mitmproxy for Raptor page-load performance tests. Mitmproxy is available as a python [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozbase/mozproxy] module for import in any python project.

== Noise (regarding performance tests) ==
Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have.
Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify.
The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points.

== Perf Sheriff ==
Using perfherder, a perf-sheriff will perform these tasks:
# Look at the graph and determine the original branch, date, revision where the alert occurred
# Look at Treeherder and determine if we have all the data.
# Retrigger jobs if needed (more noise, more retriggers)
# Once you have more data, look at the data in compare view to see if other tests/platforms have changed
# Add all related alerts you see to the summary with the reassign button

== Perfherder ==
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against '''Mozilla''' products (currently Firefox and Firefox for Android).

== Raptor ==
Raptor is a performance testing framework used for running browser page-load and browser benchmark tests. Raptor can run performance tests on Firefox desktop, Firefox android (Fennec, Geckoview example app, and the android components Reference Browser), and Google Chrome/Chromium. For more information please refer to [https://wiki.mozilla.org/Performance_sheriffing/Raptor the Raptor wiki page.]

==RelMan glossary==
[https://wiki.mozilla.org/Release_Management/Glossary These terms] are of use, especially when doing Performance sheriffing.

== Retrigger ==
Any run of the same job on the same commit for the same repo triggered after the first one.
Retriggers usually happen when the available datapoint in the graph are not enough for the sheriff to determine the culprit.

== SETA ==
SETA finds the minimum set of jobs to run in order to catch all failures that our automation has found in the recent past on Firefox development repositories.
SETA creates job priorities for all jobs found in the runnable-jobs API for that repository.

== Talos ==
Talos is our versatile performance testing framework we use at '''Mozilla'''. It was created to serve as a test runner for the existing performance tests that '''Mozilla''' was running back in 2007 as well as providing an extensible framework for new tests as they were created.

== Treeherder ==
Treeherder is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.

== Time To First Interactive (ttfi) ==
This is a newer metric and reports when the page is first expected to be usable and will respond to input quickly (with the possibility of slow responses as more content loads).
# Start looking for First Interactive at first contentful paint
# Look for the first interactive window (with no regard to in-flight requests)
# First Interactive is the start of the interactive window from step 2, first contentful paint or DOM Content Loaded, whichever is later

== R/b ==
Short name for retrigger/backfill.

== WebPageTest ==
Accurately measure, track, and report on key Firefox-performance metrics, over time.

Performance/Glossary

2020-07-09T13:52:10Z

Kmoir: /* Components */

Performance/Glossary

2020-07-09T13:47:34Z

Kmoir: /* Components */

Performance/Glossary

2020-07-09T13:44:00Z

Kmoir: /* Metrics */

Performance/Glossary

2020-07-09T13:39:25Z

Kmoir: /* Metrics */

Performance/Glossary

2020-07-07T01:39:56Z

Kmoir: /* Metrics */

Performance/Glossary

2020-07-06T19:30:34Z

Kmoir:

=== Metrics ===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| || || One or two sentences ||
* Full definition
* Related tests/coverage
* Results
* User research
|-
| SpeedIndex || Visual || (SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better. ||[https://developer.mozilla.org/en-US/docs/Glossary/Speed_index MDN Documentation]
|-
| Contentful SpeedIndex || Visual || This metric uses edge detection to calculate the amount of “content” that is visible on each frame. It was designed to have a good metric to measure the amount of text that is visible and ignore the impact of popup or login screens at the end of a page load. || [https://www.sitespeed.io/documentation/sitespeed.io/metrics/#contentful-speed-index sitespeed.io documentation]
|-
| Perceptual SpeedIndex || Visual || A metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability. || * [http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages Perceptual Speed Index (PSI) for measuring above-fold visual performance of webpages]
* [https://github.com/WPO-Foundation/visualmetrics Visual metrics github repo]
|-
| Visual Completeness || Visual || The point in time when a web browser has finished rendering viewable content in the viewport, but not the entire page. ||
|-
| First Paint event || Standard || The time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation. || [https://developer.mozilla.org/en-US/docs/Glossary/First_paint MDN documentation]
|-
| First Contentful Paint || Standard || How long it takes the browser to render the first bit of content from the DOM providing the first feedback to the user the page is actually loading. || [[MDN Documentation|https://developer.mozilla.org/en-US/docs/Glossary/First_contentful_paint]]
|-
| DOMContentLoaded event || Standard || A DOM event that is fired once the underlying document has finished being parsed and loaded, but before any of its additional resources (stylesheets, images, subframes) have finished loading. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContent MDN documentation]
|-
| Onload event || Standard || A DOM event that is fired once the underlying document has finished loading, including all of its stylesheets, images and subframes. This does not include additional resources that start to load due to script. || [https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event MDN Documentation]
|-
| Time to Interactive || Proposed standard || (TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity.

# The page has displayed useful content
# Event handlers are registered for most visible page elements
# When a user interacts with the page, the page consistently responds within 50ms - the user does not experience jank.
||
* [https://developer.mozilla.org/en-US/docs/Glossary/Time_to_interactive MDN Documentation]
* [https://github.com/WICG/time-to-interactive Proposed web standard]
|-
| Cold page load || || browser started and stabilized/idle, same profile dir as previous, site not loaded or cache cleared ||
|-
| Warm page load || || browser started and stabilized/idle, same profile dir as previous, site already loaded once, any click-throughs clicked ||
|-
| Reload || || To re-request the rendering of a document. This may mean re-retrieving the entire document and any of its resources off of the network, or might mean reloading the resources out of a local cache. ||
|-
| Backwards/Forwards navigation || || ||
|-
| Level 1 cookie blocking || Tracking Protection || ||
|-
| Level 2 cookie blocking || Tracking Protection || ||
|-
| Level 1 content blocking || Tracking Protection|| ||
|-
| Level 2 content blocking || Tracking Protection || ||
|-
| || || ||
|-
|}

=== Responsiveness (mobile vs Desktop) ===

====Scroll====

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Checkerboarding || Metrics || Checkerboarding occurs when Gecko is unable to paint the viewable portion of a webpage while we're scrolling. Instead, we just paint a color instead of the content. ||
|-
| Velocity || Metrics || ||
|-
| Touch|| Variations || ||
|-
| Keyboard || Variations || ||
|-
| Mouse/UI || Variations || ||
|-
|}

====Browser====
{| class="wikitable sortable"
|-
! Term !! Metadata !! Desktop Definition !! Mobile Definition !! Relevant Links
|-
| Pageload Jank || || || ||
|-
| Time to first blank || Open || The time from process start to display the very first painted frame. On Windows, this is currently a white rectangle, which we consider "first blank" || N/A ||
|-
| Time to first chrome paint || Open || The time from process start to displaying the first frame of the browser UI (not including the content area) || N/A ||
|-
| Time to settled about:home || Open || The time from process start to displaying a fully loaded about:home || N/A ||
|-
| Fresh profile || Open || new profile dir, user.js, first time network requests || Application process starts for the very first time. This includes onboarding, but no migration. ||
|-
| Cold start || Open || The browser is started when it hasn't already been started in this operating system session, so the disk cache is empty. For example, the first time that Firefox starts up after the operating system boots is a cold start. || Application process is not running. Not a first run, and no onboarding. The application is started either via the homescreen or applink. In GPlay terminology, there's no stored state (i.e. no `onSaveInstanceState` bundles) When the Fenix team measures cold start times, the generally measure from process start to visual completeness of the homescreen. ||
|-
| Warm start || Open || The browser starts when a previous instance of the browser was already started and shutdown within the same operating system session. || Application process is running, but the HomeActivity is not. The application is started either via the homescreen or applink. In GPlay terminology, "warm start" includes many situations and is not well defined. For example, it also includes if the app process is killed but there are `onSaveInstance` state bundles (which seems closer to Cold start)
'''This is not currently measured by the Fenix team.'''
||
|-
| Hot start || Open || Depending on user settings this will either open new window or new tab || Application process is running, HomeActivity is also running. The application is started either via the homescreen or applink, which brings the HomeActivity to the foreground.

In GPlay terminology, `onTrimMemory` may or may not have been called)

'''This is not currently measured by the Fenix team.'''
||
|-
| Delete history on close || Close || A non-default configuration that has Firefox forget all user browsing history on shut down. || N/A ||
|-
| From crash || Session Restore || || N/A ||
|-
| User initiated || Session Restore || || N/A ||
|-
| App link || Variations || N/A || ||
|-
| With WebExtension->Theme || Variations || At least one or more WebExtension themes is installed, and one is enabled. || ||
|-
| With WebExtension->Addon || Variations || At least one or more WebExtension addon is installed and enabled. || ||
|-
| Secure Proxy || Variations || The "Firefox Private Network" WebExtension. || ||
|-
| Time to interface change complete || UI || Click on a "share" button and the delay to when the display is shown. || ||
|-
| Open || Window || To open a new browser window.|| N/A ||
|-
| Close || Window || To close a new browser window. || N/A ||
|-
| Switch|| Window || To switch focus from one browser window to another. || N/A ||
|-
| Resize || Window || To resize a browser window. || N/A ||
|-
| Open || Tab || To open a new browser tab. || ||
|-
| Close || Tab || To close a new browser tab. || ||
|-
| Switch || Tab || To switch from one browser tab to another within the same window. || ||
|-
| Open || Menu || To open one of Firefox's built in toolbar menus or context menu. || ||
|-
| Close || Menu || To close one of Firefox's built in toolbar menus or context menu. || ||
|-
| Touch || Input || Input from the user via a touchscreen. || ||
|-
| Keyboard|| Input || Input from the user via the keyboard. || ||
|-
| Mouse || Input || Input from the user via a mouse. || ||
|-
|}

===Resource===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Memory || || ||
|-
| Power || || ||
|-
| CPU || || ||
|-
| GPU || || ||
|-
| Disk || || ||
|-
| Network || || ||
|-
| App idle in foreground || Network variations || ||
|-
| App idle in background || Network variations || ||
|-
| Audio || Media || ||
|-
| Dropped Frame || Video|| ||
|-
| Resolution Change || Media || ||
|-
| Stall length || Media || ||
|-
| AV sync || Media || ||
|-
| Number of stalls || Media || ||
|-
| AV sync || Media || ||
|-
| Time to payback state change || Media || ||
|-
| Real Time Communication (RTC) || Media || ||
|-
| Frame per second (FPS) || Virtual Reality (VR) || ||
|-
| Visual Distortion || Virtual Reality (VR) || ||
|-
| Z-fighting || Virtual Reality (VR) || ||
|-
| Judder || Virtual Reality (VR) || ||
|-
|}

===Components===

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| Graphics || || ||
|-
| Layout || || ||
|-
| Parsing || Javascript || ||
|-
| Compilation || Javascript || ||
|-
| Network || || ||
|-
| DOM || || ||
|-
| Storage || || ||
|-
| Media || || ||
|-
|}

Performance/Glossary

2020-07-06T19:10:19Z

Kmoir: /* Metrics */

Performance/Glossary

2020-07-06T18:19:20Z

Kmoir:

Performance/Glossary

2020-07-06T17:20:05Z

Kmoir: Created page with "{| class="wikitable sortable" |- ! Term !! Metadata !! Definition !! Relevant Links |- | || || One or two sentences || * Full definition * Related tests/coverage * Results *..."

{| class="wikitable sortable"
|-
! Term !! Metadata !! Definition !! Relevant Links
|-
| || || One or two sentences ||
* Full definition
* Related tests/coverage
* Results
* User research
|-
| SpeedIndex || Visual || (SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better. ||[https://developer.mozilla.org/en-US/docs/Glossary/Speed_index MDN Documentation]
|-
| || || ||
|-
|}

Auto-tools/Projects/Pulse

2020-02-24T14:33:00Z

Kmoir: /* To upgrade a ssl certificate on pulse.mozilla.org */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

dustin and the taskcluster team have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

* https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

Cloudampq updated their web page since we last did this so that you should be able to upload the cert yourself and have it propagate. See the admin console under "Certificate".

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

MozWeek/Berlin2020

2020-01-08T15:53:23Z

Kmoir:

'''What is it?''' -- Multiple team meetings, happening in the same city, at the same time + some opportunity to get together as one big group as well as with other teams as it makes sense. Then, on the last day, we have a fun social event for all, Mozilla-style!

'''''The information on this wiki primarily applies to Full time and contractor staff. If you are a volunteer contributor please inquire with your community manager/program manager about [[All_Hands/Contributor_nominations|contributor nominations]], interns please inquire to your coordinator. Volunteer contributors who have recieved an invitation can find relevant information [[All_Hands/Berlin2020/contributors|here]]. '''''

=='''Dates and Location'''==
Monday, January 27, 2020 - Friday, January 31, 2020 (travel days are Monday, Jan 27 & Saturday, Feb 1) in Berlin, Germany.

We have guestrooms at three hotels: [https://www.ihg.com/intercontinental/hotels/us/en/berlin/berha/hoteldetail InterContinental Berlin], [https://www.pullman-berlin-schweizerhof.com/en/ Hotel Pullman Schweizerhof], and [https://www.hotel-berlin.de/en/ Hotel Berlin, Berlin]. [https://drive.google.com/open?id=1eu_rYMxuc_hgSNqCJxW7Vn32pYMNayPa&usp=sharing View on map].

''*For those countries where rest time is required on weekends (vs. work travel), Mozilla will cover a return on the next available work day, if you choose. This needs to pre-approved and pre-arranged.''

=='''Registration'''==
This is an invite-only event. All full time and Elance/Upwork employees are expected to attend this all company event. Contractors, Vendors and seasonal employees are included on a case by case evaluation based upon team needs and upon executive review and approval.

Advance registration is required. Attendees will need to wear their event badge at all times, including to evening events. We will have security at our events who will be ensuring everyone in our space should be there. This includes facilitators, and other related business guests.

=====New Hires=====
We have a process to identify qualified regular full time new hires and Brianna will invite all new hires to register and book travel directly on their start date. No action is necessary from hiring managers other than to let them know about the event (please do not forward any links). Please work closely with your recruiting manager as they are aware of all deadlines.

All regular full time new hires must have a start date of January 6, 2020 or earlier and in workday by December 13, 2019. Final invites were extended on December 16, 2019. Contractors/vendors required executive approval and those who are hired after the approval period will not be invited, please see process/timeline [https://docs.google.com/document/d/1lbxsJwkorleL6tchh5tL_yCeFZmHz7Qm5Lumm7u7-JA/edit?ts=5d6950bd here]. If you have any questions, please email bmark@mozilla.com.

=====Volunteer Contributors=====
There are a number of policies that apply specifically to contributors that are different than for staff. Please look [[All_Hands/Berlin2020/contributors|here]] for all contributor specific instructions and policies.

'''Local contributors:'''

Please note that because of the extensive nomination and screening process we apply to all contributors who attend the event - we are unable to invite local contributors or temporary participants to the event outside of the regular nomination and screening process.

=====Registration Changes=====
Everyone who registered received a confirmation email titled "Registration Confirmation to Mozilla Berlin All Hands" which shows you how you answered each question. If you didn't get the email and/or need to make changes to your registration, please email mozilla@shworldwide.com.

=='''Immigration'''==
If you have any questions about immigration, please email immigration@mozilla.com.

==== STEP ====
'''What is STEP?'''
The [https://step.state.gov/ Smart Traveler Enrollment Program (STEP)] is a free service to allow U.S. citizens and nationals traveling abroad to enroll their trip with the nearest U.S. Embassy or Consulate.
Benefits of Enrolling in STEP
* Receive important information from the Embassy about safety conditions in your destination country, helping you make informed decisions about your travel plans.
* Help the U.S. Embassy contact you in an emergency, whether natural disaster, civil unrest, or family emergency.
* Help family and friends get in touch with you in an emergency.

==== Visas ====
You do not need a visa if you hold a passport issued from the EU, USA, Australia, New Zealand, Taiwan, or Canada.
Please see here to confirm the entry document requirements for the country you’re from: https://www.auswaertiges-amt.de/en/einreiseundaufenthalt/visabestimmungen-node/staatenlistevisumpflicht-node

If you are from a country that requires a visa to enter Germany, please plan to obtain one as early as possible as government processing times constantly change. You can find a broad overview of the process here: https://www.auswaertiges-amt.de/en/einreiseundaufenthalt/visabestimmungen-node . For more detailed instructions regarding the visa process you will need to refer to the website of your nearest Mission/Consulate General. You will be able to learn more about the process by searching for “business visa” or “visitor visa” on the website for that German Mission.

'''Please keep in mind that while you are traveling for work, you are not traveling for a German based job.''' All Hands is a business conference/event. As you book your travel please also keep in mind that you could require a transit visa if you have a layover in a country that requires it.

If you have questions specific to your circumstances, please email immigration@mozilla.com.

==== Entering Germany ====
Your passport or identity card will be checked when you arrive at an EU port or airport to make sure you’re allowed to come into the country. The passport must be valid for the whole of your stay. You may also need a visa to come into or travel through Germany, depending on your nationality.
* If you are from an EEA country or Switzerland, you can enter Germany with either a valid passport or a national identity card issued by a EEA country.
* If you need a United States Passport, start here: https://travel.state.gov/content/passports/en/passports.html. Please note it can take 4-6 weeks to receive your passport. Please plan ahead.

==== EU Visa Waiver Program Suspension - Update ====
In March 2019 the EU announced that the visa free entry requirements for third country visitors to Europe would be changing. This change will not go into affect until 1 January 2021, note this is one year after our Berlin All Hands trip. After this date, citizens from certain countries, including the USA and Canada, will be required to obtain an ETIAS travel authorization before they can enter Europe.

==== More details: ====
Mission websites in popular Mozillian locations:
* United States: https://www.germany.info/us-en/service/05-VisaEinreise/business-visa/963542
* Canada: https://canada.diplo.de/ca-en/consular-services/visa/shortstay
* UK: https://uk.diplo.de/uk-en/02/visa
* Paris: https://allemagneenfrance.diplo.de/fr-de/service/00-visa-seite
* Taiwan: https://taipei.diplo.de/tw-en/service/visa
* Beijing: https://china.diplo.de/cn-zh/service/visa-einreise/schengenvisum/1343246
* India: https://india.diplo.de/in-en/service/-/1908794

=='''Travel to Berlin'''==
Bookings to Berlin are done. The Egencia instructions were emailed to '''anyone approved to attend'''. New hires have individual deadlines based upon hire/start date. Approved contractors, vendors, seasonal, status employees were notified with an email from bmark@mozilla.com with specific instructions.

Specific instructions were about how to book, as we have a new process for anyone based outside the US, Canada & Central/South America. APAC employees will '''NOT''' be booking your All Hands travel in country based site, and will be using Egencia Americas. European employees '''WILL''' be booking in your country based site to allow access to trains and low cost airlines.

Should you need to reach someone at '''Egencia Americas''' for special circumstances or changes that can't be done online:
*Call+1 (800) 361-1120 or +1(702)939-2533; Hours of operations are Monday – Friday (except holidays) 5:00 AM– 6:00 PM PT or 8:00 AM – 9:00 PM ET or 12:00 PM - 1:00 AM UTC. If you call outside these hours, you will get an after-hours agent (not a meeting agent, who may not be as helpful) . Please ensure the agent you speak with knows you are booking for a meeting.
*Email: groupagents@customercare.egencia.com. The email is monitored Monday - Friday.

Other Egencia contact details:
*Egencia UK+EMEA - Email: customer_service@egencia.co.uk; UK Local #: +44 203 077 2536 (charged at local rate). For calls made outside the UK, please dial +00 44 161 233 5525. (From anywhere outside the EU, please dial +011 44...)
*Egencia FR - Email: service_client@egencia.fr; France Local #: +33 8 11 65 66 53. For calls made outside France, please dial +33 4 86 06 15 18 (From anywhere outside the EU, please dial +011 33)

'''Important Details:'''
Everyone should plan to arrive in Berlin Tegel Airport (TXL) on Monday, January 27 and leave on Saturday, February 1. Anyone who plans to arrive ahead of that or to stay longer, regardless of the reason, must follow the "arriving early/departing late guidelines." These guidelines also apply if you want to fly in or out of airports other than your home airport and Berlin Tegel (anything custom). A few notes:
*If you are a vendor (paid by a company other than Mozilla or Upwork), contractor, or seasonal staff, those approved will be sent separate email.
*Volunteers will be handled separate from this process.
*We work closely with Recruiting to ensure new hires are invited after their start date. No need to let us know about these.
*If you would like to invite someone who is a business partner, facilitator or some other category that isn't an employee or volunteer (not a personal guest/family), email bmark@mozilla.com.
*Change fees and changes in airfare will be covered by Mozilla for business reasons only. Please email bmark@mozilla.com ahead of calling Egencia to make the change. If the change is specific for all hands, payment can be made centrally. If a change is due to a team specific need, you will need to provide payment over the phone and submit for reimbursement with manager approval. Any changes for personal reasons must be paid by the employee directly to Egencia over the phone (see next point)
*If you'd like to change your itinerary for personal reasons, please call Egencia directly (numbers above( and be prepared to provide payment for the change fee and any change in airfare over the phone.

====Arriving Early/Departing Late Guidelines====

Our standard travel guidelines apply (pre-populated in Egencia) when booking with a few additional budget constraints. Anything booked outside of them will require approval. Most people will arrive on Monday, January 27 and leave on Saturday, February 1. Here are some exceptions:

# If you plan to spend some extra '''personal time in Berlin/Europe''' or nearby (choosing to arrive before Monday, Jan 27 or depart after Saturday, Feb 1), you'll need to create a itinerary in Egencia for standard dates/locations within the Egencia Portal and compare to the custom dates/locations you'd like. Pricing for the standard dates should be ''Round Trip only''. Pricing for custom dates/locations should be for ''Round trip or Multi city trips only''. Please share the difference via email to bmark@mozilla.com and receive approval ahead of submission in Egencia. You can ''sway up to +$100 over'' and Mozilla will cover it. Otherwise you'll need to come with an alternate itinerary that fits within the pricing (like a round trip in and out of TXL w/ longer dates, and you personally book & cover the rest). Please note that one way trips are only approved on a case by case basis. Do not book one ways without prior approval. ''We '''do not''' have the ability for employees to reimburse Mozilla for any overage.'' This scenario also applies for routing through different airports to/from Berlin Tegel than your home airport (ex. a layover in London for a few days, or flying out of something other than TXL).
# If you are attending the '''Sunday/Monday CI event''' (by invite), you can arrive on Sunday, January 26.
# If you would like to arrive early to '''recover from jetlag''', you will need manager approval for any additional costs associated with the extension. There is '''no unilateral''' "All Hands" approval based upon timezone to arrive early. For hotel, you will book and pay on your own, and expense the manager approved amount (which is coded to your cost center). This policy does not apply to volunteer contributors, any jetlag recovery costs must be self funded.
# If you are celebrating the '''Lunar New Year''', [https://wiki.mozilla.org/All_Hands/Berlin2020#Lunar_New_Year see here].
# If you live in a country where '''work travel is prohibited on weekends''', you may travel on Friday, Jan 24 and Monday, Feb 3, if you’d prefer (not required). For hotel, you will book and pay on your own, and expense the manager approved amount (which is coded to your cost center). You must have approval from our benefits team in writing (and send to bmark@mozilla.com) '''prior''' to booking any travel. Approvals and expenses will not be applied retroactively.
# If you live in a country where '''work travel is prohibited on Sunday''', you may travel on Saturday, Jan 25, if you’d prefer (not required). For hotel, you will book and pay on your own, and expense the manager approved amount (which is coded to your cost center). You must have approval from our benefits team in writing (and send to bmark@mozilla.com) '''prior''' to booking any travel. Approvals and expenses will not be applied retroactively.
# If you'd like to make changes to extend your stay on either side after booking, you will be responsible for all change fees and change in airfare.

====Booking Family Travel====
Whether your family will accompany you on your flight or join us later; and you have two options: direct with the airline (recommended) or through Egencia.

'''Direct (recommended):'''

First, figure out which flights you want to be on as an employee in Egencia. Hold them in Egencia. Find the flights you want for your family on the airline site and book them. Go back to Egencia and book your flights. Once you have both sets of confirmation numbers, call the airline and ask them to LINK your two reservations. This shows that you are traveling together and shouldn't be moved. When you call, you can also ask the airline agent to assign seats together. By booking your family direct on the airline, it gives them "priority" with the airline over people who book via a third party (expedia, booking.com, kayak, etc). These aren't guarantees, but they do help.

'''Via Egencia:'''

We do not recommend booking family through Egencia. If you'd like details about how to do it, please email bmark@mozilla.com

'''Notes:'''

* If your family is arriving on a different flights than you but would like to take advantage of the airport shuttles on Sunday/Monday and/or Saturday, please email their itinerary to mozilla@shworldwide.com.
* We are unable to accommodate volunteer contributor families/guests.
* As of 20 November, we only have rooms that can accommodate 2 adults. If a larger room is needed, it will need to be booked and paid for on your own.

====Not Flying====
Mozilla will reimburse U-Bahn tickets for those based in the Berlin area for getting to/from hotels. For more information on the U-Bahn, please see the Berlin office guide on [https://mana.mozilla.org/wiki/display/WPR/Berlin+Office+Guide#BerlinOfficeGuide-MovingAroundBerlin mana].

For those outside of the Berlin area who are taking the train, please book by November 10. Instructions were emailed to everyone.

Parking is not reimbursable.

====Travel Insurance====
Mozilla provides emergency medical accident and illness cover for all global MoCo employees/interns and their dependents. You can view more information on [https://mana.mozilla.org/wiki/display/PR/Travel+Insurance+-+Business Mana]. This coverage begins at the time the you leave home to start your business trip. It also has a provision for a 14 day extension for leisure travel outside of the business travel. If you have additional questions, please email benefits@mozilla.com.

Mozilla does not cover travel insurance for elancers, upworkers, contractors, vendors, or volunteers/community members (unless required for visa application).

====Air Travel Fine Print====
*Change fees will be covered by Mozilla for '''business reasons only'''. If you need a change and have manager approval, email bmark@mozilla.com '''prior to requesting the change with Egencia'''. Once you have approval, call Egencia to make the change at 1-877-255-1090 (note this will not be possible without prior approval so be sure to get that by way of an email from your manager to Brianna Mark). If you are changing for personal reasons, the change in airfare, change fee and Egencia fee is your responsibility.
*Mozilla will '''not''' reimburse for Business/First class upgrades, tickets or seat reservations.
*Mozilla will reimburse for one checked bag each way.
*''Mozilla Frequent Flyer perks do not apply for All Hands''.
*Flights booked outside of Egencia will not be an approved expense unless email approval by Brianna is obtained prior to booking.
*Any submitted expenses needs to have an itinerary attached to ensure it is employee expenses only and within policy.

====Ground Transportation====
=====Shuttles to hotels=====
Berlin Tegel airport is open from 4:00 - 0:00.
Shuttles will be provided from Berlin Tegel Airport (TXL) to our 3 hotels:
*Sunday, Jan 26 from 9:00 - 22:00, for flights landing between 7:45 and 21:45.
*Monday, Jan 27 from 8:00 - 23:00, for flights landing between 7:15 and 22:55.

Shuttles will drop at Hotel Berlin, Berlin and InterCon Berlin. Those staying at Pullman will get off at InterCon and walk across the street.

'''Arrival at Terminal A:'''
Once you have arrived at Terminal A, you will pick up any checked luggage at baggage claim located near your arrival gate. Once you have your checked luggage you can exit the secure area of Terminal A. There will be staff holding Mozilla signs & wearing bright blue Mozilla shirts directing you to the shuttle. You will exit the main hall of TXL and walk past the public buses to the bus parking area where the Mozilla shuttle will be. The shuttle will depart every 30 minutes (approx on the :00 and :30 each hour).

Terminal A Airlines: Aegean, Air Canada, Air France, All Nippon Airways, American Airlines, Austrian Airlines, British Airways, Brussels Airlines, China Southern Airlines, Delta, Etihad Airways, Finnair, Iberia 3672, Icelandair, KLM, LOT-Polish Airlines, Lufthansa, Qantas Airways, SAS, Scandinavian Airlines, Singapore Airlines, Swiss, TAP Portugal, Turkish Airlines, United

'''Arrival at Terminal C:'''
Once you have arrived at Terminal C, you will pick up any checked luggage at baggage claim located near your arrival gate. Once you have your checked luggage exit the secure area of Terminal C and turn right. There will be staff holding Mozilla signs & wearing bright blue Mozilla shirts directing you to the shuttle. After exiting Terminal C you will walk down an alley that leads to an elevator & staircase, take this to the upper level and cross over the covered bridge. At the end of the bridge you will be in front of the main TXL terminal. Walk past the public buses to the bus parking area where the Mozilla shuttle will be. The shuttle will depart every 30 minutes (approx on the :00 and :30 each hour).

Terminal C Airlines: Aer Lingus, easyJet, Iberia, Ryan Air, Vueling Airlines

Arrivals at other terminals should follow directions to terminal A/public bus locations and will see staff in bright blue shirts. Terminal D & E: Eurowings
Map of Airport: [[File:MOZ MAH TXL map v1.pdf|thumb|Map of Berlin Tegel Airport]]

=====Shuttles to airport=====
Shuttles from our 3 hotels to Berlin Tegel will be provided:
*Saturday, Feb 1 from 4:00 - 20:00, for flights departing 7:15 and 22:15.
*Sunday, Feb 2 from 4:00 - 12:00, for flights departing 6:00 - 14:00. ''If you are on a later flight, you can take the last shuttle at 12:00.''

Those staying at Pullman will walk to InterCon hotel for shuttles to airport.

=====Self organized transportation=====
If you are arriving outside of shuttle hours, transportation is on your own (both in booking and paying) and is not reimbursable. Taxis and Addison Lee are not reimbursable.

=====Train=====
If you are arriving by train to Hauptbahnhof:
To Hotel Berlin Berlin
* FROM HAUPTBAHNHOF STATION: Take the S-Bahn metro lines S5, S9 or S75 to the station “Zoologischer Garten” (20 min.). From there, take the bus 100 to the stop Lützowplatz.
* Via U-BAHN: Take the U-Bahn metro lines U2, U3 and U1 to the station “U-Nollendorfplatz”. Exit the station in the direction of “Karl-Heinrich-Ulrichs-Straße." The road leads directly to the hotel (approx 5 mins).
* [https://www.hotel-berlin.de/d/berlinberlinpro/media/PDF/Anfahrt_A4_E_Karl-Heinrich-Ulrichs-Strae_5c2ab7.pdf Map]

To InterCon Berlin and Pullman
* FROM HAUPTBAHNHOF STATION: Take the S-Bahn metro lines S3, S5, S7, or S9 to the station “Zoologischer Garten”. From there, walk 10 mins to hotel.

Berlin's integrated public transport is divided in 3 tariff zones (ABC). To travel within the city, including from Tegel airport to the hotels, you only need an AB ticket. Tickets can be bought at automated booths in most stations (accept credit cards), and on Metro (need coins). You can use the same ticket to travel on buses, metro, S-Bahn, U-Bahn. More information, including PDF maps, are available on [https://www.berlin.de/en/public-transportation/1772016-2913840-tickets-fares-and-route-maps.en.html this website]. A mobile app is available on stores ("BVG FahrInfo Plus") and allows to create routes between stations, or you can use directly the [https://www.bvg.de/en BVG] website.

====Getting around Berlin====
You’ll rarely hear from your Berlin-based colleagues (or anyone else actually living in the city instead of the suburbs) that they prefer or even enjoy taking the car anywhere. Many don’t even own a car unless they truly need it because driving in Berlin is not a lot of fun with busy traffic, lots (looooots!) of construction sites, a bunch of one-way streets and barely any free parking spaces. When exploring the city, we recommend that you use public transportation instead, which takes you everywhere and is pretty reliable. Or you rent bikes and even add some exercise to your city tour.

In order to make sure that you know your transportation options, here’s an overview of what you may use — and how.

=====Buying tickets=====
Most types of public transport require you to buy and validate your ticket before entering the respective vehicle. You can find ticket machines at the stations or buy a ticket at shops in one of the larger train stations. The machines usually inform you in German and in English, which makes them easy to use.

You’ll be able to buy different types of tickets depending on where you go and duration of their validity, from just one ride up to 6 days. If you plan to use public transportation for several days in a row it’s definitely recommended to get a multi-day ticket because the relative price will be lower.

Of course, there are also some peculiarities to each type of transportation when it comes to paying the fees:
U-Bahn & S-Bahn: After buying a ticket at the machine, which you’ll most likely find on the platform or, if it’s a larger train station, in the entry area, you may need to validate it at an extra machine. They’re usually located on the platform, as well. [https://welcome2berlin.weebly.com/uploads/1/2/1/2/121289663/s-bahn-ticket-machine-00-img-6428-1920x2560-otmc-10-90_orig.jpg Here's a picture of a ticket machine with the ticket validation machine to its left.]
Bus: Most bus stations don’t have ticket machines but you can buy one from the driver.
Tram: Some trains have ticket machines inside; however, they don’t always work.

Please also take into account where you’re actually planning to go when buying a ticket. Berlin has 3 fare zones: A (city center), B (wider Berlin city area) and C (outside of the actual Berlin city area; Brandenburg). Unless you’re only exploring the area around the All Hands site, you’ll probably want to get “AB” tickets. If you need to go to or from Berlin Schönefeld Airport (SXF), it is located in fare zone C, so you'll need a ticket that covers area “C”. If you have an “AB” ticket you can buy an extension ticket for a single trip.

=====U-Bahn (the subway)=====
The “U-Bahn” is our local subway system, which connects the entire city area, taking you almost everywhere within just a few minutes. There are 10 lines overall, named U1 to U9 (plus the kind of odd U55 that only connects very few stations). Big “U” signs all over Berlin help you spot the next station.

The BVG (Berliner Verkehrsbetriebe, the Berlin transport company) offer a fully interactive route map on their website. If you prefer a printed version, you can pick one up for free at one of their customer centers; the main office is located in the subway station “Alexanderplatz” in the city center. Or you just download their app, which is available for all major OS. You can also buy tickets via the app.

Pro tip: You may want to download the Berlin map in your favorite navigation app. Given that your phone plan may not apply to Europe (or it’s ridiculously expensive to go online), this can actually save your day because public WiFi isn’t as widely available in Germany as e.g. in North America.

=====Tram/Straßenbahn=====
The tram is Berlin’s overground train service within the city. There are 22 lines overall: 9 so-called MetroTram lines (names consist of “M” plus a line number) and 13 streetcar lines (where the name is a simple number). They complement the U-Bahn and S-Bahn system by running on important routes that are not covered otherwise.

Given the size of the Berlin city area, you’ll most likely need to switch between trains and types of public transport, which includes the trams. They’re usually easy to find since they’re comparatively loud, painted in shining yellow and, well, run on overground tracks through pedestrian zones and next to roads used by cars. Also, big red signs with the word “tram” on them guide the way.

=====S-Bahn=====
The S-Bahn is another overground train but covers longer distances than the tram (and doesn't run in or cross streets).

You’ll probably want to take the S-Bahn when visiting the neighbourhoods that are further out, doing a day trip to Potsdam or going to Schoenefeld Airport. If Berlin’s 3rd airport, BER, ever happens to open, transport from the city will also be covered by S-Bahn. It's also good for some trips within the city center.

=====Other trains=====
Are you planning a trip through Germany or even beyond? Then you may want to consider taking one of the trains of the Deutsche Bahn (German railway company).

Trains leave from the 7 long-distance train stations in Berlin. You can buy tickets online (bahn.de) or at the machines in each of the stations. If you’re planning a longer trip or want to travel during peak times, it’s recommended to buy your ticket in advance, add a seat reservation and take sufficient provisions for the trip since food and drinks at the “Bord-Bistro” are ridiculously expensive.

Pro tip: If you bought a seat reservation in advance and there’s someone sitting in your spot, don’t be shy and ask them to get up. It may be awkward but it’s totally normal. Also, the reservation goes away if you don’t claim your seat within a certain amount of time after getting on the train.

Even though it carries about two billion passengers annually, it’s unlikely that you’ll experience amazing service or punctuality on your Deutsche Bahn trip — which is at least a little bit funny given that it used to be owned by the government of a country that’s mostly known for its efficiency. Just kidding ;) German trains are quite comfortable to use, prices are okay and from Berlin, you’ll arrive in most other larger cities in Germany and the neighbouring countries within comparatively short times as the trains can go up to 300 km/h.

=====Busses=====
Just like in any other country, busses combine all of the inconveniences of public transportation and driving a car in the city: it’s crowded, you may need to stand for a pretty long time and it takes forever to get to your destination. Still, busses are used a lot (at least in Berlin) because they cover many short routes where there’s not S-Bahn, U-Bahn or Tram service.

When entering a bus in Berlin, operated by the aforementioned BVG, you’re theoretically required to show your ticket to the diver; some don’t really care while others are quite strict about it. Beyond that, there are a couple of additional rules you should be aware of before stepping into a BVG bus:
* Don’t stand anywhere close to a door unless you feel prepared to deal with the rage of the bus driver. They don’t mind yelling at someone in the very back of the bus or even interrupt their route until the respective person behaves as they’re expected to.
*This applies to your luggage just as much as to you as a person.
*Make sure to monitor the route closely. If you want to get off the bus you need to push the button to request a stop soon enough or stay on the bus until the driver finishes their tour.

Pro tip: If you get on a bus that isn’t too crowded let the driver know where you want to go and take a seat nearby. They’re not all grumpy. Some of them are actually quite friendly and helpful when you approach them directly.

=====Taxi & Uber=====
Uber has grown to become the most successful driving services in many countries but it is currently banned in Germany after its UberX service has been found in violation of German federal law prohibiting private citizens (e.g. without background checks) to offer transport services. Instead use one of the many (7500 in total) regular yellow licensed taxis. When calling a driver in Berlin you have several choices:

*Get a regular taxi by calling one over the phone (+49 30 20 20 20 OR +49 30 26 10 26), find a taxi stop, or stop a taxi on the street. Just make sure to have some cash on you as some of them don’t take credit cards. Remember, licensed Taxis are a light yellow ("light ivory", [https://upload.wikimedia.org/wikipedia/commons/e/e0/Taxis_at_EDDT-%28jha%29.jpg picture]).

*Call a regular taxi via the [https://www.taxi.eu/en/cities/berlin/ taxi.eu] app. This app (download: [http://itunes.apple.com/de/app/taxi.eu/id465315934 Apple iOS], [https://market.android.com/details?id=at.austrosoft.t4me.MB_BerlinTZBEU Android]) offers a similar service as Uber but gives access to the majority of Berlin's regular (and regulated) taxis (7200 of the 7500 total). The main difference is that you can either pay cash or need to be connected to the internet in order to pay through the app at the end of your ride: while you agree on a price with your Uber driver before getting into their car, taxi.eu will charge you for the actual distance/time it took to get you to your destination, which isn’t clearly determined before your arrival and only an estimate is shown during booking.

=='''Hotels'''==
Your hotel assignments/confirmations have been emailed coming from bmark@mozilla.com, not the hotel directly. Please make sure to read and check for accuracy.

In each email, you will find:
*Your assigned hotel
*The dates that have been booked for you
*Links to book extra nights beyond what has been booked for you (links expired Dec 20, 2019)

A few things to note:
#If you have any changes or questions about your reservation, email mozilla@shworldwide.com. The hotel cannot make changes to All Hands reservations so we’d like very much if you didn’t try (it complicates things).
#If you registered that you have guests joining you for all or part of the week:
*You will be responsible for covering all additional fees at the hotel.
*All hotel rooms have a maximum capacity of 2 adults (with some exceptions limited to very small children)
*If you registered by October 13:
*Guests have already been added to your hotel reservation on your behalf. You have been assigned a room that can accommodate you and your guests. Please refer to the total occupants listed in your hotel confirmation.
*If you need to add a guest, please email mozilla@shworldwide.com. Please note: all hotel rooms that can accommodate more than 2 adults have been alloted. If a larger room is needed, it will need to be booked and paid for on your own.

If you are planning to book pre/post stays and didn't complete by the deadline of Dec 20, you can book directly with the hotel online. We can not guarantee any rates or room type availability. Please use your legal name and use LDAP email so the hotel can try to match reservations and room types. Reservations booked through third party sites (hotels.com,. etc) can not be linked to all hands reservations.

Everyone will be required to present a form of payment on check-in for incidentals. For your own sanity, please do not provide a debit/cash card. If you are unable to provide a credit card, please email mozilla@shworldwide.com and we can request special accommodations.

=='''Families/Guests'''==
Of course our focus, for the majority of the week, will be on Mozilla. Everyone is expected to be present and engaged each day, during work hours (as your schedule dictates). Please do what you can to make sure your loved ones understand the kind of commitment you’ve made. Please note that what are able to do for families varies by each location. We are unable to accommodate volunteer contributor or intern families/guests at our All Hands.

====Quick summary logistics====
* Air Travel: Employees do need to book via Egencia regardless of how families are booked.
* Hotel: Family/friends are welcome to stay with you. All room rates are based upon single occupancy and include breakfast. Guests must be added to reservations in advance and any additional room expenses will be yours to cover. Most of our hotel rooms have a maximum of double occupancy (2 adults), with very limited availability of rooms for higher occupancy. Higher occupancy rooms will be assigned in the order that you register. As of 20 November, we only have rooms that can accommodate 2 adults. If a larger room is needed, it will need to be booked and paid for on your own.

Additional Guest fees
{| class="wikitable"
|-
! Hotel !! Hotel Berlin !! InterCon !! Pullman
|-
| 2nd guest fee (incl bfast) || 20,00 € || 25,00 € || 24,00 €
|-
| 3rd guest fee (incl bfast) || 50,00 € || N/A || N/A
|-
| 4th guest fee (incl bfast) || 50,00 € || N/A || N/A
|-
| Age applicable || 6+ || All || All
|}

* Lunch & Dinner: Family will be on their own for lunch daily and dinner on Tuesday, Wednesday & Thursday.
* Monday & Friday Night Events: Family members who are registered for the event and registered with the hotel, are invited to join us for our evening events.
* Childcare Services: We will not provide centralized childcare services. We have found that every family has different needs and centralized childcare does not deliver on those needs. If you are in need of help finding options for your family, we are happy to connect you with staff whose families who are coming (#allhands-childcare and #parents are great places to start), as well we are happy to help support finding of resources (although you are likely better at it than us). Given every circumstance is unique, we defer to your manager to determine what Mozilla can support you with financially.

On demand babysitting services in Berlin:
* https://extra-arms.com/for-families-babysitting/
* https://www.withourbaby.com/

====Exploring Berlin with Kids====
Berlin is a great city to explore with kids as it offers lots of fun, interesting and educational activities that are even suitable for the cold weeks of January. Which brings us to a very important recommendation we want to share right at the beginning of this section: dress your kids up appropriately to the weather but focus on layers. The onion look may not always be super fancy but both you and your kids will appreciate it: Since it can get unpleasantly cold in the Berlin winter, public places and attractions tend to heat up their indoor offers. Wearing layers makes sure that your kids neither get cold outside nor sweaty when going inside.

=====Inside activities=====
*The Deutsches Technikmuseum (German Technical Museum) is a great place to spend time on a cold January day. It’s exciting for both adults and kids and offers a lot to explore. Discover the cultural history of technology by yourself! Also, you can get there from the All Hands location in just 20 minutes by public transport.
Deutsches Technikmuseum, Trebbiner Straße 9, 10963 Berlin-Kreuzberg
Opening hours: TUE-FRI 9am-5:30pm, SAT+SUN 10am-6pm
*MACHmit! museum für Kinder is a museum like no other: Endless opportunities to discover and try new things encourage children to learn through play, whilst also gaining new and exciting experiences that they would not come across in an average day. Arts and crafts activities, lots of running and being loud and for parents a nice area where you can get coffee and snacks.
MACHmit! Museum für Kinder, Senefelderstraße 5, 10437 Berlin
Opening hours: TUE-SUN 10am-6pm
*The Aquarium Berlin in the center of Berlin is one of Europe’s best-known and most notable aquariums. Behind the building’s historic façade awaits an impressive diversity of species that few facilities in the world can rival. The Aquarium not only houses numerous extraordinary fish, it is also home to hundreds of impressive reptiles and insects. Also, it’s just a 5 minutes walk from the All Hands location.
Aquarium Berlin, Budapester Str. 32, 10787 Berlin
Opening hours: Daily 9am-6pm
*View the night sky without any clouds disrupting your view at the Zeiss-Großplanetarium! The largest planetarium in Central Europe was inaugurated in 1987. The Planetarium offers lots of events for young and older kids and is just impressive to be in. You should check it out!
Zeiss-Großplanetarium, Prenzlauer Allee 80, 10405 Berlin
Opening hours: TUE+SUN 9:30am-6pm, WED+THU 9:30am-9pm, FRI+SAT 9:30am-10:30pm
*Tropical Island: Taking a break at Tropical Island means swimming, diving or just relaxing by the Tropical Sea or Lagoon, exploring the world's largest indoor rainforest or getting your adrenaline pumping on Germany's highest water slide tower -- plus lots of other great attractions. Disclaimer: It’s crowded, so if you are not so much into big groups of people, this might not be for you -- but kids LOVE IT. You can easily reach Tropical Island within a one-hour train ride.

=====Outside activities=====
You and your kids don’t mind the cold too much? Great! There are plenty of nice things to do outside even the winter. As the train services run pretty well and Berlin is well connected even to the rural areas, you have a lot to choose from and it’s pretty easy as well as inexpensive to get there.
*Just a few minutes outside of Berlin you’ll find Karls Erlebnis-Dorf (Karl’s Adventure Village) in Elstal. In addition to offering a spacious farmers market, many attractions for children are waiting for you, including a big wooden roller coaster. Admission is free. You can get there by car, bus and train.
Karls Erlebnis-Dorf, Döberitzer Heide 1, 14641 Elstal
Opening hours: Daily 8am-7pm
*Tempelhofer Feld is a big recreation area in the Berlin districts Neukölln and Tempelhof. It’s actually the largest inner-city open space worldwide, Berlin’s most spacious urban park as well as part of larger historical project. Throughout the year, the park is used for outdoor sports like jogging, biking, roller-blading, and kiting, but is also a popular spot for wild gardening and barbecues. The Mozilla Berlin Family Day 2017 took place right there.
Accessible via 10 entrances
Opening hours: From sunrise to sunset

=='''Lunar New Year'''==
We are aware of the overlap in the Lunar New Year celebrations with the Berlin All Hands. To help folks who decide to attend the Berlin All Hands and who also celebrate Chinese New Year, we will offer the following:
* A higher airfare caps for air travel to allow for fewer layovers on Monday, January 27
* Assurance that Berlin is optional but if employee wants to attend they can shift their holiday days to another time.
* Live Airmo Feeds of Plenary (+ relevant team all hands meetings) + quick posting of recording to accommodate those who choose not to travel.
* Potentially shifting any main plenary session date/time

We are working with leadership and representatives of those most affected and will update wiki regularly with updates. Feel free to reach out to Brianna with questions.

=='''Week at a Glance'''==
View [https://docs.google.com/spreadsheets/d/1kIwKEmlYE9tMuWRvSNZnYDGWFwEumcHVp1s0LVPz-e8/edit#gid=0 Week at a glance]

====Monday====
*Monday is travel/arrival day
*Registration
*Welcome Reception from 6:00 pm - 9:00 pm

====Tuesday====
*Breakfast in hotel restaurants
*Lunch in hotels of your team homeroom
*Dinner on own

====Wednesday====
*Breakfast in hotel restaurants
*Lunch in hotels of your team homeroom
*[https://berlinallhandsjanuary2020.sched.com/event/YKzH/demos Demos] from 4pm - 6pm ([https://wiki.mozilla.org/All_Hands/Berlin2020/Demos full list])
*Dinner on own

====Thursday====
*Breakfast in hotel restaurants
*Lunch in hotels of your team homeroom
*Dinner on own

====Friday====
*Breakfast in hotel restaurants
*Lunch in hotels of your team homeroom
*Closing event at Motorwerk

====Saturday====
*Breakfast in hotel restaurants
*Departure day only. No scheduled activities.

===Berlin All Hands Event Calendar===
Here: https://berlinallhandsjanuary2020.sched.com/.

Don't see stuff for your org yet? Don't fret! The schedule changes regularly as meetings and events are confirmed. Keep checking back. Team specific events are managed by [https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=EAs&title=Mozilla+EA+Directory team EAs].

=====Create an account=====
We don’t recommend using the same email & password as anything like bank accounts, etc. We care about your security! If you already have a Sched account from past All Hands, it still works, log in with that.

=====Add items to your calendar=====
Select the circle on any agenda item to add it to your calendar (you do need to have an account & be logged in to do this)

You can also share a link to meetings to invite others. Go into the meeting and copy the short link. You can email that out to anyone and they can quickly add it to their calendar.

=====Subscribe to GCal Calendar Link=====
Click on the mobile phone on the right hand side of the screen. All the calendar options are available here.
You have the option to choose ALL meetings or YOUR meetings. Unless you have 400 items on your calendar, just select your calendar. It will add anything on your calendar to your GCal (also an option for Outlook and iCal). It syncs once per day.

The "only syncs once per day" only applies to Google Calendar. With almost all other clients (like Apple Calendar, Outlook, or the calendar app on your phone) you can set the refresh interval, and Sched's instructions recommend 1 hour. Note - the gcal function has been buggy for some users in the past. If it doesn't update with new items, delete the calendar and download again.

Warning: This is a link that utilizes your username for the .ics file.

=====From Mobile=====
Visit from any mobile device (https://berlinallhandsjanuary2020.sched.com/mobile-site) - bookmark or add to your homescreen for quick access. There is a bonus icon you get by doing this. It caches the last time you opened the page offline and refreshes anytime you are connected.

=====Filters=====
'''Filters'''

We have filtering functionality. You can filter by:
Departments (ex: People)
AND
Functional Teams (ex: People & Capabilities)

*Search by Room, Speaker/Leader

'''Further Filtering'''
*Audience - who should be there (ex: Team only or Invite)
*Homerooms (you can quickly see what is happening in homerooms, by team) - why do you care? If you have a cross team meeting in their room, its a quick way to search
*Views - Lots of view options. It defaults to the simple view, but there are quite a few options.

Have a tip? Feel free to add it!

=='''Food, Drink & Events'''==
Lunch & snacks will be provided and paid for centrally for attendees. Breakfast is provided Tuesday - Saturday as part of your guestroom reservation - you must eat breakfast in the hotel you sleep in. You will not be reimbursed if you go to another hotel and pay to eat there.

Allergies/preferences: We will ensure that all food/environmental allergies are taken into consideration and will always have gluten-free and vegan options. If you have severe allergies that we need to know about; you can indicate in registration.

===Hosted Evening Events===
We have evening events on the Monday and Friday nights.

====Monday Night Welcome Reception====
6 pm - 9 pm, InterCon hotel

====Friday Night at Motorwerk====
6:00 pm - 11:00 pm

======Getting to/from the event======
Shuttles depart from the hotels starting at 5:00 pm. Shuttles will return starting at 8:00 pm. The last departure returning will leave at 11:05 pm. The only way to access the event is via our shuttles.

=====Coat Check=====
Coat check will be available for clothing only, no bags.

====Tuesday, Wednesday & Thursday Night Dinners====
You'll be on your own for Tuesday, Wednesday & Thursday and will have a similar expense policy from past all hands (50EUR/night - 150EUR total to include food, beverage, transportation, exchange fees, etc).

Here is how this will work:

For all evenings, once your meetings have concluded, you and your team, friends, new acquaintances, are free to explore and to find somewhere great to eat that suits you. Each of you can expense a total of 150EUR over the three days (or 50EUR/night).

This amount includes:
*Meal cost, including tax & gratuity
*Any beverages
*Transportation to/from the restaurant
*Conversion fees (for credit cards) or cash withdrawal fees

Anything over the 150EUR for the three evenings will be your own expense. The fine print:
*If your team is hosting an evening event 1 of the nights and the payment is coordinated (meaning, you don’t have to open your wallet and pay), you can expense up to 100EUR for the other night.
*You will be asked (later) to submit a Berlin All Hands only expense report. You can submit ONE report for Berlin only and must be submitted no later than February 28, 2020.
*No expenses over 50 EUR per night will be [https://docs.google.com/document/d/1XUD5KYotSr5SNi0q4ldgGG3bzBu-G3q4l5i-JiPE5ko/edit approved].

Volunteer Contributors will have a separate process that will be communicated directly.

=====Groups, Restaurants & Reservations=====
*Group dinners - All large (eight or more people) reservations must go through Lisa Carlson (email lcarlson@mozilla.com) and dinners must stay within the per diem per night (50EURpp including, food, beverage, tax, gratuity and transportation), and employees attending must not also expense their per diem for that night.

In no case should employees expense or use MoCo corporate cards to cover purchases of alcohol outside of team dinners or the 50EUR individual per diem.

=='''Safety & Security'''==
=====Alcohol at Events=====
To better support and sustain an environment (and workplace culture) where people feel safe and included, we have made a set of changes regarding alcohol at our events. In all cases, our approach aligns with our [https://www.mozilla.org/en-US/about/governance/policies/participation/ Community Participation Guidelines] (“CPG”).
*All participants are '''required to read and acknowledge our new Community Participation Guidelines as a condition of participation'''.
*We will '''limit bar-servings to beer and wine''' and ensure an equal number and quality (i.e. not just Coke) of non-alcoholic drink options are available and displayed.
*Team dinners should be '''thoughtful about the potential exclusionary nature of alcohol when planning'''.
*Clearly outlined, communicated (to event teams, HR and managers) and understood '''escalation process''' for behavior that might be deemed counter to the spirit of our CPG.

=====Device Security=====
If you are traveling to the All Hands with a device that has Mozilla data (laptop, personal cell phone/tablet with @mozilla gmail, etc) on it and your device has been retained for further inspection by border agents, or if your device has been inspected outside your immediate presence - and you believe your credentials have been compromised - you must notify the Enterprise Information Security team as soon as possible at infosec@mozilla.com or by calling Mozilla End User Services at +1 650-963-8811. (This number will be staffed 24x7)

We will work with you to reset your credentials and help you get your device back to a known good state either by getting you a new one (if it’s been taken), or by resetting it back to a verifiable Mozilla-approved installation.

=====Physical Security=====
Badges are required to access all meeting spaces and evening events.

=='''Accessibility'''==
====Evenings====
Monday Night reception is at the InterContinental Berlin. No stairs or elevators required.

====Meeting spaces====
* InterCon: Most meeting space levels are accessible by elevators.
* Pullman: All meeting space levels are accessible by guestroom elevators.
* Hotel Berlin: All meeting space is on the ground level.

====Listen Systems====
We will have Listen Systems available for any meeting in the main plenary space. Visit the AV booth to pick one up.

====Bathrooms====
TBA

====Breastfeeding/pumping stations====
Milk Stork Services are reimbursable - please contact benefits@mozilla.com for more information.

====Public transit====
Useful info for transit accessibility: https://wheelchairtravel.org/berlin/public-transportation/

=='''Sustainability'''==
'''Hotel Berlin, Berlin''' is certified partner of visitBerlin we stand for sustainable, green meetings and events. For more information, visit https://convention.visitberlin.de/en/sustain.

'''InterContinental Hotel Berlin''' is Green Globe Certified. Green Globe Certification is the worldwide sustainability system based on internationally accepted criteria for sustainable operation and management of travel and tourism businesses. Operating under a worldwide license, Green Globe Certification is based in California, USA and is represented in over 83 countries. Green Globe Certification is a member of the Global Sustainable Tourism Council, supported by the United Nations World Tourism Organization. Additonally, as part of IHG’s Green Engage™ program the InterContinental Berlin has implemented best practices at all operational levels. The innovative Environmental Management System not only measures the day-to-day energy, water and waste consumption, but also provides recommended actions – ‘green solutions’ – to improve energy conservation and the property’s carbon footprint score.

=='''Berlin: In your spare time (likely pre/post)'''==
=====First things first=====
Germany overall is widely diverse with regard to mentality, language, architecture, history, and more. In this guide we’re focusing on Berlin. However, if you have the opportunity to see a bit more of the country, you should definitely take it. We may also add that if you plan for a larger Europe trip, while there might be overlaps and similarities between the individual European countries, there are also lots and lots of differences -- and we’re by far not only referring to language. If you want to learn more about any other places you’re planning to go reach out to Mozillians who’re based there or otherwise know the city/country well. First-hand information is usually the most valuable and we have so many amazing people from anywhere in the world at our organization that it’s almost impossible that you don’t find someone who can help!

=====Exploring the city -- beyond Charlottenburg and Mitte!=====
For the All Hands week we’re going to be in Charlottenburg, one of the neighbourhoods most visited by tourists. Along with Mitte (basically the city center), it’s an attractive destination for shopping and sightseeing. Nevertheless, we can only recommend that you leave the usual tourist trail in order to get a more diverse impression of Germany’s capital. There’s, for example, kind-of-hipster Kreuzberg, where our German office is located -- along with a bunch of amazing bars, restaurants, clubs and much more. Or take vibrant Friedrichshain, which is just across the Spree river from Kreuzberg, or family-friendly Prenzlauer Berg, all of which have their own interesting tourist attractions that are complemented by a relaxed local vibe. Overall, Berlin has 12 boroughs and a total of 96 officially recognized localities, so there’s clearly a lot to see!

=====Orientation=====
Berlin is a pretty large city (891 qm, or 344 sq mi), so even Berliners use navigation apps regularly -- especially when leaving their “Kiez”. We therefore recommend that you download the Berlin map to your phone, as well, in order to make orientation easier, especially when doing some sightseeing.

This is also due to peculiarities that may make orientation less obvious, such as house numbers which do not necessarily run in the same direction (up or down) everywhere: On a lot of streets, the numbers ascend on one side and descend on the other. So to avoid getting you lost, you should check the numbering scheme first: you can find the name of the street at nearly every street corner. The same sign will usually state the range of house numbers in that segment.

Thanks to the subway system in particular, you can actually visit multiple neighbourhoods on the same day even if they’re not close. You can go from east to west in not much more than half an hour (if you’re lucky and have a good connection) and, even though it’s recommended to dedicate at least a few hours to each of the neighbourhoods you visit in order to get a proper impression, you should make use of it. Only by visiting different parts of the city that don’t share all of their history and demographics, you’ll be able to truly “see” Berlin.

=====Sightseeing & history=====
As mentioned (multiple times) before Berlin is full of history. It’s almost impossible to walk around the city for half an hour without coming across some sort of historical site. This ranges from a huge range of museums and galleries, which you’ll best get informed about through a guidebook, as well as historical buildings, such as churches.

If you don’t mind the cold weather, make sure to set aside some time for sites outside. You could, for example, explore cold war history and go to Checkpoint Charlie, the former border crossing between East and West Berlin, which is undoubtedly one of the city’s top attractions. Another recommendation would be a visit to Hohenschönhausen, a former Stasi prison where East German dissidents were incarcerated. Tours of the prison are conducted daily and often led by a former prisoner, who will give you a real insight into this fascinating, albeit traumatic time in history.

If you’re looking for something more cheerful and/or a bunch of suggestions for historical sightseeing options by neighbourhood, you should check out Wikivoyage, a free web-based travel guide that has also been called the “Wikipedia of travel guides”. Take a look at the article on City West, if you want to explore the vicinity of our All Hands location. Literally every other Berlin tourist would likely favor the eastern districts of Friedrichshain, Prenzlauer Berg and (western) Kreuzberg, for its lower price and a lot of international bars, cafés and many other tourist attractions.

Links
https://en.wikivoyage.org/wiki/Berlin/City_West
https://en.wikivoyage.org/wiki/Berlin/East_Central

=====Food!=====
Being an international city, almost any cuisine you can imagine is available in Berlin. Still, we recommend that you try out some actual German food during your stay. For example, go to a local bakery and you’ll be stunned by the variety: There are over 300 types of dark and white breads, and more than 1,200 different kinds of “Brötchen” and “Kleingebäck” (bread rolls and mini-breads). We’re so much into bread that every region in Germany even has their own term for bread rolls (“Brötchen” vs. “Schrippe” vs. “Weck” vs. Semmel”...). Bread is central to the “German diet”: For breakfast, many people eat it with cheese, sausage products, honey or marmalade and “Abendbrot” (“evening bread” = dinner) usually consists of sandwiches.

If you’re rather looking for a warm meal, you should try a Currywurst (“curry sausage”). Over the last couple of years, the Berlin foodie scene has truly exploded and you can get so much more than the stereotypical type of curry sausage nowadays, from plain to fancy, for meat lovers or for vegans. You can get them in restaurants, at street food markets or even from salespeople in the pedestrian zones.

When visiting restaurants that offer German cuisine, you’ll notice that meat is a main ingredient for many meals -- though there are also plenty of fish and vegetarian options nowadays. With regard to side dishes, potatoes (in a lot of different varieties, such as cooked, roasted, mashed or as cakes) are very popular as well as noodles.

In Germany as in many other countries it’s quite common to meet friends and family for dinner and go to a bar or pub afterwards. Berlin has a lot to offer in that respect and most places have their very individual flair. If you’re not into the aforementioned beer options, you could go for a (non-alcoholic or boozed) cocktail or a good glass of wine. Since Germany has several renown wine-growing regions, there’s a lot to choose from!

=='''All Hands Expense Policy'''==
1. All "All Hands" Expenses must be submitted on 1 (and only 1) Expense report (e.g. Berlin All Hands Expense Report). Each expense must be tagged with "All Hands - Jan 2020"

2. It must contain only those expenses relative to the All Hands Event in Berlin.

3. If your submitted expense report for All Hands is submitted outside these guidelines, it will be rejected and you will be asked to re-submit with only All Hands Expenses

4. The deadline to submit the All Hands Expense Report is '''February 28, 2020'''.

5. Expenses related to team events (except those coordinated through Lisa Carlson), parking, room service, mini-bar charges, and food/drink costs above the vouched amounts, will not be approved.

6. When submitting your expense report through Egencia, make sure to "attached the PDF" (toggle on mobile, check box on desktop).

'''The intention of our all hands are to centrally organize a structure that includes:'''
*Meals (2 evening events + breakfast, lunch, drinks and snacks Tuesday - Friday)
*Transportation
*Accommodations

Expenses submitted can not exceed the approved amounts. Any social events (except dinners/team building activities coordinated through Lisa Carlson) that are not part of our central plan will generally be self-organized and funded by participants.

=====Travel expenses policy=====
Reimbursement will be made for necessary and reasonable expenses.

* Getting to/from home to your airport: We ask that you use the most economical option, while balancing your personal needs - whether that be mileage/parking, public transportation or ride shares. We will not reimburse for private towncar services or car rental.
* Per standard Mozilla travel policy, Travel Meal Policy: Employees are eligible for travel meals when traveling. For Meals: There is a $75 USD per day meal policy while traveling. This covers breakfast (est. $15), lunch (est. $20), and dinner (est. $40). Receipts: Receipts are required. When submitting a meal for reimbursement, please include how many people attended the meal. There is no need to include names.
* We will not reimburse for seat/class upgrades. Frequent flyer status does not apply for All Hands.
* We will reimburse for bag checking fees for one bag.

Any other expenses must be approved by your manager ahead of time. Any expenses for extending your stay for business reasons - such as additional hotel nights, meals, etc must be approved by your manager before booking and travel.

=====Cell phone reimbursement policy=====
Cell phone reimbursement must be approved by your manager prior to submitting the expense. Teams will decide for their staff what is appropriate to expense.

=====Internet reimbursement policy=====
Internet will be provided in all guestrooms and meeting space in all hotels. If you opt to upgrade/add service, those costs are not reimbursable, unless previously approved by your manager and are for business reasons.

=='''Activities'''==

=====Lockpicking Event=====

On Wednesday Evening, there will be a Lockpicking event run by :decoder. The event will start at 6:30 PM and there will be dinner available at the venue for all the participants as well. [https://berlinallhandsjanuary2020.sched.com/event/Ya9V/lockpicking-dinner Advance registration] is required.

We will be joined by four other lockpickers from Berlin, two of them being championship winners on several national competitions. The same two people are also engineers working in the lock industry, so there will be plenty of interesting things to hear and learn this evening.

For better planning, we ask that people fill out the [https://berlinallhandsjanuary2020.sched.com/event/Ya9V/lockpicking-dinner registration form], we are limited to 100 participants initially. If the course ends up being fully booked and you still want to participate, you can join us at 8:00 PM on that same evening (without any guarantees on wait time and without dinner., but we still need you to [https://berlinallhandsjanuary2020.sched.com/event/Ya9V/lockpicking-dinner register]). We are confident that we will be able to accommodate everyone who wants to join. However, registering makes the planning a lot easier.

'''''What is Lockpicking?'''''

Lockpicking is a sport where you open locks (e.g. padlocks) without destroying them using manipulation tools ("picks").

'''''Why would I want to do this?'''''

Lockpicking is a great activity to practice patience, imagination and dexterity. From an engineering perspective, it helps to strengthen your sense for what is going on in that "black box" and even has some parallels to debugging. However, lockpicking is first and foremost about having fun and the lock being a challenge. So no matter what you work on, I promise you this will be a fun experience and also a good opportunity to put your laptop/phone aside for a while.

'''''Do I need to know anything about this prior to participating?'''''

Absolutely not. We will teach you how locks work, how to handle the tools and everything else you need to know. The only thing you need to bring with you is some patience.

'''''I have some tools, should I bring them?'''''

We will have tools at the venue and would ask people to not bring their own tools. Traveling with lock picks is possible under certain conditions but the legal situation, especially when it comes to flying, is complicated and depends on the countries you travel through. Hence we ask people to not travel with their tools because it can get you delayed and cause other legal issues.

'''''Isn't this illegal?'''''

Lockpicking as a sport/hobby is legal in many countries (including most of the EU), as long as you only open your own locks. Of course, opening locks for which you don't have permission to do so can get you into trouble and is strongly discouraged by the community.

However, there are a few countries around the world, including a handful of states in the US and some provinces in Canada where the possession of the tools (lock picks) is illegal without a license. If you want to do this at home and don’t know the rules for your country, feel free to ask us at the venue and we will try to help find the answers.

=====Daily Morning Yoga=====

Join Mozillians for a morning yoga practice from '''6:30-7:30am Tuesday-Friday IN TBD space''' [link to location coming!].

Sharon Bautista of the Firefox User Research team will lead a mixed-level yoga practice. Some prior experience with basic yoga poses (downdog, warrior poses, sun salutation A, etc.) will be helpful. We’ll practice asanas (physical postures) for about 50 minutes and use the last 10 minutes for guided pranayama (breath work) and/or meditation. The practice will be scaffolded with options so you can make it more gentle or more vigorous, depending on your preference.

Dress in comfortable clothing that you can move in, and bring your own yoga mat or large towel to use as a mat. An additional smaller towel may be helpful. All are welcome. If you’re on the fence about joining or have questions, don’t hesitate to ping Sharon: sbautista on Slack.

=====Swim crew=====
* Pullman - Lizzard, jdavidson, bkal, jmok, ??
* Intercontinental - ??

Let's coordinate here and in #brrlin to figure out times and which pool is best.

I'm interested in a before or after dinner swim during the week, but also could potentially do early morning! - Liz

====Running group====

See [https://docs.google.com/document/d/1hyhlVBWuG7TKsjRDh78F1zN7Y5K-HBUobbms5yzZ1dU/edit#heading=h.28r2nxazs8yq running signup doc] for meeting times or check out the #running channel in slack.

Firefox/Python 3 Migration

2019-12-24T03:23:15Z

Kmoir: /* Port a Mach Command */

This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

== Why This is Important ==

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]

Python 2 will reach it's End Of Life on '''[https://www.python.org/doc/sunset-python-2/ January 1st 2020]''' (here's [https://pythonclock.org/ a countdown!]).

== What Will Happen If We Don't Upgrade? ==

The [https://www.python.org/doc/sunset-python-2/ PSF said it best]:

<blockquote style="background-color: #F0F0FF; padding: 1em">"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time."
</blockquote>

== How We Get There ==

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

# Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
# Make key support libraries run in both Python 2 and Python 3. (mach-core)
# Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
# Port Python 2-only components to Python 3 piecemeal.
# Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

== Getting Involved ==

=== Find the others ===

Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.

=== What you'll need ===

You will need a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build working Firefox build environment].

=== Where to start: ===

==== Remove more of the "excluded" paths from the py2 and py3 linters ====

See [https://pad.mozilla.org/p/py3 this Etherpad] for instructions. (Etherpad is blank, needs to be fixed)

Roughly:

* Pick a directory [https://pad.mozilla.org/p/py3 from the list], and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
* Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
* Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
* To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
* To install `futurize`:
pip install --user future
* To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
* Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

# Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
# Use [https://bugzilla.mozilla.org/show_bug.cgi?id=1559975 bug 1559975]
# Set one of callek, ahal, or catlee as the code reviewer

==== Enable Python 3 on more Python unittests ====

Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:

# Find a test that is skipped with Python 3, you can use [https://searchfox.org/mozilla-central/search?q=.*(skip|fail)-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=.ini%24 this query] to find these tests.
# Edit the manifest and remove the Python 3 skip-if annotation.
# Run the test with `./mach python-test --python 3 path/to/test`.
# Fix any errors and repeat the previous step.
# Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in [https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml python.yml].
# Push the task to try on all platforms it is configured for.

==== Port a Mach Command ====

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2.

The whitelist contained in the mach command is py2commands

py2commands="
addtest
analyze
android
android-emulator
artifact
awsy-test
bootstrap..."

Pick the command that you'd like to convert and then:
# Remove it from the whitelist
# Run: ./mach <command>
# See what breaks and fix it!
# Repeat with a variety of different flags and arguments
# Make sure any Python unittests running in CI have Python 3 enabled (see above)

== How to Port the Code ==

=== Before starting ===

If you haven't ported modules before (or need a refresher), read the [https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]. It has a good outline of the porting process.

=== Porting modules in the Firefox source tree ===

* We've already standardized on Python 2.7 across the tree.
* Make sure you can test the code you're working on with python3!
** If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
** If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
* [http://python-future.org/futurize.html#stage-1-safe-fixes `futurize` Stage 1 fixes] can be used to transform code into code that's compatible with both python2.7 and python3.
* The `six` module is available in-tree, feel free to use it as a dependency.
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).

=== Porting examples ===

Prefer <code>[https://six.readthedocs.io/ six]</code> when possible. Make use of <code>six.PY2</code>, <code>six.PY3</code>, <code>six.moves</code>, <code>six.string_types</code>, <code>six.integer_types</code>, etc. This make it easy to find the py2 code again so we can remove it.

==== <code>import</code> statements ====

Use the <code>[https://six.readthedocs.io/#module-six.moves six.moves]</code> module:

Before:
<pre>
import __builtin__

__builtin__.__import__ = myimporthook()
</pre>

After:
<pre>
from six.moves import builtins

builtins.__import__ = myimporthook()
</pre>

==== <code>if</code> statements ====

Use <code>six.PY2</code> and <code>six.PY3</code>:

<pre>
import six

if six.PY2:
# py2 stuff
else:
# py3 stuff
</pre>

If six isn't available or you are in the imports section of the module then this is fine too:

<pre>
if sys.version_info < (3,):
# py2 stuff
else:
# py3 stuff
</pre>

==== String data ====

Use <code>six.string_types</code>, <code>six.text_type</code>, <code>six.binary_type</code>, etc. Also see the <code>six</code> docs for "[https://six.readthedocs.io/#binary-and-text-data Binary and text data]":

Before:
<pre>
# using unicode
def write(self, buf):
if isinstance(buf, unicode):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# using types.StringTypes
if isinstance(value, types.StringTypes):
...
</pre>

After:
<pre>
import six

# fixed unicode
def write(self, buf):
if isinstance(buf, six.text_type):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# fixed types.StringTypes
if isinstance(value, six.string_types):
...
</pre>

==== <code>subprocess.check_output()</code> ====

Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>. The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream. However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

<pre>
output = subprocess.check_output([callargs])
lines = output.splitlines() # Broken in Python 3
</pre>

After:

<pre>
output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()
</pre>

See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details.

=== Porting Resources ===

[https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]

[http://python-future.org/compatible_idioms.html Cheat Sheet: Writing Python 2-3 compatible code]. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

== References ==

[https://bugzilla.mozilla.org/showdependencytree.cgi?id=1496527&hide_resolved=1 Bug 1496527] tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

[https://docs.google.com/document/d/1Uo6GYztM4_3mYJ1P8Tj1M2VCXktaOXjUid8AIojO8Oo/edit?usp=sharing Python 3 tree migration notes]

[https://docs.google.com/document/d/1AzM3EOThG7GeMSCS5CcaTkojQEneaNj3N07igLBkyvc/edit# mach Python 3 migration roadmap]

Firefox/Python 3 Migration

2019-12-24T03:12:35Z

Kmoir: /* Remove more of the "excluded" paths from the py2 and py3 linters */

This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

== Why This is Important ==

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]

Python 2 will reach it's End Of Life on '''[https://www.python.org/doc/sunset-python-2/ January 1st 2020]''' (here's [https://pythonclock.org/ a countdown!]).

== What Will Happen If We Don't Upgrade? ==

The [https://www.python.org/doc/sunset-python-2/ PSF said it best]:

<blockquote style="background-color: #F0F0FF; padding: 1em">"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time."
</blockquote>

== How We Get There ==

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

# Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
# Make key support libraries run in both Python 2 and Python 3. (mach-core)
# Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
# Port Python 2-only components to Python 3 piecemeal.
# Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

== Getting Involved ==

=== Find the others ===

Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.

=== What you'll need ===

You will need a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build working Firefox build environment].

=== Where to start: ===

==== Remove more of the "excluded" paths from the py2 and py3 linters ====

See [https://pad.mozilla.org/p/py3 this Etherpad] for instructions. (Etherpad is blank, needs to be fixed)

Roughly:

* Pick a directory [https://pad.mozilla.org/p/py3 from the list], and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
* Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
* Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
* To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
* To install `futurize`:
pip install --user future
* To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
* Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

# Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
# Use [https://bugzilla.mozilla.org/show_bug.cgi?id=1559975 bug 1559975]
# Set one of callek, ahal, or catlee as the code reviewer

==== Enable Python 3 on more Python unittests ====

Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:

# Find a test that is skipped with Python 3, you can use [https://searchfox.org/mozilla-central/search?q=.*(skip|fail)-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=.ini%24 this query] to find these tests.
# Edit the manifest and remove the Python 3 skip-if annotation.
# Run the test with `./mach python-test --python 3 path/to/test`.
# Fix any errors and repeat the previous step.
# Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in [https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml python.yml].
# Push the task to try on all platforms it is configured for.

==== Port a Mach Command ====

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2. Pick the command that you'd like to convert and then:

# Remove it from the whitelist
# Run: ./mach <command>
# See what breaks and fix it!
# Repeat with a variety of different flags and arguments
# Make sure any Python unittests running in CI have Python 3 enabled (see above)

== How to Port the Code ==

=== Before starting ===

If you haven't ported modules before (or need a refresher), read the [https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]. It has a good outline of the porting process.

=== Porting modules in the Firefox source tree ===

* We've already standardized on Python 2.7 across the tree.
* Make sure you can test the code you're working on with python3!
** If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
** If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
* [http://python-future.org/futurize.html#stage-1-safe-fixes `futurize` Stage 1 fixes] can be used to transform code into code that's compatible with both python2.7 and python3.
* The `six` module is available in-tree, feel free to use it as a dependency.
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).

=== Porting examples ===

Prefer <code>[https://six.readthedocs.io/ six]</code> when possible. Make use of <code>six.PY2</code>, <code>six.PY3</code>, <code>six.moves</code>, <code>six.string_types</code>, <code>six.integer_types</code>, etc. This make it easy to find the py2 code again so we can remove it.

==== <code>import</code> statements ====

Use the <code>[https://six.readthedocs.io/#module-six.moves six.moves]</code> module:

Before:
<pre>
import __builtin__

__builtin__.__import__ = myimporthook()
</pre>

After:
<pre>
from six.moves import builtins

builtins.__import__ = myimporthook()
</pre>

==== <code>if</code> statements ====

Use <code>six.PY2</code> and <code>six.PY3</code>:

<pre>
import six

if six.PY2:
# py2 stuff
else:
# py3 stuff
</pre>

If six isn't available or you are in the imports section of the module then this is fine too:

<pre>
if sys.version_info < (3,):
# py2 stuff
else:
# py3 stuff
</pre>

==== String data ====

Use <code>six.string_types</code>, <code>six.text_type</code>, <code>six.binary_type</code>, etc. Also see the <code>six</code> docs for "[https://six.readthedocs.io/#binary-and-text-data Binary and text data]":

Before:
<pre>
# using unicode
def write(self, buf):
if isinstance(buf, unicode):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# using types.StringTypes
if isinstance(value, types.StringTypes):
...
</pre>

After:
<pre>
import six

# fixed unicode
def write(self, buf):
if isinstance(buf, six.text_type):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# fixed types.StringTypes
if isinstance(value, six.string_types):
...
</pre>

==== <code>subprocess.check_output()</code> ====

Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>. The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream. However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

<pre>
output = subprocess.check_output([callargs])
lines = output.splitlines() # Broken in Python 3
</pre>

After:

<pre>
output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()
</pre>

See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details.

=== Porting Resources ===

[https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]

[http://python-future.org/compatible_idioms.html Cheat Sheet: Writing Python 2-3 compatible code]. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

== References ==

[https://bugzilla.mozilla.org/showdependencytree.cgi?id=1496527&hide_resolved=1 Bug 1496527] tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

[https://docs.google.com/document/d/1Uo6GYztM4_3mYJ1P8Tj1M2VCXktaOXjUid8AIojO8Oo/edit?usp=sharing Python 3 tree migration notes]

[https://docs.google.com/document/d/1AzM3EOThG7GeMSCS5CcaTkojQEneaNj3N07igLBkyvc/edit# mach Python 3 migration roadmap]

Firefox/Python 3 Migration

2019-12-24T02:47:51Z

Kmoir: /* Remove more of the "excluded" paths from the py2 and py3 linters */

This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

== Why This is Important ==

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]

Python 2 will reach it's End Of Life on '''[https://www.python.org/doc/sunset-python-2/ January 1st 2020]''' (here's [https://pythonclock.org/ a countdown!]).

== What Will Happen If We Don't Upgrade? ==

The [https://www.python.org/doc/sunset-python-2/ PSF said it best]:

<blockquote style="background-color: #F0F0FF; padding: 1em">"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time."
</blockquote>

== How We Get There ==

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

# Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
# Make key support libraries run in both Python 2 and Python 3. (mach-core)
# Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
# Port Python 2-only components to Python 3 piecemeal.
# Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

== Getting Involved ==

=== Find the others ===

Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.

=== What you'll need ===

You will need a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build working Firefox build environment].

=== Where to start: ===

==== Remove more of the "excluded" paths from the py2 and py3 linters ====

See [https://pad.mozilla.org/p/py3 this Etherpad] for instructions.

Roughly:

* Pick a directory [https://pad.mozilla.org/p/py3 from the list], and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
* Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
* Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
* To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
* To install `futurize`:
pip install --user future
* To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
* Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

# Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
# Use [https://bugzilla.mozilla.org/show_bug.cgi?id=1559975 bug 1559975]
# Set one of callek, ahal, or catlee as the code reviewer

==== Enable Python 3 on more Python unittests ====

Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:

# Find a test that is skipped with Python 3, you can use [https://searchfox.org/mozilla-central/search?q=.*(skip|fail)-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=.ini%24 this query] to find these tests.
# Edit the manifest and remove the Python 3 skip-if annotation.
# Run the test with `./mach python-test --python 3 path/to/test`.
# Fix any errors and repeat the previous step.
# Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in [https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml python.yml].
# Push the task to try on all platforms it is configured for.

==== Port a Mach Command ====

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2. Pick the command that you'd like to convert and then:

# Remove it from the whitelist
# Run: ./mach <command>
# See what breaks and fix it!
# Repeat with a variety of different flags and arguments
# Make sure any Python unittests running in CI have Python 3 enabled (see above)

== How to Port the Code ==

=== Before starting ===

If you haven't ported modules before (or need a refresher), read the [https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]. It has a good outline of the porting process.

=== Porting modules in the Firefox source tree ===

* We've already standardized on Python 2.7 across the tree.
* Make sure you can test the code you're working on with python3!
** If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
** If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
* [http://python-future.org/futurize.html#stage-1-safe-fixes `futurize` Stage 1 fixes] can be used to transform code into code that's compatible with both python2.7 and python3.
* The `six` module is available in-tree, feel free to use it as a dependency.
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).

=== Porting examples ===

Prefer <code>[https://six.readthedocs.io/ six]</code> when possible. Make use of <code>six.PY2</code>, <code>six.PY3</code>, <code>six.moves</code>, <code>six.string_types</code>, <code>six.integer_types</code>, etc. This make it easy to find the py2 code again so we can remove it.

==== <code>import</code> statements ====

Use the <code>[https://six.readthedocs.io/#module-six.moves six.moves]</code> module:

Before:
<pre>
import __builtin__

__builtin__.__import__ = myimporthook()
</pre>

After:
<pre>
from six.moves import builtins

builtins.__import__ = myimporthook()
</pre>

==== <code>if</code> statements ====

Use <code>six.PY2</code> and <code>six.PY3</code>:

<pre>
import six

if six.PY2:
# py2 stuff
else:
# py3 stuff
</pre>

If six isn't available or you are in the imports section of the module then this is fine too:

<pre>
if sys.version_info < (3,):
# py2 stuff
else:
# py3 stuff
</pre>

==== String data ====

Use <code>six.string_types</code>, <code>six.text_type</code>, <code>six.binary_type</code>, etc. Also see the <code>six</code> docs for "[https://six.readthedocs.io/#binary-and-text-data Binary and text data]":

Before:
<pre>
# using unicode
def write(self, buf):
if isinstance(buf, unicode):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# using types.StringTypes
if isinstance(value, types.StringTypes):
...
</pre>

After:
<pre>
import six

# fixed unicode
def write(self, buf):
if isinstance(buf, six.text_type):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# fixed types.StringTypes
if isinstance(value, six.string_types):
...
</pre>

==== <code>subprocess.check_output()</code> ====

Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>. The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream. However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

<pre>
output = subprocess.check_output([callargs])
lines = output.splitlines() # Broken in Python 3
</pre>

After:

<pre>
output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()
</pre>

See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details.

=== Porting Resources ===

[https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]

[http://python-future.org/compatible_idioms.html Cheat Sheet: Writing Python 2-3 compatible code]. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

== References ==

[https://bugzilla.mozilla.org/showdependencytree.cgi?id=1496527&hide_resolved=1 Bug 1496527] tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

[https://docs.google.com/document/d/1Uo6GYztM4_3mYJ1P8Tj1M2VCXktaOXjUid8AIojO8Oo/edit?usp=sharing Python 3 tree migration notes]

[https://docs.google.com/document/d/1AzM3EOThG7GeMSCS5CcaTkojQEneaNj3N07igLBkyvc/edit# mach Python 3 migration roadmap]

Firefox/Python 3 Migration

2019-12-23T13:32:42Z

Kmoir: /* Remove more of the "excluded" paths from the py2 and py3 linters */

This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

== Why This is Important ==

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[https://ahal.ca/blog/2019/python-3-at-mozilla/]

Python 2 will reach it's End Of Life on '''[https://www.python.org/doc/sunset-python-2/ January 1st 2020]''' (here's [https://pythonclock.org/ a countdown!]).

== What Will Happen If We Don't Upgrade? ==

The [https://www.python.org/doc/sunset-python-2/ PSF said it best]:

<blockquote style="background-color: #F0F0FF; padding: 1em">"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time."
</blockquote>

== How We Get There ==

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

# Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
# Make key support libraries run in both Python 2 and Python 3. (mach-core)
# Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
# Port Python 2-only components to Python 3 piecemeal.
# Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

== Getting Involved ==

=== Find the others ===

Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.

=== What you'll need ===

You will need a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build working Firefox build environment].

=== Where to start: ===

==== Remove more of the "excluded" paths from the py2 and py3 linters ====

See [https://pad.mozilla.org/p/py3 this Etherpad] for instructions.

Roughly:

* Pick a directory [https://pad.mozilla.org/p/py3 from the list], and put your name beside it so we know it's ported
* Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
* Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
* To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
* To install `futurize`:
pip install --user future
* To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
* Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

# Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
# Use [https://bugzilla.mozilla.org/show_bug.cgi?id=1559975 bug 1559975]
# Set one of callek, ahal, or catlee as the code reviewer

==== Enable Python 3 on more Python unittests ====

Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:

# Find a test that is skipped with Python 3, you can use [https://searchfox.org/mozilla-central/search?q=.*(skip|fail)-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=.ini%24 this query] to find these tests.
# Edit the manifest and remove the Python 3 skip-if annotation.
# Run the test with `./mach python-test --python 3 path/to/test`.
# Fix any errors and repeat the previous step.
# Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in [https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml python.yml].
# Push the task to try on all platforms it is configured for.

==== Port a Mach Command ====

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2. Pick the command that you'd like to convert and then:

# Remove it from the whitelist
# Run: ./mach <command>
# See what breaks and fix it!
# Repeat with a variety of different flags and arguments
# Make sure any Python unittests running in CI have Python 3 enabled (see above)

== How to Port the Code ==

=== Before starting ===

If you haven't ported modules before (or need a refresher), read the [https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]. It has a good outline of the porting process.

=== Porting modules in the Firefox source tree ===

* We've already standardized on Python 2.7 across the tree.
* Make sure you can test the code you're working on with python3!
** If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
** If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
* [http://python-future.org/futurize.html#stage-1-safe-fixes `futurize` Stage 1 fixes] can be used to transform code into code that's compatible with both python2.7 and python3.
* The `six` module is available in-tree, feel free to use it as a dependency.
* [https://black.readthedocs.io/en/stable/index.html Black] can format Python 2 code and Python 3 code (but [https://black.readthedocs.io/en/stable/installation_and_usage.html it requires Py3.6+ to run]).

=== Porting examples ===

Prefer <code>[https://six.readthedocs.io/ six]</code> when possible. Make use of <code>six.PY2</code>, <code>six.PY3</code>, <code>six.moves</code>, <code>six.string_types</code>, <code>six.integer_types</code>, etc. This make it easy to find the py2 code again so we can remove it.

==== <code>import</code> statements ====

Use the <code>[https://six.readthedocs.io/#module-six.moves six.moves]</code> module:

Before:
<pre>
import __builtin__

__builtin__.__import__ = myimporthook()
</pre>

After:
<pre>
from six.moves import builtins

builtins.__import__ = myimporthook()
</pre>

==== <code>if</code> statements ====

Use <code>six.PY2</code> and <code>six.PY3</code>:

<pre>
import six

if six.PY2:
# py2 stuff
else:
# py3 stuff
</pre>

If six isn't available or you are in the imports section of the module then this is fine too:

<pre>
if sys.version_info < (3,):
# py2 stuff
else:
# py3 stuff
</pre>

==== String data ====

Use <code>six.string_types</code>, <code>six.text_type</code>, <code>six.binary_type</code>, etc. Also see the <code>six</code> docs for "[https://six.readthedocs.io/#binary-and-text-data Binary and text data]":

Before:
<pre>
# using unicode
def write(self, buf):
if isinstance(buf, unicode):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# using types.StringTypes
if isinstance(value, types.StringTypes):
...
</pre>

After:
<pre>
import six

# fixed unicode
def write(self, buf):
if isinstance(buf, six.text_type):
buf = buf.encode('utf-8')
BytesIO.write(self, buf)

# fixed types.StringTypes
if isinstance(value, six.string_types):
...
</pre>

==== <code>subprocess.check_output()</code> ====

Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>. The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream. However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

<pre>
output = subprocess.check_output([callargs])
lines = output.splitlines() # Broken in Python 3
</pre>

After:

<pre>
output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()
</pre>

See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details.

=== Porting Resources ===

[https://docs.python.org/3/howto/pyporting.html Porting Guide on python.org]

[http://python-future.org/compatible_idioms.html Cheat Sheet: Writing Python 2-3 compatible code]. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

== References ==

[https://bugzilla.mozilla.org/showdependencytree.cgi?id=1496527&hide_resolved=1 Bug 1496527] tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

[https://docs.google.com/document/d/1Uo6GYztM4_3mYJ1P8Tj1M2VCXktaOXjUid8AIojO8Oo/edit?usp=sharing Python 3 tree migration notes]

[https://docs.google.com/document/d/1AzM3EOThG7GeMSCS5CcaTkojQEneaNj3N07igLBkyvc/edit# mach Python 3 migration roadmap]

Auto-tools/Projects/Pulse

2019-12-13T16:13:36Z

Kmoir: /* Admin Procedures */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

dustin and the taskcluster team have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

* https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

WeeklyUpdates/2019-11-25

2019-11-29T15:02:46Z

Kmoir: /* Welcome! */

{{WeeklyUpdateNav}}
* Every Monday @ 11:00am Pacific Time (19:00 UTC)
* Watch at https://mzl.la/project-meeting-2019-11-25 or anonymously at https://air.mozilla.org. You can also watch on the Mozilla YouTube channel at https://www.youtube.com/watch?v=HEfZ1s0z2Sw.
* join #weekly-project-call on irc.mozilla.org or Slack for backchannel discussion
* If you plan on presenting, please join the Zoom video chat 20 minutes prior to the start of the meeting and announce to the A/V Technicians that you will be speaking so that they can confirm your Audio and Video.
* '''Presenters only:''' Zoom Meeting ID: 828 817 988 [https://mozilla.zoom.us/j/828817988 https://mozilla.zoom.us/j/828817988.] Do '''not''' use this room if you're not planning to speak.
* One-tap mobile
** +16465588656,,828817988# US (New York)
** +17207072699,,828817988# US
* Dial-in by your location
** +1 646 558 8656 US (New York)
** +1 720 707 2699 US
** 877 853 5257 US Toll-free
** +61 2 8015 2088 Australia
** +61 8 7150 1149 Australia
** 1800 893 423 Australia Toll-free
** +1 647 558 0588 Canada
** +33 1 8288 0188 France
** +33 7 5678 4048 France
** 0 805 082 588 France Toll-free
** +49 30 5679 5800 Germany
** +49 69 8088 3899 Germany
** +49 30 3080 6188 Germany
** 800 724 3138 Germany Toll-free
** +852 5808 6088 Hong Kong, China
** +44 203 695 0088 United Kingdom
** +44 203 966 3809 United Kingdom
** +44 203 051 2874 United Kingdom
** 0 800 031 5717 United Kingdom Toll-free

__TOC__

= Weekly Project Status Meeting Agenda =

Items in this section will be shared during the live status meeting.

== Friends of Mozilla [[Image:Tree.gif|Friends of Mozilla]] ==

== Upcoming Events ==

=== This Week ===

=== Monday, {{#time:d F|{{SUBPAGENAME}}}} ===

=== Tuesday, {{#time:d F|{{SUBPAGENAME}} +1 day}} ===

=== Wednesday, {{#time:d F|{{SUBPAGENAME}} +2 days}} ===
==== Homebrew Website Club ====
The '''Homebrew Website Club''' is a growing world-wide network of meetups for everyone who wants to take back their web experience from social media silos, and own their online identities, content, and interactions.

* 17:30-19:30 GMT at
** '''Nottingham (UK)'''
* Details and '''RSVP: https://indieweb.org/events/2019-11-27-homebrew-website-club'''

=== Thursday, {{#time:d F|{{SUBPAGENAME}} +3 days}} ===

=== Friday, {{#time:d F|{{SUBPAGENAME}} +4 days}} ===

=== Saturday, {{#time:d F|{{SUBPAGENAME}} +5 days}} ===

=== Sunday, {{#time:d F|{{SUBPAGENAME}} +6 days}} ===

=== Next Week ===
=== Saturday Sunday 7-8 December ===
==== 🌁 IndieWebCamp San Francisco ====
'''IndieWebCamp San Francisco''' is an all-levels BarCamp-like collaboration hosted at Mozilla [[SF]] for two days ('''Saturday & Sunday''') of roundtable sessions, brainstorming, creating, teaching, and a hackathon to help us all take back our data and lives on the web!

Details and tickets: https://2019.indieweb.org/sf

Any questions? Ask organizer [[Tantek]] on Slack.

=== Later Events and Available Tickets ===

'''Mozilla DevRel Complimentary Tickets'''

Our DevRel Sponsorship team sometimes receives complimentary tickets to share with Mozillians interested in attending the following events. Please fill out this form [https://airtable.com/shrRFuIEm7j2a0Gtu Ticket RequestForm ].
If you have questions, reach out to the DevSponsorship Team at devsponsorship@mozilla.com for more details.

'''[https://www.openup.global/?lang=en/ OPenUP Summit]''' Taipei | 2019-11-30 to 12-01

'''[https://ntnu.edu/wac2019 Web Audio Conference 2019]''' Trondheim, Norway | 2019-12-04 to 2019-12-06

'''[https://halfstackconf.com/ Halfstack]''' Phoenix, AZ | 2020-01-17

''' Mozilla Sponsored Developer Events'''

Current list of Mozilla sponsored developer events (with their start dates) around the globe (DevRel Events team will update):

Open UP Summit | 11/30-12/1 2019 | Taipei

Web Audio Conference 2019 | 12/4/19 | Trondheim, Norway

[https://2019.indieweb.org/sf 🌁 IndieWebCamp San Francisco] | 12/7/19 - 12/8/19 | [[MozSF]], San Francisco

Halfstack | 01/17/20 | Phoenix

PyCascades 2020 | 89/2/20 | Portland

You Got This 2020 | 01/18/20 | Birmingham, UK

[https://2020.indieweb.org/austin 🌮 IndieWebCamp Austin 2020] | 02/22/20 - 02/23/20 | Austin, Texas, USA

PerfMatters Conference 2020 | 03/31/20 - 04/02/20 | Redwood City, CA USA

JS Kongress Munich | 04/15-16/20 | Munich, Germany

[https://2020.indieweb.org/summit ⛰ IndieWeb Summit 2020] | 06/27/20 - 06/28/20 | Portland, Oregon, USA

== Speakers ==

The limit is '''3 minutes per topic'''. It's like a lightning talk, but don't feel that you have to have slides in order to make a presentation. If you plan on showing a video, you need to contact the Air Mozilla team before the day of the meeting or you will be deferred to the next week. The meeting is streamed in a 4:3 format in order to allow for split screen. If your slides are 16:9 "widescreen" format, please indicate in the "Sharing" column below.

{| class="fullwidth-table wikitable"
|-
! [https://mozillians.org/u/USERNAME Presenter]
! Title
! Topic
! Location
! Sharing
! Media
! More Details
|-
| Who Are You?
| What Do You Do?
| What are you going to talk about?
| Where are you presenting from? (Moz Space, your house, space)
| Will you be sharing your screen? (yes/no, 4:3 or 16:9)
| Links to slides or images you want displayed on screen
| Link to where audience can find out more information
|-
| Andrew Krug
| Staff Security Engineer, Enterprise Information Security
| IT MinIT
| Remote, Via PreRecorded Video
| Please play [https://drive.google.com/open?id=1W4b875PcToUg4kghNxMvgTH8ED77NOt7 video]
| n/a
|
* [https://drive.google.com/open?id=1W4b875PcToUg4kghNxMvgTH8ED77NOt7 Video]
* [https://wiki.mozilla.org/IT/WeeklyMinIT#2019-11-25 MinIT Wiki Shownotes]
|-
| Asa Dotzler
| Team Firefox
| Weekly Firefox Update
| MTV
| no
| n/a
| [https://wiki.mozilla.org/Firefox/Roadmap/Updates#2019-11-25 2019-11-25]
|-
| Till Schneidereit
| Senior Manager, Research Engineering
| Emerging Technologies Weekly Update
| Berlin
| no
| n/a
| [https://wiki.mozilla.org/WeeklyUpdates/EmergingTechnology#November_25th.2C_2019 ET update]
|-
|}

= Welcome! =

Let's say hello to some new Mozillians! If you are not able to join the meeting live, you can add a link to a short video introducing yourself.

{| class="fullwidth-table wikitable"
|-
! ''Who is being introduced?''
! ''Who are you? (the introducer)''
! ''Where are you doing the introduction?''
! ''Where are they from?''
! ''How will they be part of Mozilla?''
|-
| Nishant Bhasin
| Garvan Keeley
| Toronto
| Toronto
| Senior Software Engineer, Firefox for iOS
|-
| Zeid Zabaneh
| Kim Moir
| Toronto
| Toronto
| Senior Software Engineer, Developer productivity
|-
| Marc Leclair
| Vicky Chin
| Toronto
| Toronto
| Software Engineer, Android Performance
|-
| Erik Nordin
| Sean Voisen
| Home (SF Bay Area)
| PDX
| Software Engineer, Platform Layout

|-
| Their Name
| Your Name
| Intro location
| Their Location
| Their Role
|-
|}

[[Category:Weekly Updates]]
[[Category:Meeting Notes]]

WeeklyUpdates/2019-11-25

2019-11-22T22:19:58Z

Kmoir: /* Welcome! */

WeeklyUpdates/2019-09-23

2019-09-23T17:50:01Z

Kmoir: /* Welcome! */

{{WeeklyUpdateNav}}
* Every Monday @ 11:00am Pacific Time (19:00 UTC)
* Watch at https://mzl.la/project-meeting-2019-09-23 or anonymously at https://air.mozilla.org. You can also watch on the Mozilla YouTube channel at https://www.youtube.com/watch?v=Rbp5LBBQLkU.
* join #weekly-project-call on irc.mozilla.org or Slack for backchannel discussion
* If you plan on presenting, please join the Zoom video chat 20 minutes prior to the start of the meeting and announce to the A/V Technicians that you will be speaking so that they can confirm your Audio and Video.
* '''Presenters only:''' Zoom Meeting ID: 828 817 988 [https://mozilla.zoom.us/j/828817988 https://mozilla.zoom.us/j/828817988.] Do '''not''' use this room if you're not planning to speak.
* One-tap mobile
** +16465588656,,828817988# US (New York)
** +17207072699,,828817988# US
* Dial-in by your location
** +1 646 558 8656 US (New York)
** +1 720 707 2699 US
** 877 853 5257 US Toll-free
** +61 2 8015 2088 Australia
** +61 8 7150 1149 Australia
** 1800 893 423 Australia Toll-free
** +1 647 558 0588 Canada
** +33 1 8288 0188 France
** +33 7 5678 4048 France
** 0 805 082 588 France Toll-free
** +49 30 5679 5800 Germany
** +49 69 8088 3899 Germany
** +49 30 3080 6188 Germany
** 800 724 3138 Germany Toll-free
** +852 5808 6088 Hong Kong, China
** +44 203 695 0088 United Kingdom
** +44 203 966 3809 United Kingdom
** +44 203 051 2874 United Kingdom
** 0 800 031 5717 United Kingdom Toll-free

__TOC__

= Weekly Project Status Meeting Agenda =

Items in this section will be shared during the live status meeting.

== Friends of Mozilla [[Image:Tree.gif|Friends of Mozilla]] ==

== Upcoming Events ==

=== This Week ===

=== Monday, {{#time:d F|{{SUBPAGENAME}}}} ===

=== Tuesday, {{#time:d F|{{SUBPAGENAME}} +1 day}} ===

=== Wednesday, {{#time:d F|{{SUBPAGENAME}} +2 days}} ===
* Add-ons Community Meeting
** 15:30 UTC / 8:30am PST | See [[Add-ons/Community/Meetings|wiki]] for call-in info

=== Thursday, {{#time:d F|{{SUBPAGENAME}} +3 days}} ===

=== Friday, {{#time:d F|{{SUBPAGENAME}} +4 days}} ===

=== Saturday, {{#time:d F|{{SUBPAGENAME}} +5 days}} ===

=== Sunday, {{#time:d F|{{SUBPAGENAME}} +6 days}} ===

=== Next Week ===

=== Later Events and Available Tickets ===

'''Mozilla DevRel Complimentary Tickets'''

Our DevRel Sponsorship team sometimes receives complimentary tickets to share with Mozillians interested in attending the following events. Please fill out this form [https://airtable.com/shrRFuIEm7j2a0Gtu Ticket RequestForm ].
If you have questions, reach out to the DevSponsorship Team at devsponsorship@mozilla.com for more details.

'''[https://jsconfbp.com/ JS Conf & CSS Conf Budapest]''' Budapest, Hungary | 2019-09-25 & 2019-09-26-27

'''[https://ruhrjs.de/ RuhrJS 2019]''' Rotunde Bochum - Bochum Germany | 2019-10-05 to 2019-10-06

'''[http://connect.tech/ CONNECT.TECH]''' Atlanta, GA, USA | 2019-10-16 to 2019-10-18

'''[https://2018.ffconf.org/ ffconf]''' ffconf (formerly known as Full Frontal) Brighton, England | 2019-10-08 to 2019-10-08

'''[https://www.rust-belt-rust.com: Rust Belt Rust]'''| Dayton, OH, USA | 2019-10-18-19

'''[https://scriptconf.org/ Script'19: Javascript for Everyone]'''| Linz, Austria | 2019-10-25

'''[https://perfnow.nl/ performance.now ()]'''| Amsterdam, The Netherlands | 2019-11-21 to 2019-11-22

'''[https://halfstackconf.com/ Halfstack]''' London | 2019-11-22

'''[https://2019.webconf.asia/ Webconf Asia]''' Hong Kong | 2019-11-20-23

'''[https://ntnu.edu/wac2019 Web Audio Conference 2019]''' Trondheim, Norway | 2019-12-04 to 2019-12-06

'''[https://halfstackconf.com/ Halfstack]''' Phoenix, AZ | 2020-01-17

''' Mozilla Sponsored Developer Events'''

Current list of Mozilla sponsored developer events (with their start dates) around the globe (DevRel Events team will update):

Finch Front End | 9/23-25/19 | Edinburgh, UK

CSSConf Budapest 2019 | 9/25/19 | Budapest, Hungary

JSConf Budapest 2019 | 9/26-27/19 | Budapest, Hungary

PromptConf | 09/28/19 | Chicago

ARTIFACT Conference | 9/30-10/2 19 | Austin, Texas US

[https://indieweb.org/2019/Amsterdam ❌ IndieWebCamp Amsterdam] | 9/28/19 - 9/29/19 | Amsterdam, NL | 9/28/19 - 9/29/19 | Amsterdam, NL

View Source 2019 | 9/30/19 - 10/2/19 | Amsterdam, NL

Fronteers | 10/3/19 | Amsterdam, NL

RuhrJS 2019 | 10/5/19 | Bochum, Germany

[https://indieweb.org/2019/NYC 🗽 IndieWebCamp New York City 2019] | 10/5/19 - 10/6/19 | New York City, NY, USA

Paris Web 2019 | 10/10-12/19 | Bois-Colombes and Paris, France

Web Engines Hackfest 2019 | 10/14/19 | Galicia, Spain

CONNECT.TECH | 10/16/19 | Atlanta, GA, USA

Rust Belt Rust | 10/18-19/19 | Dayton, OH, USA

[https://indieweb.org/2019/Brighton 🎪 IndieWebCamp Brighton 2019] | 10/19/19 - 10/20/19 | Brighton, England

2019 LLVM Developers' Meeting | 10/22/19 | San Jose

Rust.Tokyo | 10/26/19 | Tokyo

a11yTO Conf | 10/24/19 | Toronto

Script'19: Javascript for Everyone 10/25/19 | Linz, Austria

Security BSides Portland | 10/25-26/19 | Portland, USA

beyond tellerrand // | BERLIN 2019 | 11/4/19 | Berlin, Germany

CascadiaJS | 11/07/19 | Seattle

ffconf | 11/8/19 | Brighton, UK

RustFest Barcelona | 11/09/19 | Barcelona

Performance now |11/21-22/19 | Amsterdam, The Netherlands

Halfstack | 11/22/19| London

WebConf Asia | 11/20-23| Hong Kong

webclerks community conference | 11/25/19 | Vienna

Web Audio Conference 2019 | 12/4/19 | Trondheim, Norway

dotJS 2019 | 12/5-6/19

Halfstack | 01/17/20 | Phoenix

You Got This 2020 | 01/18/20 | Birmingham, UK

JS Kongress Munich | 04/15-16/20 | Munich, Germany

[https://indieweb.org/2020 ⛰ IndieWeb Summit 2020] | 06/27/20 - 06/28/20 | Portland, Oregon, USA

== Speakers ==

The limit is '''3 minutes per topic'''. It's like a lightning talk, but don't feel that you have to have slides in order to make a presentation. If you plan on showing a video, you need to contact the Air Mozilla team before the day of the meeting or you will be deferred to the next week. The meeting is streamed in a 4:3 format in order to allow for split screen. If your slides are 16:9 "widescreen" format, please indicate in the "Sharing" column below.

{| class="fullwidth-table wikitable"
|-
! [https://mozillians.org/u/USERNAME Presenter]
! Title
! Topic
! Location
! Sharing
! Media
! More Details
|-
| Who Are You?
| What Do You Do?
| What are you going to talk about?
| Where are you presenting from? (Moz Space, your house, space)
| Will you be sharing your screen? (yes/no, 4:3 or 16:9)
| Links to slides or images you want displayed on screen
| Link to where audience can find out more information
|-
| Josh Matthews
| Engagement Whip, Mixed Reality
| Emerging Technologies weekly update
| Toronto
| no
| n/a
| [https://wiki.mozilla.org/WeeklyUpdates/EmergingTechnology#September_23rd.2C_2019 ET headlines]
|-
| Asa Dotzler
| Team Firefox
| Weekly Firefox Update
| MTV
| no
| n/a
| [https://wiki.mozilla.org/Firefox/Roadmap/Updates#2019-09-23 2019-09-23]
|-
|}

= Welcome! =

Let's say hello to some new Mozillians! If you are not able to join the meeting live, you can add a link to a short video introducing yourself.

{| class="fullwidth-table wikitable"
|-
! ''Who is being introduced?''
! ''Who are you? (the introducer)''
! ''Where are you doing the introduction?''
! ''Where are they from?''
! ''How will they be part of Mozilla?''
|-

|-
| Their Name
| Your Name
| Intro location
| Their Location
| Their Role
|-
| Ricky Stewart
| Chris Manchester
| SF Office
| Chicago Remote
| Senior Software Engineer, Developer Productivity
|-
| Alex Williams
| Jofish Kaye
| Remote Mountain View
| Remote South Carolina
| HCI Research Intern, Emerging Technologies
|-
|}

[[Category:Weekly Updates]]
[[Category:Meeting Notes]]

WeeklyUpdates/2019-09-23

2019-09-23T13:08:03Z

Kmoir: /* Welcome! */

{{WeeklyUpdateNav}}
* Every Monday @ 11:00am Pacific Time (19:00 UTC)
* Watch anonymously at https://air.mozilla.org.
* join #weekly-project-call on irc.mozilla.org or Slack for backchannel discussion
* If you plan on presenting, please join the Zoom video chat 20 minutes prior to the start of the meeting and announce to the A/V Technicians that you will be speaking so that they can confirm your Audio and Video.
* '''Presenters only:''' Zoom Meeting ID: 828 817 988 [https://mozilla.zoom.us/j/828817988 https://mozilla.zoom.us/j/828817988.] Do '''not''' use this room if you're not planning to speak.
* One-tap mobile
** +16465588656,,828817988# US (New York)
** +17207072699,,828817988# US
* Dial-in by your location
** +1 646 558 8656 US (New York)
** +1 720 707 2699 US
** 877 853 5257 US Toll-free
** +61 2 8015 2088 Australia
** +61 8 7150 1149 Australia
** 1800 893 423 Australia Toll-free
** +1 647 558 0588 Canada
** +33 1 8288 0188 France
** +33 7 5678 4048 France
** 0 805 082 588 France Toll-free
** +49 30 5679 5800 Germany
** +49 69 8088 3899 Germany
** +49 30 3080 6188 Germany
** 800 724 3138 Germany Toll-free
** +852 5808 6088 Hong Kong, China
** +44 203 695 0088 United Kingdom
** +44 203 966 3809 United Kingdom
** +44 203 051 2874 United Kingdom
** 0 800 031 5717 United Kingdom Toll-free

__TOC__

= Weekly Project Status Meeting Agenda =

Items in this section will be shared during the live status meeting.

== Friends of Mozilla [[Image:Tree.gif|Friends of Mozilla]] ==

== Upcoming Events ==

=== This Week ===

=== Monday, {{#time:d F|{{SUBPAGENAME}}}} ===

=== Tuesday, {{#time:d F|{{SUBPAGENAME}} +1 day}} ===

=== Wednesday, {{#time:d F|{{SUBPAGENAME}} +2 days}} ===

=== Thursday, {{#time:d F|{{SUBPAGENAME}} +3 days}} ===

=== Friday, {{#time:d F|{{SUBPAGENAME}} +4 days}} ===

=== Saturday, {{#time:d F|{{SUBPAGENAME}} +5 days}} ===

=== Sunday, {{#time:d F|{{SUBPAGENAME}} +6 days}} ===

=== Next Week ===

=== Later Events and Available Tickets ===

'''Mozilla DevRel Complimentary Tickets'''

Our DevRel Sponsorship team sometimes receives complimentary tickets to share with Mozillians interested in attending the following events. Please fill out this form [https://airtable.com/shrRFuIEm7j2a0Gtu Ticket RequestForm ].
If you have questions, reach out to the DevSponsorship Team at devsponsorship@mozilla.com for more details.

'''[https://jsconfbp.com/ JS Conf & CSS Conf Budapest]''' Budapest, Hungary | 2019-09-25 & 2019-09-26-27

'''[https://ruhrjs.de/ RuhrJS 2019]''' Rotunde Bochum - Bochum Germany | 2019-10-05 to 2019-10-06

'''[http://connect.tech/ CONNECT.TECH]''' Atlanta, GA, USA | 2019-10-16 to 2019-10-18

'''[https://2018.ffconf.org/ ffconf]''' ffconf (formerly known as Full Frontal) Brighton, England | 2019-10-08 to 2019-10-08

'''[https://www.rust-belt-rust.com: Rust Belt Rust]'''| Dayton, OH, USA | 2019-10-18-19

'''[https://scriptconf.org/ Script'19: Javascript for Everyone]'''| Linz, Austria | 2019-10-25

'''[https://perfnow.nl/ performance.now ()]'''| Amsterdam, The Netherlands | 2019-11-21 to 2019-11-22

'''[https://halfstackconf.com/ Halfstack]''' London | 2019-11-22

'''[https://2019.webconf.asia/ Webconf Asia]''' Hong Kong | 2019-11-20-23

'''[https://ntnu.edu/wac2019 Web Audio Conference 2019]''' Trondheim, Norway | 2019-12-04 to 2019-12-06

'''[https://halfstackconf.com/ Halfstack]''' Phoenix, AZ | 2020-01-17

''' Mozilla Sponsored Developer Events'''

Current list of Mozilla sponsored developer events (with their start dates) around the globe (DevRel Events team will update):

Finch Front End | 9/23-25/19 | Edinburgh, UK

CSSConf Budapest 2019 | 9/25/19 | Budapest, Hungary

JSConf Budapest 2019 | 9/26-27/19 | Budapest, Hungary

PromptConf | 09/28/19 | Chicago

ARTIFACT Conference | 9/30-10/2 19 | Austin, Texas US

[https://indieweb.org/2019/Amsterdam ❌ IndieWebCamp Amsterdam] | 9/28/19 - 9/29/19 | Amsterdam, NL | 9/28/19 - 9/29/19 | Amsterdam, NL

View Source 2019 | 9/30/19 - 10/2/19 | Amsterdam, NL

Fronteers | 10/3/19 | Amsterdam, NL

RuhrJS 2019 | 10/5/19 | Bochum, Germany

[https://indieweb.org/2019/NYC 🗽 IndieWebCamp New York City 2019] | 10/5/19 - 10/6/19 | New York City, NY, USA

Paris Web 2019 | 10/10-12/19 | Bois-Colombes and Paris, France

Web Engines Hackfest 2019 | 10/14/19 | Galicia, Spain

CONNECT.TECH | 10/16/19 | Atlanta, GA, USA

Rust Belt Rust | 10/18-19/19 | Dayton, OH, USA

[https://indieweb.org/2019/Brighton 🎪 IndieWebCamp Brighton 2019] | 10/19/19 - 10/20/19 | Brighton, England

2019 LLVM Developers' Meeting | 10/22/19 | San Jose

Rust.Tokyo | 10/26/19 | Tokyo

a11yTO Conf | 10/24/19 | Toronto

Script'19: Javascript for Everyone 10/25/19 | Linz, Austria

Security BSides Portland | 10/25-26/19 | Portland, USA

beyond tellerrand // | BERLIN 2019 | 11/4/19 | Berlin, Germany

CascadiaJS | 11/07/19 | Seattle

ffconf | 11/8/19 | Brighton, UK

RustFest Barcelona | 11/09/19 | Barcelona

Performance now |11/21-22/19 | Amsterdam, The Netherlands

Halfstack | 11/22/19| London

WebConf Asia | 11/20-23| Hong Kong

webclerks community conference | 11/25/19 | Vienna

Web Audio Conference 2019 | 12/4/19 | Trondheim, Norway

dotJS 2019 | 12/5-6/19

Halfstack | 01/17/20 | Phoenix

You Got This 2020 | 01/18/20 | Birmingham, UK

JS Kongress Munich | 04/15-16/20 | Munich, Germany

[https://indieweb.org/2020 ⛰ IndieWeb Summit 2020] | 06/27/20 - 06/28/20 | Portland, Oregon, USA

== Speakers ==

The limit is '''3 minutes per topic'''. It's like a lightning talk, but don't feel that you have to have slides in order to make a presentation. If you plan on showing a video, you need to contact the Air Mozilla team before the day of the meeting or you will be deferred to the next week. The meeting is streamed in a 4:3 format in order to allow for split screen. If your slides are 16:9 "widescreen" format, please indicate in the "Sharing" column below.

{| class="fullwidth-table wikitable"
|-
! [https://mozillians.org/u/USERNAME Presenter]
! Title
! Topic
! Location
! Sharing
! Media
! More Details
|-
| Who Are You?
| What Do You Do?
| What are you going to talk about?
| Where are you presenting from? (Moz Space, your house, space)
| Will you be sharing your screen? (yes/no, 4:3 or 16:9)
| Links to slides or images you want displayed on screen
| Link to where audience can find out more information
|-
|}

= Welcome! =

Let's say hello to some new Mozillians! If you are not able to join the meeting live, you can add a link to a short video introducing yourself.

{| class="fullwidth-table wikitable"
|-
! ''Who is being introduced?''
! ''Who are you? (the introducer)''
! ''Where are you doing the introduction?''
! ''Where are they from?''
! ''How will they be part of Mozilla?''
|-

|-
| Their Name
| Your Name
| Intro location
| Their Location
| Their Role
|-
| Ricky Stewart
| Chris Manchester
| SF Office
| Chicago Remote
| Senior Software Engineer, Build Systems
|-
|}

[[Category:Weekly Updates]]
[[Category:Meeting Notes]]

Auto-tools/Projects/Pulse

2019-03-18T15:40:37Z

Kmoir: /* To upgrade a ssl certificate on pulse.mozilla.org */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

kmoir and glob have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

* https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T15:40:18Z

Kmoir: /* To upgrade a ssl certificate on pulse.mozilla.org */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

kmoir and glob have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

* https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T15:39:09Z

Kmoir: /* To upgrade a ssl certificate on pulse.mozilla.org */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

kmoir and glob have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

<pre>
https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
</pre>

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T15:38:53Z

Kmoir: /* To upgrade a ssl certificate on pulse.mozilla.org */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

kmoir and glob have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

<text>
https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
</text>

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T15:38:25Z

Kmoir: /* Admin Procedures */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

kmoir and glob have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

== To upgrade a ssl certificate on pulse.mozilla.org ==

Open a bug with IT to generate a new certificate
{{bug|1532325}}

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T15:05:52Z

Kmoir: /* Admin Procedures */

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

jgriffin and mcote have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Auto-tools/Projects/Pulse

2019-03-18T14:57:30Z

Kmoir: deprecating buildbot content

= Introducing Pulse =

Pulse is a managed [http://www.rabbitmq.com RabbitMQ] cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by [http://cloudamqp.com CloudAMQP].

[[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]] is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by [http://heroku.com Heroku].

We have a discussion forum available via the standard trio of [news:mozilla.tools.pulse USENET newsgroup], [https://lists.mozilla.org/listinfo/tools-pulse mailing list], and [https://groups.google.com/forum/#!forum/mozilla.tools.pulse Google Group].

File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse Webtools :: Pulse]. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]].

= System Description =

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

= Specification =

Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1]
service with [https://www.rabbitmq.com/extensions.html RabbitMQ extensions] for publishing messages from Mozilla
infrastructure. The aim is provide hooks that subscribers can
use to integrate and extend Mozilla infrastructure.

== Authentication ==

Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]],
available at https://pulseguardian.mozilla.org. This service SHALL issue
an ''accessToken'' for any ''clientId'' that is registered
with ''authorized'' email address.
The accessToken is strictly secret and MUST NOT be shared
publicly. The clientId is not secret. When establishing an AMQP
connection, the clientId and accessToken MUST be used as
''username'' and ''password'', respectively.

== Authorized Users ==

Pulse is intended to be open to all Mozillians who want to
extend or integrate with Mozilla infrastructure. To guard against
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

== Publishers ==

Publishers MUST name ''exchanges'' in the form <code>exchange/<clientId>/<name></code> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be ''topic exchanges'' and they MUST be declared ''durable''.

Messages MUST contain a UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON] payload, and
their <code>Content-Type</code> MUST be <code>application/json</code>.
Messages SHOULD NOT be larger than 8 kB; deviations may be
feasible for low-traffic exchanges. Messages MUST NOT contain
secret or sensitive information; all exchanges and messages
SHALL be considered public.

A message SHOULD carry a ''routing key'', in which fields have a
fixed index from the left. Additionally, a message MAY be
''cced'' to multiple routing keys, using the RabbitMQ
[https://www.rabbitmq.com/sender-selected.html ''Sender-selected Distribution''] extension.

Messages SHOULD be ''durable'' and SHOULD be published over
RabbitMQ [https://www.rabbitmq.com/confirms.html ''confirm-publish'' channels]. Otherwise, the
documentation MUST clearly reflect that messages from the
given exchange do not exhibit deliver ''at-least-once'' semantics.

== Subscribers ==

Subscribers MUST name ''queues'' in the form
<code>queue/<clientId>/<name></code>; attempts to name a queue otherwise
SHALL result in an authorization error. Queues MAY ''consume''
from any exchange prefixed <code>exchange/</code>; attempts to consume
from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ
[https://www.rabbitmq.com/maxlength.html ''Queue Length Limit''] extension. Subscribers MUST NOT let
their queues grow unbounded; if left unattended, ''Pulse''
SHALL notify the owner by email. Additionally, ''Pulse'' MAY delete
a queue which exceeds defined limits. Subscribers SHOULD specify a
prefetch limit using the RabbitMQ [https://www.rabbitmq.com/consumer-prefetch.html ''Consumer Prefetch'' limit] extension.

Subscribers SHOULD use either ''durable'' queues or
''auto-delete'' queues. Implementors are recommended to aim
for deliver-''at-least-once'' semantics.

== Appendix A: Everything in Bullet Points ==

This is a summary of the above.

Pulse:
* MUST offer registration at <code>pulse.mozilla.org</code>
* MUST support [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] and these RabbitMQ extensions:
** [https://www.rabbitmq.com/confirms.html Confirms]
** [https://www.rabbitmq.com/consumer-prefetch.html Consumer Prefetch]
** [https://www.rabbitmq.com/maxlength.html Queue Length Limit]
** [https://www.rabbitmq.com/sender-selected.html Sender-selected Distribution]
* SHOULD exhibit deliver-''at-least-once'' semantics
* MAY delete queues that grows beyond ''Pulse'' defined limits
* SHALL notify owner by email when a queue grows close to ''Pulse''-defined limits.

Publishers:
* SHOULD use [https://www.rabbitmq.com/confirms.html confirm-publish channels]

Exchanges:
* MUST be named <code>exchange/<clientId>/<name></code>
* MUST be topic exchanges
* MUST be durable

Messages:
* MUST be UTF-8-encoded [http://tools.ietf.org/html/rfc7159 JSON]
* MUST carry <code>application/json</code> as <code>Content-Type</code>
* SHOULD be durable
* SHOULD be less than 8 KiB (for good performance)
* MAY be CC'ed to [https://www.rabbitmq.com/sender-selected.html multiple routing keys]
* MUST NOT contain private or sensitive information
* SHOULD have a routing key where fields have a fixed index from the left

Subscribers:
* SHOULD specify a [https://www.rabbitmq.com/consumer-prefetch.html ''consumer prefetch'' limit]

Queues:
* MUST be named <code>queue/<clientId>/<name></code>
* MAY have a [https://www.rabbitmq.com/maxlength.html limited length]
* MUST not grow unbounded

= Let's Use It =

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

== Python Pulse client library ==

The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

== Go (golang) Pulse client library ==

This can be found here:
* http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):
* http://taskcluster.github.io/taskcluster-client-go/

= Contributing =

To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

<bugzilla>
{
"quicksearch": "status:new",
"product": "Webtools",
"component": "Pulse",
"f1": "bug_mentor",
"o1": "isnotempty",
"include_fields": "id,summary,priority,status"
}
</bugzilla>

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed",
"product": "Webtools",
"component": "Pulse"
}
</bugzilla>

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

= Road Map =

See the [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Pulse&product=Webtools prioritized bug list] for all open issues and feature requests.

= Security Model =

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

* All users, publishers and consumers alike, must have their own accounts (no guest/public users).
* Only publishers should be able to declare exchanges.
* Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
* Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on <code>"^exchange/<username>/.*$"</code> and <code>"^queue/<username>/.*$"</code>. They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. <code>queue/<username>/<random string></code>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

= Admin Procedures =

jgriffin and mcote have access to the Pulse cluster on CloudAMQP and the following related services:

* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.
* pulsetranslator service, which normalizes Buildbot messages, is currently running on Heroku and may need to be reset from time to time.

= More reading =

* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse.
* [https://mrcote.info/blog/2015/02/16/pulse-update/ Update] on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

* [http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/ Mozilla Pulse and RabbitMQ]
* [http://christian.legnitto.com/blog/2010/08/20/push-notifications-for-bugzilla/ Push notifications for Bugzilla]
** This idea has been shelved for the time being, both due to security issues and performance. See [[BMO/ChangeNotificationSystem]] for an alternative.
* [http://christian.legnitto.com/blog/2010/09/10/bugzilla-amqp-is-now-buzgilla-push-supports-the-stomp-protocol/ bugzilla-amqp is now buzgilla-push, supports the STOMP protocol]
** See above note. Just keeping these links for interest's sake.
* [http://christian.legnitto.com/blog/2010/10/15/whats-up-with-pulse/ What’s up with Pulse?]
* [http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/ GitHub AMQP integration service hook live!]
** Tangentially related to Pulse.
* [http://christian.legnitto.com/blog/2010/12/06/lots-of-pulse-changes-out-of-prototype-mode-this-week/ Lots of Pulse changes, out of prototype mode this week!]
* [http://christian.legnitto.com/blog/2010/12/22/another-update-on-pulse/ Another update on Pulse]

Phabricator/Bugzilla Equivalents

2019-02-06T14:35:35Z

Kmoir: /* Watch a component */

Some common tasks in Bugzilla do not have direct analogues in Phabricator and others may not be obvious to a new user. This page serves to document these common tasks and how to handle similar situations when using Phabricator.

= Making requests to other users =

== Request '''review?''' on a patch ==
Patches may be [https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches submitted to Phabricator] for review using [https://github.com/mozilla-conduit/review '''moz-phab''']. It is possible to ask for review as part of patch submission (See the [https://github.com/mozilla-conduit/review#execution '''moz-phab''' documentation]).

Note that Phabricator has two reviewer designations, normal and "blocking". Setting a user as a "blocking" reviewer is similar to the common use of a '''review?''' flag in that it requires review from that user or an explicit removal. Normal reviewers work differently in that acceptance from any reviewer removes the review from that user's queue (i.e. Only a single reviewer must accept). Using blocking reviewers mimics the Bugzilla workflow most closely, and there is a '''submit.always_blocking''' [https://github.com/mozilla-conduit/review#configuration configuration] option in '''moz-phab''' which will default to making all reviewers blocking.

== Request '''feedback?''' on a patch ==
Phabricator does not distinguish between asking for final review and requesting other forms of feedback. It is best to use an explicit comment explaining you would like feedback, along with one of:

* Requesting review from the user you'd like feedback from.
[[File:Non-blocking-review.png|320px]]

* Using an '''@''' mention. Including '''@<username>''' in a comment on Phabricator notifies the provided user of your message. The notification is a one-time action which does not require acknowledgement, so it is best to make it clear with your text that you are expecting feedback.
[[File:At-mention.png|479px]]

== Make a '''needinfo?''' request ==
Phabricator does not have a feature directly comparable to Bugzilla's '''needinfo?''' request. We suggest one of the following alternatives:

* Use an '''@''' mention. Including '''@<username>''' in a comment on Phabricator notifies the provided user of your message. The notification is a one-time action which does not require acknowledgement, so it is best to make it clear with your text that you expect a response.
[[File:At-mention.png|479px]]

* Add the user as a '''blocking reviewer''' and indicate your intent by including a comment. The Phabricator revision cannot be '''Accepted''' until the indicated user has either '''Accepted''' the revision, '''Resigned''' as a reviewer, or removed as a reviewer. Since this is re-purposing the normal request for review it is best to include a comment making your intended use clear.
[[File:Blocking-reviewer.png|324px]]

== Request '''review?''' of an image file ==
If you have a stand-alone image you'd like feedback on you may still do so using Bugzilla. Since the '''review?''' flag may be disabled for the component, please use a '''needinfo?''' or '''feedback?''' flag.

Phabricator does have the [https://phabricator.services.mozilla.com/pholio/ Pholio] application for commenting on mocks. Unfortunately it is not well integrated with the rest of our workflows and you cannot flag specific users for review. You can find Pholio in the main menu as "Design Review", and a new image can be uploaded by clicking [https://phabricator.services.mozilla.com/pholio/create/ "Create Mock"] on the main Pholio page.

== Clear a '''review?''' flag from your own patch ==
There are two ways to remove a patch from the queue of your reviewers:

* Explicitly edit the reviewers and remove the reviewer using the "Change Reviewers" action.
[[File:Change-Reviewers.png|206px]]

* "Plan Changes" to your revision, which removes the revision from their queue until you have posted a new version of the diff.
[[File:Plan-Changes.png|205px]]

Both of these actions are taken using the "Add Action..." dropdown at the bottom of the revision's page. After you have added the action (along with any comments) you must click the "Submit" button.

== Request '''approval-mozilla-<repo>?''' uplift on a patch ==
At this time '''approval-mozilla-<repo>?''' requests should use the current [[Release Management/Uplift rules]]. '''approval-mozilla-<repo>?''' can be set on the Bugzilla attachment which corresponds to the revision in Phabricator. There is a plan to move uplift approval to Phabricator fully, look out for an announcement when that is further along.

== Request '''sec-approval?''' on a patch ==
At this time '''sec-approval?''' requests should use the current [[Security/Bug Approval Process]]. '''sec-approval?''' can be set on the Bugzilla attachment which corresponds to the revision in Phabricator. There is a plan to move '''sec-approval''' to Phabricator fully, look out for an announcement when that is further along.

== Request '''checkin-needed''' to land your changes ==
At this time '''checkin-needed''' requests should use the current [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/How_to_Submit_a_Patch#Getting_the_patch_checked_into_the_tree checkin-needed process]. '''checkin-needed''' can be set on the bug linked to the Phabricator revision you wish to land. There is a plan to move '''checkin-needed''' to Phabricator fully, look out for an announcement when that is further along (Tracking [https://bugzilla.mozilla.org/show_bug.cgi?id=1514807 Bug 1514807]).

= Responding to requests =

== Clear a '''review?''' flag asked of you ==
If you'd like to remove yourself as a reviewer you can do so in one of two ways:

* Explicitly edit the reviewers and remove yourself using the "Change Reviewers" action.
[[File:Change-Reviewers-reviewer.png|216px]]

* Add the "Resign as Reviewer" action, which marks you as resigned. Resigning has mostly the same effect as removing yourself from the reviewer list except for the case when a project you're a member of has been asked for review. Resigning in the project case explicitly marks you as not a reviewer and is easier than removing the project and re-adding all other members.
[[File:Resign-As-Reviewer.png|204px]]

Both of these actions are taken using the "Add Action..." dropdown at the bottom of the revision's page. After you have added the action (along with any comments), you must click the "Submit" button.

= Querying for Information =

== Query a user's review queue ==
You may query a user's review queue by crafting an advanced differential search query. Take the following steps:
# Visit this example query page: https://phabricator.services.mozilla.com/differential/query/IeE2zgiqj9EX/
# Change "Responsible Users" to the user you're interested in.
#* You probably do ''not'' want "Exact User: <username>" as it excludes revisions for the user's projects.
# Click "Search".
# The user's review queue are items under the "Must Review" and "Ready to Review" headings.

== Query a user's activity ==
You may query a user's activity by visiting their profile page: https://phabricator.services.mozilla.com/p/<username>/. An example user's profile page is here: https://phabricator.services.mozilla.com/p/bzbarsky/. The "Recent Activity" box lists only recent activity, but you may click the "View All" button to be taken to an advanced query of the user's activity.

== Save a frequently used query ==
Most query advanced query pages have a "Save Query" button on the results page. Clicking this allows you to give the query a name and add it to your personal query list. After saving a query you can share it with others by copying the URL of the results page (Note, this URL does not include the saved query name and is quite opaque).

= Other Tasks =

== Upload a WIP patch ==
Work in progress patches can be uploaded to Phabricator in the same way as a normal patch, although generally you would not add reviewers. If you'd like someone to look at your WIP patch you may add reviewers but it is best to explicitly mention what you expect from the reviewer.

== Watch a component ==
Phabricator has a notification system, [https://phabricator.services.mozilla.com/herald/ Herald], which allows you to create custom notification rules. This can be used to subscribe yourself to revisions that touch certain parts of the code base, or even add yourself as a reviewer. You can view example rules other user's have created [https://phabricator.services.mozilla.com/herald/query/all/ here].

== How do I deal with non-mozilla branch patches after bugzilla review flags are disable on components on Feb 28? ==

In the announcement, it states

Initially, only code landing in mozilla-central will be required to be reviewed in Phabricator. Once we have implemented more processes in Phabricator, we will move reviews of all of the mozilla-central “branches” to Phabricator.

What does this mean for cases where a separate branch patch needs to be written after Feb 28 before it's possible to review the other branches in Phabricator?

You can review anything Phabricator, however Lando can't land to the other "branches" at this time. Uplift approval will also be moved at a later date.

Phabricator/Bugzilla Equivalents

2019-01-31T22:19:20Z

Kmoir: /* Upload WIP patches */

Some common tasks in Bugzilla do not have direct analogues in Phabricator and others may not be obvious to a new user. This page serves to document these common tasks and how similar situations should be handled when using Phabricator.

= Tasks =

== Querying for Information ==

=== Query a user's review queue ===
You may query a user's review queue by crafting an advanced differential search query. Take the following steps:
* Visit this example query page: https://phabricator.services.mozilla.com/differential/query/IeE2zgiqj9EX/
* Change "Responsible Users" to the user you're interested in.
** You probably do ''not'' want "Exact User: <username>" as it excludes revisions for the user's projects.
* Click "Search".
* The user's review queue are items under the "Must Review" and "Ready to Review" headings.

=== Query a user's activity ===
You may query a user's activity by visiting their profile page: https://phabricator.services.mozilla.com/p/<username>/. An example user's profile page is here: https://phabricator.services.mozilla.com/p/bzbarsky/. The "Recent Activity" box lists only recent activity, but you may click the "View All" button to be taken to an advanced query of the user's activity.

=== Save a frequently used query ===
Most query advanced query pages will have a "Save Query" button on the results page. Clicking this will allow you to give the query a name and add it to your personal query list. After saving a query you can share it with others by copying the URL of the results page (Note, this url does not include the saved query name and is quite opaque).

== Making requests to other users ==

=== Make a '''needinfo?''' request ===
Phabricator does not have a feature directly comparable to Bugzilla's '''needinfo?''' request. We suggest one of the following alternatives:
* Use an '''@''' mention. Including '''@<username>''' in a comment on Phabricator will notify the provided user of your message. This is a one-time notification which does not require acknowledgement, so it is best to make it clear with your text that you expect a response.
* Add the user as a '''blocking reviewer''' and indicate your intent by including a comment. This will prevent the Phabricator revision from becoming '''Accepted''' until the indicated user has either '''Accepted''' the revision, '''Resigned''' as a reviewer, or been removed as a reviewer. Since this is re-purposing the normal request for review it is best to include a comment making your intended use clear.

=== Request '''feedback?''' on a patch ===
Phabricator does not distinguish between asking for final review and requesting other forms of feedback. It is best to use an explicit comment explaining you would like feedback, along with one of:
* Requesting review from the user you'd like feedback from.
* Using an '''@''' mention. Including '''@<username>''' in a comment on Phabricator will notify the provided user of your message. This is a one-time notification which does not require acknowledgement, so it is best to make it clear with your text that you are expecting feedback.

=== Upload WIP patches ===

TODO: Not sure if this workflow should be integrated into the request feedback section.

=== Request '''review?''' on a patch ===
TODO: moz-phab, add reviewer, blocking vs non-blocking.

=== Clear a '''review?''' flag from your own patch ===
TODO: plan changes vs edit reviewers.

=== Clear a '''review?''' flag asked of you ===
TODO: request changes vs resign.

=== Request uplift approval on a patch ===
TODO

=== Request '''checkin-needed''' to land your changes ===
TODO: Move to phabricator tag.

=== Request '''review?''' of an image file ===
TODO: Pholio

=== Request '''sec-approval?''' on a patch ===
TODO

Phabricator/Bugzilla Equivalents

2019-01-31T22:18:59Z

Kmoir: /* Request feedback? on a patch */

Some common tasks in Bugzilla do not have direct analogues in Phabricator and others may not be obvious to a new user. This page serves to document these common tasks and how similar situations should be handled when using Phabricator.

= Tasks =

== Querying for Information ==

=== Query a user's review queue ===
You may query a user's review queue by crafting an advanced differential search query. Take the following steps:
* Visit this example query page: https://phabricator.services.mozilla.com/differential/query/IeE2zgiqj9EX/
* Change "Responsible Users" to the user you're interested in.
** You probably do ''not'' want "Exact User: <username>" as it excludes revisions for the user's projects.
* Click "Search".
* The user's review queue are items under the "Must Review" and "Ready to Review" headings.

=== Query a user's activity ===
You may query a user's activity by visiting their profile page: https://phabricator.services.mozilla.com/p/<username>/. An example user's profile page is here: https://phabricator.services.mozilla.com/p/bzbarsky/. The "Recent Activity" box lists only recent activity, but you may click the "View All" button to be taken to an advanced query of the user's activity.

=== Save a frequently used query ===
Most query advanced query pages will have a "Save Query" button on the results page. Clicking this will allow you to give the query a name and add it to your personal query list. After saving a query you can share it with others by copying the URL of the results page (Note, this url does not include the saved query name and is quite opaque).

== Making requests to other users ==

=== Make a '''needinfo?''' request ===
Phabricator does not have a feature directly comparable to Bugzilla's '''needinfo?''' request. We suggest one of the following alternatives:
* Use an '''@''' mention. Including '''@<username>''' in a comment on Phabricator will notify the provided user of your message. This is a one-time notification which does not require acknowledgement, so it is best to make it clear with your text that you expect a response.
* Add the user as a '''blocking reviewer''' and indicate your intent by including a comment. This will prevent the Phabricator revision from becoming '''Accepted''' until the indicated user has either '''Accepted''' the revision, '''Resigned''' as a reviewer, or been removed as a reviewer. Since this is re-purposing the normal request for review it is best to include a comment making your intended use clear.

=== Request '''feedback?''' on a patch ===
Phabricator does not distinguish between asking for final review and requesting other forms of feedback. It is best to use an explicit comment explaining you would like feedback, along with one of:
* Requesting review from the user you'd like feedback from.
* Using an '''@''' mention. Including '''@<username>''' in a comment on Phabricator will notify the provided user of your message. This is a one-time notification which does not require acknowledgement, so it is best to make it clear with your text that you are expecting feedback.

=== Upload WIP patches ===

Not sure if this workflow should be integrated into the request feedback section.

=== Request '''review?''' on a patch ===
TODO: moz-phab, add reviewer, blocking vs non-blocking.

=== Clear a '''review?''' flag from your own patch ===
TODO: plan changes vs edit reviewers.

=== Clear a '''review?''' flag asked of you ===
TODO: request changes vs resign.

=== Request uplift approval on a patch ===
TODO

=== Request '''checkin-needed''' to land your changes ===
TODO: Move to phabricator tag.

=== Request '''review?''' of an image file ===
TODO: Pholio

=== Request '''sec-approval?''' on a patch ===
TODO

DeveloperServices

2018-12-18T22:02:56Z

Kmoir: /* Roster */

<section begin=summary />{{TeamSummary
|teamtitle=Developer Services
|teampagelocation=DeveloperServices
|teamlead=Kendall Libby
|mailinglist=developer-services@mozilla.org
|description=The Developer Services team works on version control systems and related tools at Mozilla.
* We work on hg, git-internal, and vcs-sync.
* We support the try repository.
* We also provide operational support for Treeherder (DB only), and DXR.
}}<section end=summary />

= Oncall for VCS =
"Oncall" is really "escalation path" for services we handle. Start with the service owner, then irc channel (below), then escalate to oncall if no response.
{| class="fullwidth-table"
| style="background:#EFEFEF;" |'''Role'''
| style="background:#EFEFEF;" |'''Name'''
| style="background:#EFEFEF;" |'''IRC Contact'''
|-
| Primary
| Kendall Libby
| fubar
|-
| Backup on call
| Jacob Watkins
| dividehex
|}
As always with service issues, please indicate urgency and contact information in any page or voice mail. Using Nagios paging is most reliable. For less urgent contact methods, see [[#contact|contact info]] below.

= Roster =
{| class="fullwidth-table"
| style="background:#EFEFEF;" |'''Name'''
| style="background:#EFEFEF;" |'''Contact'''
| style="background:#EFEFEF;" | '''Location'''
| style="background:#EFEFEF;" | '''Project(s)'''
| style="background:#EFEFEF;" | '''Working Hours'''
|-
|Kendall Libby
|fubar
|Boston, MA, USA
|Treeherder (DB), DXR, hg, git-internal
|ET
|-
|Jake Watkins
|dividehex
|California, USA
|Autoland, Treeherder (DB)
|PT
|-
|David House
|dhouse
|Montana, USA
|DXR, vcssync
|MT
|-
|Connor Sheehan
|sheehan
|Toronto, CA
|hg
|ET
|-
|Steven Macleod
|smacleod
|Toronto, CA
|hg
|ET
|-
|Byron Jones
|glob
|Perth, AUS
|hg
|GMT+08:00
|}

<div id="contact"> </div>
To contact all of the above, use the list developer-services@mozilla.org or [ircs://irc.mozilla.org/#vcs #vcs].

Additional folks work on various modifications and extensions to the version control systems. They can be reached at dev-version-control@lists.mozilla.org or [ircs://irc.mozilla.org/#vcs #vcs].

The other systems we support follow the "devops" model -- the first point of contact for operations and support are the following channels on irc:
* [ircs://irc.mozilla.org/#bteam #bteam] for BMO
* [ircs://irc.mozilla.org/#treeherder #treeherder] for Treeherder ([https://groups.google.com/forum/#!forum/mozilla.tools.treeherder discussion list])
* [ircs://irc.mozilla.org/#mozreview #mozreview] for MozReview
* [ircs://irc.mozilla.org/#static #static] for DXR

= Meetings =
* [[DeveloperServices/TeamMeetings|https://wiki.mozilla.org/DeveloperServices/TeamMeetings]]

= Services =
* [[/Account Recovery]] - how to re-enable account access
* [[/git]] - details of our Git DVCS system
* [[/hg]] - details of our Mercurial DVCS system

DeveloperServices

2018-07-17T19:21:12Z

Kmoir: /* Roster */

<section begin=summary />{{TeamSummary
|teamtitle=Developer Services
|teampagelocation=DeveloperServices
|teamlead=Kendall Libby
|mailinglist=developer-services@mozilla.org
|description=The Developer Services team works on version control systems and related tools at Mozilla.
* We work on hg, git-internal, and svn.
* We support the try repository.
* We also provide operational support for BMO, MozReview, Treeherder (DB only), and DXR.
}}<section end=summary />

= Oncall for VCS =
"Oncall" is really "escalation path" for services we handle. Start with the service owner, then irc channel (below), then escalate to oncall if no response.
{| class="fullwidth-table"
| style="background:#EFEFEF;" |'''Role'''
| style="background:#EFEFEF;" |'''Name'''
| style="background:#EFEFEF;" |'''IRC Contact'''
|-
| Primary
| Kendall Libby
| fubar
|-
| Backup on call
| Jacob Watkins
| dividehex
|}
As always with service issues, please indicate urgency and contact information in any page or voice mail. Using Nagios paging is most reliable. For less urgent contact methods, see [[#contact|contact info]] below.

= Roster =
{| class="fullwidth-table"
| style="background:#EFEFEF;" |'''Name'''
| style="background:#EFEFEF;" |'''Contact'''
| style="background:#EFEFEF;" | '''Location'''
| style="background:#EFEFEF;" | '''Project(s)'''
| style="background:#EFEFEF;" | '''Working Hours'''
|-
|Kendall Libby
|fubar
|Boston, MA, USA
|BMO, Treeherder (DB), MozReview, DXR, hg, git
|ET
|-
|Jake Watkins
|dividehex
|California, USA
|MozReview, Autoland, Treeherder (DB)
|PT
|-
|David House
|dhouse
|Montana, USA
|BMO, DXR, vcssync
|MT
|-
|Gregory Szorc
|gps
|San Francisco, CA, USA
|hg, MozReview, build module owner
|PT
|-
|Connor Sheehan
|sheehan
|Toronto, CA
|hg
|ET
|-
|Steven Macleod
|smacleod
|Toronto, CA
|hg
|ET
|-
|Byron Jones
|glob
|Perth, AUS
|hg
|GMT+08:00
|}

<div id="contact"> </div>
To contact all of the above, use the list developer-services@mozilla.org or [ircs://irc.mozilla.org/#vcs #vcs].

Additional folks work on various modifications and extensions to the version control systems. They can be reached at dev-version-control@lists.mozilla.org or [ircs://irc.mozilla.org/#vcs #vcs].

The other systems we support follow the "devops" model -- the first point of contact for operations and support are the following channels on irc:
* [ircs://irc.mozilla.org/#bteam #bteam] for BMO
* [ircs://irc.mozilla.org/#treeherder #treeherder] for Treeherder ([https://groups.google.com/forum/#!forum/mozilla.tools.treeherder discussion list])
* [ircs://irc.mozilla.org/#mozreview #mozreview] for MozReview
* [ircs://irc.mozilla.org/#static #static] for DXR

= Meetings =
* [[DeveloperServices/TeamMeetings|https://wiki.mozilla.org/DeveloperServices/TeamMeetings]]

= Services =
* [[/Account Recovery]] - how to re-enable account access
* [[/git]] - details of our Git DVCS system
* [[/hg]] - details of our Mercurial DVCS system

Engineering Workflow/Road Map

2018-05-10T16:33:01Z

Kmoir: /* Build backend */

= Engineering Workflow Road Map =

These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.

== Current ==

=== Phabricator integration ===

Although our [[Phabricator]] instance has been up since September 2017, there has been ongoing work to integrate it into our tools, most notably [[BMO]]. There is a tracking bug ({{bug|1381498}}) for the remaining work before we announce general availability and start encouraging people to move over.

The last remaining item of note is forking Arcanist so that it works with git-cinnabar ({{bug|1443375}}).

Impact:
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
* Some integration with existing tools and processes, balancing complexity and disruption.
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community.

=== Lando ===

[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is (or at least will be) functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.

Impact:
* One-button landing offers an easier way to land code and minimal context switching from reviewing code.
* Queued landings due to tree closures means developers don't have to be around when they open to land their code.
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).

=== Vendor manifests ===

Provide a common structure, process, and tools for managing third-party code in mozilla-central. An [https://groups.google.com/forum/?pli=1#!topic/mozilla.dev.platform/-B8MECCeJnM Intent to Require] was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug ({{bug|1454867}}) for progress.

Impact:
* Easier to identify and find third-party code in mozilla-central.
* Easier to determine metadata about third-party code, including original source and current revision.
* Easier to update third-party code.

=== BMO usability ===

Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.

Impact:
* User delight!
* Less friction for new users.
* Better expressibility and accuracy in comments.

=== Hg scalability and efficiency ===

Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg

Impact:
* Multi-region redundancy and autoscaling for hgweb servers for additional resiliency
* Faster checkouts to enable shorter end to end times for local and CI builds

=== Build backend ===

Implement a new build backend for both local and CI builds. Deploy a new distributed sccache system written in Rust to enable fast, secure local builds in offices. Implement local build telemetry that submits data to the generic data ingestion pipeline to allow us to identify and reduce inefficiencies in local builds.

Impact
* Reduce the incidence of clobber builds
* Faster incremental builds
* Secure and IT supported mechanism for faster local builds in offices

== Future ==

=== Enforced reviewers ===

=== Intent-to-land ===

=== Lando Try support ===

=== Lando confidential-revision support ===

=== Lando stacked landing ===

=== cinnabar-less Git(Hub) gecko-dev support ===

=== Lando headless client API support ===

=== Lando and Phabricator backout support ===

=== Transplant service improvements ===

=== Lando Git support ===

=== Auth0 integration with BMO ===

=== Markdown in BMO ===

=== Run tests from source directory in local and ci builds ===

=== Overhauling release repositories and build configurations to use named branches ===

=== Support for platforms other than Linux for new build backend ===

=== Cross-compile a Windows build with clang-cl from Linux ===

=== Artifact caching experiments for builds ===

Engineering Workflow/Road Map

2018-05-10T16:26:17Z

Kmoir: /* New build backend */

= Engineering Workflow Road Map =

These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.

== Current ==

=== Phabricator integration ===

Although our [[Phabricator]] instance has been up since September 2017, there has been ongoing work to integrate it into our tools, most notably [[BMO]]. There is a tracking bug ({{bug|1381498}}) for the remaining work before we announce general availability and start encouraging people to move over.

The last remaining item of note is forking Arcanist so that it works with git-cinnabar ({{bug|1443375}}).

Impact:
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
* Some integration with existing tools and processes, balancing complexity and disruption.
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community.

=== Lando ===

[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is (or at least will be) functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.

Impact:
* One-button landing offers an easier way to land code and minimal context switching from reviewing code.
* Queued landings due to tree closures means developers don't have to be around when they open to land their code.
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).

=== Vendor manifests ===

Provide a common structure, process, and tools for managing third-party code in mozilla-central. An [https://groups.google.com/forum/?pli=1#!topic/mozilla.dev.platform/-B8MECCeJnM Intent to Require] was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug ({{bug|1454867}}) for progress.

Impact:
* Easier to identify and find third-party code in mozilla-central.
* Easier to determine metadata about third-party code, including original source and current revision.
* Easier to update third-party code.

=== BMO usability ===

Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.

Impact:
* User delight!
* Less friction for new users.
* Better expressibility and accuracy in comments.

=== Hg scalability and efficiency ===

Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg

Impact:
* Multi-region redundancy and autoscaling for hgweb servers for additional resiliency
* Faster checkouts to enable shorter end to end times for local and CI builds

=== Build backend ===

Implement a new build backend for both local and CI builds. Implement a new distributed sccache system written in Rust to enable fast, secure local builds in offices.

Impact
* Reduce the incidence of clobber builds
* Faster incremental builds
* Secure and IT supported mechanism for faster local builds in offices

== Future ==

=== Enforced reviewers ===

=== Intent-to-land ===

=== Lando Try support ===

=== Lando confidential-revision support ===

=== Lando stacked landing ===

=== cinnabar-less Git(Hub) gecko-dev support ===

=== Lando headless client API support ===

=== Lando and Phabricator backout support ===

=== Transplant service improvements ===

=== Lando Git support ===

=== Auth0 integration with BMO ===

=== Markdown in BMO ===

=== Run tests from source directory in local and ci builds ===

=== Overhauling release repositories and build configurations to use named branches ===

=== Support for platforms other than Linux for new build backend ===

=== Cross-compile a Windows build with clang-cl from Linux ===

=== Artifact caching experiments for builds ===

Engineering Workflow/Road Map

2018-05-10T16:21:59Z

Kmoir: /* Future */

= Engineering Workflow Road Map =

These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.

== Current ==

=== Phabricator integration ===

Although our [[Phabricator]] instance has been up since September 2017, there has been ongoing work to integrate it into our tools, most notably [[BMO]]. There is a tracking bug ({{bug|1381498}}) for the remaining work before we announce general availability and start encouraging people to move over.

The last remaining item of note is forking Arcanist so that it works with git-cinnabar ({{bug|1443375}}).

Impact:
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
* Some integration with existing tools and processes, balancing complexity and disruption.
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community.

=== Lando ===

[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is (or at least will be) functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.

Impact:
* One-button landing offers an easier way to land code and minimal context switching from reviewing code.
* Queued landings due to tree closures means developers don't have to be around when they open to land their code.
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).

=== Vendor manifests ===

Provide a common structure, process, and tools for managing third-party code in mozilla-central. An [https://groups.google.com/forum/?pli=1#!topic/mozilla.dev.platform/-B8MECCeJnM Intent to Require] was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug ({{bug|1454867}}) for progress.

Impact:
* Easier to identify and find third-party code in mozilla-central.
* Easier to determine metadata about third-party code, including original source and current revision.
* Easier to update third-party code.

=== BMO usability ===

Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.

Impact:
* User delight!
* Less friction for new users.
* Better expressibility and accuracy in comments.

=== Hg scalability and efficiency ===

Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg

Impact:
* Multi-region redundancy and autoscaling for hgweb servers for additional resiliency
* Faster checkouts to enable shorter end to end times for local and CI builds

=== New build backend ===

Implement a new build backend for both local builds and builds in CI

Impact
* Reduce the incidence of clobber builds
* Faster incremental builds

== Future ==

=== Enforced reviewers ===

=== Intent-to-land ===

=== Lando Try support ===

=== Lando confidential-revision support ===

=== Lando stacked landing ===

=== cinnabar-less Git(Hub) gecko-dev support ===

=== Lando headless client API support ===

=== Lando and Phabricator backout support ===

=== Transplant service improvements ===

=== Lando Git support ===

=== Auth0 integration with BMO ===

=== Markdown in BMO ===

=== Run tests from source directory in local and ci builds ===

=== Overhauling release repositories and build configurations to use named branches ===

=== Support for platforms other than Linux for new build backend ===

=== Cross-compile a Windows build with clang-cl from Linux ===

=== Artifact caching experiments for builds ===

Engineering Workflow/Road Map

2018-05-10T16:09:54Z

Kmoir: /* Hg scalability and efficiency */

= Engineering Workflow Road Map =

These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.

== Current ==

=== Phabricator integration ===

Although our [[Phabricator]] instance has been up since September 2017, there has been ongoing work to integrate it into our tools, most notably [[BMO]]. There is a tracking bug ({{bug|1381498}}) for the remaining work before we announce general availability and start encouraging people to move over.

The last remaining item of note is forking Arcanist so that it works with git-cinnabar ({{bug|1443375}}).

Impact:
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
* Some integration with existing tools and processes, balancing complexity and disruption.
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community.

=== Lando ===

[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is (or at least will be) functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.

Impact:
* One-button landing offers an easier way to land code and minimal context switching from reviewing code.
* Queued landings due to tree closures means developers don't have to be around when they open to land their code.
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).

=== Vendor manifests ===

Provide a common structure, process, and tools for managing third-party code in mozilla-central. An [https://groups.google.com/forum/?pli=1#!topic/mozilla.dev.platform/-B8MECCeJnM Intent to Require] was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug ({{bug|1454867}}) for progress.

Impact:
* Easier to identify and find third-party code in mozilla-central.
* Easier to determine metadata about third-party code, including original source and current revision.
* Easier to update third-party code.

=== BMO usability ===

Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.

Impact:
* User delight!
* Less friction for new users.
* Better expressibility and accuracy in comments.

=== Hg scalability and efficiency ===

Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg

Impact:
* Multi-region redundancy and autoscaling for hgweb servers for additional resiliency
* Faster checkouts to enable shorter end to end times for local and CI builds

=== New build backend ===

Implement a new build backend for both local builds and builds in CI

Impact
* Reduce the incidence of clobber builds
* Faster incremental builds

== Future ==

=== Enforced reviewers ===

=== Intent-to-land ===

=== Lando Try support ===

=== Lando confidential-revision support ===

=== Lando stacked landing ===

=== cinnabar-less Git(Hub) gecko-dev support ===

=== Lando headless client API support ===

=== Lando and Phabricator backout support ===

=== Transplant service improvements ===

=== Lando Git support ===

=== Auth0 integration with BMO ===

=== Markdown in BMO ===

Engineering Workflow/Road Map

2018-05-10T15:23:00Z

Kmoir: Add developer workflow items

= Engineering Workflow Road Map =

These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.

== Current ==

=== Phabricator integration ===

Although our [[Phabricator]] instance has been up since September 2017, there has been ongoing work to integrate it into our tools, most notably [[BMO]]. There is a tracking bug ({{bug|1381498}}) for the remaining work before we announce general availability and start encouraging people to move over.

The last remaining item of note is forking Arcanist so that it works with git-cinnabar ({{bug|1443375}}).

Impact:
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
* Some integration with existing tools and processes, balancing complexity and disruption.
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community.

=== Lando ===

[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is (or at least will be) functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.

Impact:
* One-button landing offers an easier way to land code and minimal context switching from reviewing code.
* Queued landings due to tree closures means developers don't have to be around when they open to land their code.
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).

=== Vendor manifests ===

Provide a common structure, process, and tools for managing third-party code in mozilla-central. An [https://groups.google.com/forum/?pli=1#!topic/mozilla.dev.platform/-B8MECCeJnM Intent to Require] was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug ({{bug|1454867}}) for progress.

Impact:
* Easier to identify and find third-party code in mozilla-central.
* Easier to determine metadata about third-party code, including original source and current revision.
* Easier to update third-party code.

=== BMO usability ===

Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.

Impact:
* User delight!
* Less friction for new users.
* Better expressibility and accuracy in comments.

=== Hg scalability and efficiency ===

Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg

Impact
* Multi-region redundancy and autoscaling for hgweb servers for additional resiliency
* Faster checkouts to enable shorter end to end times for local and CI builds

=== New build backend ===

Implement a new build backend for both local builds and builds in CI

Impact
* Reduce the incidence of clobber builds
* Faster incremental builds

== Future ==

=== Enforced reviewers ===

=== Intent-to-land ===

=== Lando Try support ===

=== Lando confidential-revision support ===

=== Lando stacked landing ===

=== cinnabar-less Git(Hub) gecko-dev support ===

=== Lando headless client API support ===

=== Lando and Phabricator backout support ===

=== Transplant service improvements ===

=== Lando Git support ===

=== Auth0 integration with BMO ===

=== Markdown in BMO ===

Template:NextReleaseDate

2017-09-21T15:22:43Z

Kmoir:

2017-09-28

Template:Version/Gecko/release/current

2017-09-21T15:09:43Z

Kmoir: Merge day

Template:Version/Gecko/beta/current

2017-09-21T15:09:37Z

Kmoir: Merge day

Template:Version/Gecko/central/current

2017-09-21T15:09:30Z

Kmoir: Merge day

Template:Version/Gecko/release/next

2017-09-21T15:09:23Z

Kmoir: Merge day

Template:NextReleaseDate

2017-09-21T15:09:16Z

Kmoir: Merge day

2017-10-03

ReleaseEngineering/How To/Update TryChooser

2017-08-15T15:23:50Z

Kmoir:

{{Release Engineering How To|Update the Try Syntax}}
{{Release Engineering Deployment|trychooser}}

== Updating Trychooser app ==
We need to keep the try syntax chooser webpage located at https://github.com/mozilla-releng/services/releng_frontend/src/static/trychooser/index.html up to date so developers will continue to '''use''' it and we benefit from the lowered build/test load.

To keep the site up to date, check out the [https://github.com/mozilla-releng/services/releng_frontend/src/static/trychooser/index.html] and update index.html with new test/talos suite names - be sure to change both the value and the label. Eg:

<pre><li><label><input type="checkbox" name="talos_suite" value="SUITE_NAME">SUITE_NAME</label></li></pre>

== Deploying Trychooser ==
Trychooser is webops-style app, see [[ReleaseEngineering/How_To/Update_a_Releng_Cluster_Site]] where $site is 'trychooser'.

ReleaseEngineering/How To/Adjust a Worker's Master

2017-08-14T20:42:18Z

Kmoir: Created page with "{{Release Engineering How To|Adjust a Worker's Master}} Edit by editing the machine in [https://github.com/catlee/tools/tree/master/lib/python/slavealloc] to either adjust its..."

{{Release Engineering How To|Adjust a Worker's Master}}
Edit by editing the machine in [https://github.com/catlee/tools/tree/master/lib/python/slavealloc] to either adjust its pool or (not preferred outside of development) lock it to a master. Then, if necessary, reboot the worker machine.

ReleaseEngineering/How To/Adjust a Slave's Master

2017-08-14T20:41:24Z

Kmoir: Blanked the page

ReleaseEngineering/How To/Adjust a Slave's Master

2017-08-14T20:40:36Z

Kmoir:

{{Release Engineering How To|Adjust a Worker's Master}}
In the era of slavealloc, this is done by editing the slave in [https://github.com/catlee/tools/tree/master/lib/python/slavealloc] to either adjust its pool or (not preferred outside of development) lock it to a master. Then, if necessary, reboot the slave.