View source for Glean/Adding or changing Glean metric types

{{Warning|1=This process is deprecated. The new process requires filing a bug in the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Data+Platform+and+Tools&component=Glean%3A+SDK Data Platform and Tools::Glean SDK] Bugzilla component and creating a proposal to discuss the requested change. |heading=Process deprecated}}

=<big>'''Important''': read me first</big>=
Are you just looking to file a request for a new metric type or changing and existing one? Then '''all you need to do is filing a bug in the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Data%20Platform%20and%20Tools&component=Glean%20Metric%20Types Data platforms &amp; tools::Glean Metric Types component] in Bugzilla.'''

If you are interested in knowing more about how your request is handled in the Glean ecosystem, please keep on reading!

= Background =
Glean is Mozilla’s modern product analytics and telemetry solution that provides data for our new products. It aims to be easy to integrate, reliable and transparent by providing SDKs and integrated tools.

One of the Glean principles is to provide [https://mozilla.github.io/glean/book/user/metrics/index.html higher-level metric types] that map semantically to what users want to measure: for example, it is helpful for both validation and analysis to know that something is a [https://mozilla.github.io/glean/book/user/metrics/counter.html counter] rather than just a more general &quot;integer&quot;, as this implies that its value cannot be less than or equal to 0.

The current offered metric types were designed to cover the majority of Mozilla use-cases, but we know that new use-cases will come up. Some have already ([https://bugzilla.mozilla.org/show_bug.cgi?id=1557299 UrlMetricType], [https://bugzilla.mozilla.org/show_bug.cgi?id=1557353 StringList vs StringSet], [https://bugzilla.mozilla.org/show_bug.cgi?id=1557358 dropping labelled booleans], [https://bugzilla.mozilla.org/show_bug.cgi?id=1595723 coarse timing distributions], [https://bugzilla.mozilla.org/show_bug.cgi?id=1597874 error stacks], [https://github.com/mozilla/application-services/issues/2127 changes to quantity/counters], enumerations, ratios).

= Motivation =

The base set of metric types offered by Glean, from our [https://docs.google.com/document/d/186GI_2ljdqWqWTBk5P3QjHD3BwpMyd8kRayRAuz1-dk/edit#heading=h.9x8ipfgxldmm initial design document], were designed by going [https://docs.google.com/document/d/186GI_2ljdqWqWTBk5P3QjHD3BwpMyd8kRayRAuz1-dk/edit#heading=h.v20313883vlc through the pings sent by our mobile products] and identifying the higher level metric types required to reach feature parity with the existing telemetry system. The design document was reviewed by data engineering and that process helped smoothing out some of the rough edges of the existing legacy system. However, no metric type was fundamentally new and this meant we did not have to answer questions such as:

* can the type answer the business questions we’re adding instrumentation for?
* can the type be used to leak user data?
* does the metric type require custom processing when ingested?

While the Glean/Telemetry team has experience and historical knowledge that could inform the answer to these questions, other teams have a much deeper expertise on these topics. Their opinions and recommendations are vital for the process of adding new metric types.

For this reason, the Glean/Telemetry team alone cannot make a call about whether or not a request for a new metric type is reasonable. The Glean end-to-end tech lead must be responsible for that and must base their decision on the feedback from the consulted stakeholders.

= The committee =

As mentioned, adding a new metric type to the Glean ecosystem does not exclusively have implications for the Glean SDKs. All the teams involved in the Glean ecosystem, in addition to the team or individuals (NOTE: Teams or individuals part of the committee can file requests as well, if needed.) requesting the new metric type or changes to the existing ones, need to be consulted.

This process structure attempts to bring together all the points of view of the different stakeholders of the Glean ecosystem. The volume of the incoming requests is expected to decrease over time, as the Glean offering becomes more complete and comprehensive.

The following sections depicts the three roles identified to move the process along.

== The requester ==

This is the team or individuals asking for the change or the addition of a new metric type in Glean.

{| class="wikitable"
|-
! Team Name !! Member name(s) !! Domain of expertise/angle
|-
| Requester || (depends on the request)|| <ul><li>Why is the new metric type required?</li><li>
What is the data that needs to be collected?</li><li>What is the specific question that the data needs to answer?</li></ul>
|}

== The triage owner ==

This is the team or individuals that are responsible for triaging incoming requests and make sure that all the requests are acted on.

{| class="wikitable"
|-
! Team Name !! Member name(s) !! Domain of expertise/angle
|-
| Triage owner || The owner of the bugzilla component || <ul><li>Is not required to be involved in the decision making process.</li><li>
Guarantees that each request is acted on within 6 business days.</li></ul>
|}

== The Glean end-to-end tech lead ==

The individual who has an end-to-end understanding of the Glean ecosystem, oversees its strategy and long-term goals.


{| class="wikitable"
|-
! Team Name !! Member name(s) !! Domain of expertise/angle
|-
| Glean end-to-end tech lead || Jan-Erik Rediger || <ul><li>Does the requested type fit into the product strategy?</li><li>
Does the cost for implementing this outweigh its benefits?</li><li>If the requested change can be technically achieved, should it still be done?</li></ul>
|}

== The consulted stakeholders ==

This table attempts to capture the stakeholders that need to be consulted with, as they work or have to do with the Glean ecosystem.

{{Note|Teams can declare that the triage owner can rely on any individual on the team, instead of specific owner names.  However, teams that do not want specific individuals to be mentioned are '''required''' to provide a mechanism for the triage owner to reach out to the team and make sure to receive an answer in a timely manner.}}

{| class="wikitable"
|-
! Team Name !! Member name(s) !! Domain of expertise/angle
|-
| Data Science || Any available data scientist ([https://jira.mozilla.com file a JIRA ticket]) || <ul><li>How would the new type impact analyses?</li><li>Can the requested type be used to answer meaningful questions in a scalable way?</li><li>Can the use-case be satisfied by using any existing metric type?</li><li>Will the data be easy to misinterpret, and are there ways to minimize that?</li><li>Help vet definitions posted to organization</li></ul>
|-
| Data Stewards || Any available steward ([https://wiki.mozilla.org/Firefox/Data_Collection to reach out]) || <ul><li>Does the new type pose privacy challenges?</li><li>Should the data-review process be changed to address this new type?</li></ul>
|-
| Data Tools || <ul><li>Rob Miller</li></ul> || <ul><li>Can this metric type be unambiguously aggregated?</li><li>Will this metric pose problems when trying to plot it?</li><li>Can this metric type be accessed?</li><li>Will this metric type introduce previously unseen complexity to our aggregation process?</li><li>Will this metric type's data (or its aggregate data) introduce new complexity when we import it into the low latency dbs used by our web-based data tools?</li></ul>
|-
| Pipeline || <ul><li>Mark Reid</li></ul> || <ul><li>Would this create problems with the payload?</li><li>How would the new type translate to BQ types?</li><li>Can the new type be represented at all in a convenient way?</li><li>Can the use-case be satisfied by using any existing metric type?</li></ul>
|-
| Client (SDKs) || <ul><li>Travis Long</li><li>Jan-Erik Rediger</li></ul> || <ul><li>Would the new type violate any Glean SDKs principles?</li><li>Would the API be reasonably ergonomic?</li><li>Would the API work on all the supported platforms?</li><li>Can the use-case be satisfied by using any existing metric type?</li><li>Are there any performance concerns? (speed, size on disk, bandwidth, etc.)</li></ul>
|}

= The processes =

This section outlines the two processes involved in changing or adding metric types. The workflow starts with the user making the request. After that, the requested changes are discussed among the different stakeholders listed in the previous sections.

== Requirements ==

Before any of the following processes can take place, the following requirements need to be satisfied:

# Representatives for each team of the committee must be nominated and added to this document in the [[Glean/Adding or changing Glean metric types#The_committee|committee]] section.
# Managers or representatives for each team of the committee must sign-off on this proposal, at the top of the document.
# A new Bugzilla component, &quot;Data platforms &amp; tools::Glean Metric Types&quot;, must be created.
# All the members of the committee must subscribe to the Bugzilla component.
# Representatives for each team must nominate the triage owner for the Bugzilla component.
# A bugzilla form for submitting requests must be created (see [[Glean/Adding or changing Glean metric types#The_custom_Bugzilla_form|the related paragraph]]).
# A [[Glean/Adding or changing Glean metric types#The_discussion_document|discussion document template]] must be available to be forked by the triage owner.
# Documentation for requesting new metric types or changing existing ones must be available on the [https://mozilla.github.io/glean/book/index.html Book of Glean].

== The proposal process ==

This section describes how users should file a request for either changing or adding a new metric type.

# User files a bug using a custom form in the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Data%20Platform%20and%20Tools&component=Glean%20Metric%20Types Data platforms &amp; tools::Glean Metric Types component] in Bugzilla.
# The triage owner of the Bugzilla component prioritizes this within 6 business days and kicks off the [[#heading=h.4fjjr7hkx33l|decision making process]].
# Once the decision process is completed, the bug is closed with a comment outlining the decision that was made.

=== The custom Bugzilla form ===

The form contains the following information:

* A description of the data that needs to be recorded.
* A raw sample of the data that needs to be recorded. This is in the abstract, and not any particular implementation details about its representation in the payload or the database.
* The business question/use-case that requires the data to be recorded.
* How the data would be consumed.
* Why existing metric types are not enough.
* The timeline by which the data needs to be collected.

== The decision making process ==

This section outlines the process with which a decision is made when a new request comes in.

<ol style="list-style-type: decimal;">
<li><p>The [[Glean/Adding or changing Glean metric types#The_triage_owner|triage owner]] of the Bugzilla component triages the request.</p></li>
<li><p>The triage owner copies all the information from the bug into a [[Glean/Adding or changing Glean metric types#The_discussion_document|document]] (for allowing easier async communication through comments). Rich text should be copied, not raw markdown.</p></li>
<li><p>The triage owner attaches the document to the bug.</p>
<ul><li>The triage owner files a [https://jira.mozilla.com ticket in JIRA] for the Data Science team with the same template used for Bugzilla.</li>
<li>The triage owner adds a link to the JIRA issue linking to Bugzilla and the other way around (any required update and public communication will still happen exclusively on Bugzilla, the JIRA ticket will exclusively be used to track the work item by Data Science).</li></ul>
</li>
<li><p>The [[Glean/Adding or changing Glean metric types#The_Glean_end-to-end_tech_lead|Glean end-to-end tech lead]] designates the initial group '''required''' to suggest a technical solution (non-required stakeholders are still allowed and encouraged to contribute at this stage).</p>
<ul><li>This is the '''design''' stage and should last, at most, 2 weeks.</li>
<li>The [[Glean/Adding or changing Glean metric types#The_triage_owner|triage owner]] flags all the members of the committee on Bugzilla with a needinfo request (and comments on JIRA).</li>
</ul></li>
<li><p>Members of the committee discuss the content of the document using the google docs commenting system.</p></li>
<li><p>The [[Glean/Adding or changing Glean metric types#The_Glean_end-to-end_tech_lead|Glean end-to-end tech lead]] ends the design stage once the proposal reaches maturity.</p>
<ul><li>The [[Glean/Adding or changing Glean metric types#The_Glean_end-to-end_tech_lead|Glean end-to-end tech lead]] writes a short summary of the proposal at the top of the document.</li>
<li>The proposal moves to the '''comment''' stage which should last 6 business days.</li>
<li>The [[Glean/Adding or changing Glean metric types#The_triage_owner|triage owner]] flags all the members of the committee on Bugzilla with a needinfo request (and comments on JIRA).</li>
<li>Reviewers are expected to leave comments within 6 business days.</li></ul>
</li>
<li><p>After 6 business days, the '''comment''' stage is declared finished and the [[Glean/Adding or changing Glean metric types#The_Glean_end-to-end_tech_lead|Glean end-to-end tech lead]] marks the proposal as either '''approved''' or '''rejected'''.</p></li>
<li><p>(''optional'') The [[Glean/Adding or changing Glean metric types#The_Glean_end-to-end_tech_lead|Glean end-to-end tech lead]] can call for a meeting to be organized to further discuss the document, if needed.</p></li>
<li><p>If more information is required from the ''requester'', they are flagged on the document with a comment.</p></li>
<li><p>The teams that will need to do the implementation work (whatever implementation work applies for fully supporting the discussed work in the Glean ecosystem including, but not limited to, SDKs, pipeline, tooling, ...) will need to provide an estimate of the required work. The effort has to be weighted in the final decision.</p></li>
<li><p>All the members of the committee sign off on the document (in the traceability matrix at the top of the document) '''or '''leave a comment about their concern with the proposal.</p></li>
<li><p>The triage owner closes the bug and leaves a comment with the decision outcome.</p>
<ul>
<li><p>The triage owner makes the document publicly accessible for the public to consult, if needed, by attaching it to the bug as a Markdown document.</p></li>
<li><p>'''If the decision is to proceed''' with the change, the triage owner files the required bugs in the relevant components (e.g. for new metric types, file [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&bug_type=defect&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox100=---&cf_status_firefox101=---&cf_status_firefox99=---&cf_status_firefox_esr91=---&cf_tracking_firefox100=---&cf_tracking_firefox101=---&cf_tracking_firefox99=---&cf_tracking_firefox_esr91=---&component=Glean%3A%20SDK&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-721=X&flag_type-737=X&flag_type-799=X&flag_type-800=X&flag_type-803=X&flag_type-936=X&flag_type-947=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=P3&product=Data%20Platform%20and%20Tools&rep_platform=Unspecified&status_whiteboard=%5Bglean-sdk%3Am%3F%5D&target_milestone=---&version=unspecified a new bug for the Glean SDKs], see [https://mozilla.github.io/glean/dev/core/new-metric-type.html "Adding a new metric type"] for details about the implementation).</p></li>
<li><p>'''If the decision is to not make any change''' to metric types, the committee must identify and recommend an existing metric type to use instead.</p></li></ul>
</li></ol>

'''Important''': the triage owner will be responsible for driving the conversations and make sure that a decision is reached within the expected timeframe.

=== The discussion document ===

In addition to a copy of all the data present in the [[Glean/Adding or changing Glean metric types#The_custom_Bugzilla_form|bugzilla form]], the discussion document must contain the following sections:

* a traceability matrix at the top of the document, listing all the committee members for facilitating the discovery of the sign-offs;
** this should have at least two sections: '''Glean end-to-end tech lead '''and '''consulted'''.
* a section for each team represented in the committee to add their considerations about the request; ideally each section should summarise their decision with respect to the proposal and highlight any critical issue;

The template is available [https://docs.google.com/document/d/18Kw4aVSpDs9dQS_a-BPRGRxibjCttftH0F83UksfK9w here] (v2).

= Q&amp;A =

'''Q''': What’s the acceptance criteria for changes/new metrics?

A: In addition to the food for thought from [[#heading=h.pis3bipl0x4v|this section]], the committee must consider the following aspects:

* the proposed changes/new metric '''must not '''break any of the Glean principles;
* if changing an existing metric type, the change '''must not''' break any existing usage;
** an email must be sent to [mailto:fx-data-dev@mozilla.org fx-data-dev@mozilla.org] to announce the intent to change the metric.
** if it breaks existing uses, the requester must commit to fix the breakage (or partner with the appropriate stakeholders, if identifiable) as agreed with the committee.

'''Q''': What if the committee disagrees on the decision for the changes/new metrics?

A: The Glean end-to-end tech lead has veto power: they can either decide to proceed with the change (e.g. because of strategic product needs) or to reject the request (e.g. because of cost/benefit ratio)