Firefox/Accounts/Progress Report/2017Q3: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Corrected August 21st update)
(completed updates for objective 3)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
__NOTOC__
__NOTOC__
<big><big>'''August 21, 2017'''</big></big>
<big><big>'''September 25, 2017'''</big></big>


== Q3 OKR Progress ==
== Q3 OKR Progress ==
Line 27: Line 27:
|align="center"|70%
|align="center"|70%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|100%
|align="left"|
|align="center"|1.0
|
|
Priorities for support of Quantum/Photon in 57:
Priorities for support of Quantum/Photon in 57:
*[https://waffle.io/mozilla/fxa?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Issues for 57 release.]
*[https://waffle.io/mozilla/fxa?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Issues for 57 release.]
Carried over from the last sprint:
*[https://github.com/mozilla/fxa-auth-server/issues/1285 Issue 1285: Device names with non-BMP characters are rejected.]
*[https://github.com/mozilla/fxa-auth-db-mysql/issues/229 Issue 229: Update device name column to utf8mb4]
|
|
*25% remaining to fix. Fixed some bugs this cycle.
*Done!
*Animation on svg’s needs to be started. Once we’re done with the non-BMP char device names (unicode), that will free up some time (#5 on the list).
*Rest seem pretty manageable but just need to get them to the top of the list.
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Burndown Chart]
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Burndown Chart]
|-
|-
Line 47: Line 42:
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|100%
|align="left"|
|align="center"|1.0
|
|
Quality Priorities for Q3:
Quality Priorities for Q3:
*[https://waffle.io/mozilla/fxa?milestone=FxA-131:%20Q3%20 Quality related P1s that we’ve identified to fix in Q3.]
*[https://waffle.io/mozilla/fxa?milestone=FxA-131:%20Q3%20 Quality related P1s that we’ve identified to fix in Q3.]
Active in this sprint:
*[https://github.com/mozilla/fxa-content-server/issues/5196 Issue 5196: Sign-up/in flow: Don't ask to connect a mobile device when you started flow on mobile]
*[https://github.com/mozilla/fxa-content-server/issues/5197 Issue 5197: Sign-in flow: Never send users to about:preferences after Sign-in] requires a browser patch.
*[https://github.com/mozilla/fxa-content-server/issues/5009 Issue 5009: Only show "email verified" on /connect_another_device if the user actually verfied their email.] confirmation]
|
|
*Following desired burn-down pace.
*Done!
*Anything that needs browser work? Yes #5197 requires a browser patch. Never send users to about: preferences after Sign-in
*Completed 13 out of 22, or 59%.
*Prioritize sign in/sign up flow for train-95 - know what we shouldn’t do but not necessarily what we should be doing. Trying to narrow down a solution that will work for everyone.  #5196 #5009 #5197
 
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-131:%20Q3%20Quality Quality P1s Burndown]
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-131:%20Q3%20Quality Quality P1s Burndown]
|-
|-
Line 69: Line 58:
|align="center"|50%
|align="center"|50%
|align="center"|40%
|align="center"|40%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0.0
|
|
*Supports the onboarding team, marketing retention team and Mozilla.org team to improve user retention (through converting first run users to multi-device users).
*Supports the onboarding team, marketing retention team and Mozilla.org team to improve user retention (through converting first run users to multi-device users).
*Ask for email first: Instead of a user needing to know to "Sign In" or "Sign Up", they visit the "Connect to Firefox Accounts" (or similar) page, where they simply enter their email. We determine if an account exists and forward the user to the appropriate sign in or sign up screen.
*[https://sql.telemetry.mozilla.org/dashboard/fxa-q3-okr-1-3 Dashboards related to user retention]
*[https://sql.telemetry.mozilla.org/dashboard/fxa-q3-okr-1-3 Dashboards related to user retention]
*[https://github.com/mozilla/fxa-content-server/pull/5332 Issue 5332: ABC test for CAD phase 3 (deep link)]  
*[https://github.com/mozilla/fxa-content-server/pull/5332 Issue 5332: ABC test for CAD phase 3 (deep link)]  
*[https://github.com/mozilla/fxa-content-server/issues/5194 Issue 5194: Sign-up/in flow: Ask for email first]
|
|
*Now that we’re using strict multi-device, numbers are going down, leading us to have less confidence.
*Didn't achieve this. Had a lot of questions about the metrics.
*Uptick in engage but no change in ?
*Graph not moving in the right direction
*Try to show CAD screen more often
|-
|-
|align="left"| 1.4 Fix newsletter subscription when email is confirmed on a 2nd device.
|align="left"| 1.4 Fix newsletter subscription when email is confirmed on a 2nd device.
Line 89: Line 73:
|align="center"|95%
|align="center"|95%
|align="center"|95%
|align="center"|95%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|1.0
|
|
Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.
Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.
|
|
*Didn’t seem to be any big change in their metrics. Marketing is expected to look at them again.
*Done!
*Number didn’t drop to 0 so we didn’t break it.
*Trending down even before our work landed. So not sure if the fix is working. Maybe the situation it fixed was small numbers and just not noticeable. We should be able to go through the flow ourselves and have Ben verify.
|-
|-
|rowspan=2| 2. To have improved the quality of the Firefox Account experience on mobile.
|rowspan=2| 2. To have improved the quality of the Firefox Account experience on mobile.
   
   
| align="left"|2.1 Measure and A/B test best push opt-in screen on iOS.  
| align="left"|2.1 Measure and A/B test best push opt-in screen on iOS.  
|align="center"|50%
|align="center"|50%
Line 107: Line 88:
|align="center"|55%
|align="center"|55%
|align="center"|55%
|align="center"|55%
|align="center"|
|align="center"|50%
|align="center"|
|align="center"|50%
|align="center"|
|align="center"|0.7
|align="left"|
|align="left"|
*[https://docs.google.com/spreadsheets/d/1tPE4KSNd_RU-ahSAK7a5hMcP5EAKGMlHJHegna87xLg/edit#gid=0 FxA Sync prioritization backlog for iOS]
*[https://docs.google.com/spreadsheets/d/1tPE4KSNd_RU-ahSAK7a5hMcP5EAKGMlHJHegna87xLg/edit#gid=0 FxA Sync prioritization backlog for iOS]
*[https://mozilla.invisionapp.com/share/76CPM7AC2#/screens/240561291_Desktop-_Connect_Another_Device Proposed mock up]
*[https://mozilla.invisionapp.com/share/76CPM7AC2#/screens/240561291_Desktop-_Connect_Another_Device Proposed mock up]
|
|
*No change. Waiting for a bunch of stuff to land. But happy with code and getting it through the review process with the iOS team.
*PR reviewed but not merged.
*Review brought up some interesting points to address.
|-
|-
| align="left"|2.2 Complete 80% of Q3 mobile bugs.
| align="left"|2.2 Complete 80% of Q3 mobile bugs.
Line 121: Line 103:
|align="center"|60%
|align="center"|60%
|align="center"|60%
|align="center"|60%
|align="center"|
|align="center"|65%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.8
|align="left"|  
|align="left"|  
Identification and fixing of the most impactful bugs on iOS
Identification and fixing of the most impactful bugs on iOS  
*[https://waffle.io/mozilla/fxa?milestone=FxA-132:%20Q3%20Mobile Mobile Bugs for Q3]
Currently working on:
*[https://github.com/mozilla/fxa-content-server/issues/5279  Issue 5279: Update Confirm view to handle when verification is using Push]
*[https://github.com/mozilla/fxa-bugzilla-mirror/issues/335 Issue 335: Display FxA avatar in settings [bz1382756]]
*[https://github.com/mozilla/fxa-bugzilla-mirror/issues/332 Issue 332: Use Firefox Account server's /.well-known/fxa-client-configuration to configure servers in Firefox for iOS [bz1237407]]
|
|
*Did the easy ones first. Medium level ones are left to do. Should have 9/13 done by next week.
*8 out of 13 are done - 80% of 80% done as of 9/25
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-132:%20Q3%20Mobile Mobile Quality Burndown Chart]
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-132:%20Q3%20Mobile Mobile Quality Burndown Chart]
|-
|-
Line 141: Line 118:
|align="center"|50%  
|align="center"|50%  
|align="center"|25%
|align="center"|25%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0.3
|align="left"|
|align="left"|
Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
*Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
*Feature doc and Issues coming soon.
|  
|
*Pushed forward the challenge with auto-focus.
*Not confident that we’ll have QR codes ready to test
*There are things we got done this quarter that we can roll forward.
*Too many unknowns to have any confidence
*Great analysis.
*We can try a file input but that would be a crappy experience. Have never tried to import a photo into the encoder. What if there’s too much noise in the photo or the QR code?
*We can play with the one built into iOS but still might not have time to figure it out by the end of the quarter.
*Confidence level for use of a QR code is <25%; type a code maybe >50%
*Need to regroup now that we know more about technical limitations and figure out what we want to do. Need a smaller discussion about what the next steps are.
*Ryan will reach out to iOS team.
|-
|-
| align="left"|3.2 Ship secondary email.
| align="left"|3.2 Ship secondary email.
Line 161: Line 133:
|align="center"|90%
|align="center"|90%
|align="center"|90%
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.9
|
|
*We are allowing users to add a secondary email. Allows users to change primary email.
*We are allowing users to add a secondary email. Allows users to change primary email.
*Issues for changing primary email:
**[https://github.com/mozilla/fxa-bugzilla-mirror/issues/278 Issue 278: Add ability for a FxA user to change their primary email]
**[https://github.com/mozilla/fxa-bugzilla-mirror/issues/341 Issue 341: Detect when account email has changed, and update local state to match]
**[https://github.com/mozilla/fxa-auth-server/issues/1999 Issue 1999: (node:48) DeprecationWarning: DeprecationWarning for mem.emailRecord: Use mem.accountRecord]
**[https://github.com/mozilla/fxa-bugzilla-mirror/issues/341 Issues 341: Detect when account email has changed, and update local state to match]
|
|
*Shipped
*Caching bug fixed
*Shaking out the downstream bugs in terms of getting the devices to update themsleves. That’s browser work that we don’t have a lot of control over.
*Client code is in
*Grisha is working on Android piece.
*Works for AMO, and they have an issue to fix one little thing
*Still confident we an get it out the door.
*Pocket was the only remaining thing.
|-
|-
| align="left"|3.3 Land support for relier encryption keys.  
| align="left"|3.3 Land support for relier encryption keys.  
Line 182: Line 149:
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|80%
|align="center"|
|align="center"|0.9
|align="left"|
|align="left"|
Relier encryption keys  
Relier encryption keys  
|
|
*Waiting to engage crypto consultants.
*Have done a lot of good work on it but it’s not totally closed out.
*Still working on the reviews.
*Currently under scrutiny by two crypto specialists.
*Can’t ship until we clear the security reviews.
*[https://github.com/mozilla/fxa/issues/245 Issue 245: Prototype key support api for Firefox Notes]
*[https://github.com/mozilla/fxa/issues/245 Issue 245: Prototype key support api for Firefox Notes]
|-
|-
Line 196: Line 166:
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.9
|align="left"|
|align="left"|
Implementation of Amplitude
Implementation of Amplitude
|
|
*Getting help from Amplitude to move forward
*Should be collecting real-time data this week.
*Still discovering new things
*We’ve haven’t completely tied everything off, but it’s basically done.
*Should have environment with train-95 that we can send events to.
*Working in stage and currently fine tuning.
|-
|-
|}
|}

Latest revision as of 01:53, 30 September 2017

September 25, 2017

Q3 OKR Progress

Objective Key Result Confidence Features Notes
Jul 10 Jul 24 Aug 8 Aug 21 Sep 5 Sep 18 Score
1. To have improved the quality for the Firefox Account experience on desktop. 1.1 Close ALL (100%) issues in the Quantum/Photon milestone. (Crouton) 60% 65% 70% 75% 90% 100% 1.0

Priorities for support of Quantum/Photon in 57:

1.2 Close 50% of our Q3 quality backlog. 50% 75% 75% 90% 100% 1.0

Quality Priorities for Q3:

1.3 Increase by 15% the rate of users that become multi-device as a result of going through our first run flow. 50% 50% 50% 40% 0% 0% 0.0
  • Didn't achieve this. Had a lot of questions about the metrics.
1.4 Fix newsletter subscription when email is confirmed on a 2nd device. 95% 95% 95% 95% 90% 90% 1.0

Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.

  • Done!
2. To have improved the quality of the Firefox Account experience on mobile. 2.1 Measure and A/B test best push opt-in screen on iOS. 50% 40% 55% 55% 50% 50% 0.7
  • PR reviewed but not merged.
  • Review brought up some interesting points to address.
2.2 Complete 80% of Q3 mobile bugs. N/A 60% 60% 60% 65% 90% 0.8

Identification and fixing of the most impactful bugs on iOS

3. To have completed all features that are still in flight from Q1 and Q2. 3.1 Have a device pairing flow ready to A/B test. 75% 75% 50% 25% 0% 0% 0.3
  • Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
  • Pushed forward the challenge with auto-focus.
  • There are things we got done this quarter that we can roll forward.
  • Great analysis.
3.2 Ship secondary email. 80% 80% 90% 90% 90% 90% 0.9
  • We are allowing users to add a secondary email. Allows users to change primary email.
  • Caching bug fixed
  • Client code is in
  • Works for AMO, and they have an issue to fix one little thing
  • Pocket was the only remaining thing.
3.3 Land support for relier encryption keys. 70% 70% 75% 75% 75% 80% 0.9

Relier encryption keys

3.4 Integrate 100% of high priority Amplitude events. 75% 75% 75% 75% 75% 90% 0.9

Implementation of Amplitude

  • Should be collecting real-time data this week.
  • We’ve haven’t completely tied everything off, but it’s basically done.
  • Working in stage and currently fine tuning.