Firefox/Accounts/Progress Report/2017Q3

August 8, 2017

Q3 OKR Progress

Objective	Key Result	Confidence	Features	Notes
Jul 10	Jul 24	Aug 8	Aug 21	Sep 4	Sep 18	Score
1. To have improved the quality for the Firefox Account experience on desktop.	1.1 Close ALL (100%) issues in the Quantum/Photon milestone. (Crouton)	60%	65%	70%	75%				Priorities for support of Quantum/Photon in 57: Crouton Flow Issues for 57 release. Carried over from the last sprint: Issue 1285: Device names with non-BMP characters are rejected. Issue 229: Update device name column to utf8mb4	Fixed additional bugs in the past sprint. 25% remaining to fix, down from 37%. Animation on svg’s needs to be started (Issue 5168: Discuss how we should animate new illustrations). Once we’re done with the Issue 1285: Device names with non-BMP characters are rejected, which is priority #5 on the list, that will free up some time. Rest seem pretty manageable but just need to get them to the top of the list. Crouton Flow Burndown Chart
	1.2 Close 50% of our Q3 quality backlog.		50%	75%	75%				Quality Priorities for Q3: Quality related P1s that we’ve identified to fix in Q3. Active in this sprint: Issue 5196: Sign-up/in flow: Don't ask to connect a mobile device when you started flow on mobile Issue 5197: Sign-in flow: Never send users to about:preferences after Sign-in requires a browser patch. Issue 5009: Only show "email verified" on /connect_another_device if the user actually verfied their email. confirmation]	Continue to burn down issues along the projected ideal line. Prioritizing sign in/sign up flow for train-95. Know what we shouldn’t do but not necessarily what we should be doing. Trying to narrow down a solution that will work for everyone. Quality P1s Burndown shows we’re on the ideal line
	1.3 Increase by 15% the rate of users that become multi-device as a result of going through our first run flow.	50%	50%	50%	40%				Supports the onboarding team, marketing retention team and Mozilla.org team to improve user retention (through converting first run users to multi-device users). Ask for email first: Instead of a user needing to know to "Sign In" or "Sign Up", they visit the "Connect to Firefox Accounts" (or similar) page, where they simply enter their email. We determine if an account exists and forward the user to the appropriate sign in or sign up screen. Dashboards related to user retention Issue 5332: ABC test for CAD phase 3 (deep link) Issue 5194: Sign-up/in flow: Ask for email first	FxA team's A/B/C testing continues. Now that we’re measuring "strict multi-device", numbers are going down. Confidence level for this key result has dropped because of questions around why this is happening. Maybe try to show CAD screen more often? A/B/C testing is set up as follows: Mobile Push (Phase 1) for ⅓ of the users Mobile Push (Phase 2) for ⅓ of the users Mobile Push (Phase 3) for ⅓ of the users Marketing is evaluating the effectiveness of copy variations for the sign in/sign up flows. Their hypothesis: If we provide users with clear and concise copy about the features and benefits they’ll find in using a Fx account they’ll be more likely sign up, increasing conversions in the page and contribute to improved overall DAU/MAU. (Secondarily, users with Fx Accounts that are not logged in will find the new copy compelling enough to login.) Much of the Email First work will ship week of Aug 28 as part of Train-94. Before this feature is offered to users, rigorous internal testing must be conducted because of the scope of the change, and the marketing copy experiment will need to have completed.
	1.4 Fix newsletter subscription when email is confirmed on a 2nd device.	95%	95%	95%	95%				Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.	Didn’t seem to be any big change in marketing's metrics after fixing the problem. Marketing is expected to look at the metrics again. Number didn’t drop to 0, so we didn’t break it. It was trending down even before our work landed. So not sure if the fix is working. Maybe the situation it fixed was small numbers and just not noticeable. We should be able to go through the flow ourselves and have Ben verify.
2. To have improved the quality of the Firefox Account experience on mobile.	2.1 Measure and A/B test best push opt-in screen on iOS.	50%	40%	55%	55%				FxA Sync prioritization backlog for iOS Proposed mock up	No change. Waiting for a bunch of stuff to land. But happy with code and getting it through the review process with the iOS team.
	2.2 Complete 80% of Q3 mobile bugs.	N/A	60%	60%	60%				Identification and fixing of the most impactful bugs on iOS Mobile Bugs for Q3 Currently working on: Issue 5279: Update Confirm view to handle when verification is using Push Issue 335: Display FxA avatar in settings [bz1382756] Issue 332: Use Firefox Account server's /.well-known/fxa-client-configuration to configure servers in Firefox for iOS [bz1237407]	Top priority for train-94 is getting in all the iOS prefs stuff. No PRs have been merged yet. FxA iOS verification links open at almost the same rate as iOS for sign-in, ~45%. This is a significant increase over 8% previously. Issue 337: Open FxA verification links in Fx for iOS [bz1382776] Mobile Quality Burndown Chart
3. To have completed all features that are still in flight from Q1 and Q2.	3.1 Have a device pairing flow ready to A/B test.	75%	75%	50%	25%				Allow users to test out device pairing. This does not necessarily mean that QR codes are used. Feature doc and Issues coming soon.	Not confident that we’ll have QR codes ready to test Too many unknowns to have any confidence We can try a file input but that would be a crappy experience. Have never tried to import a photo into the encoder. What if there’s too much noise in the photo or the QR code? We can play with the one built into iOS but still might not have time to figure it out by the end of the quarter. Confidence level for use of a QR code is <25%; type a code maybe >50% Need to regroup now that we know more about technical limitations and figure out what we want to do. Need a smaller discussion about what the next steps are. Ryan will reach out to iOS team.
	3.2 Ship secondary email.	80%	80%	90%	90%				We are allowing users to add a secondary email. Allows users to change primary email. Issues for changing primary email: Issue 278: Add ability for a FxA user to change their primary email Issue 341: Detect when account email has changed, and update local state to match Issue 1999: (node:48) DeprecationWarning: DeprecationWarning for mem.emailRecord: Use mem.accountRecord Issues 341: Detect when account email has changed, and update local state to match	Shipped Shaking out the downstream bugs in terms of getting the devices to update themsleves. That’s browser work that we don’t have a lot of control over. Grisha is working on Android piece. Still confident we an get it out the door.
	3.3 Land support for relier encryption keys.	70%	70%	75%	75%				Relier encryption keys	Waiting to engage crypto consultants. Issue 245: Prototype key support api for Firefox Notes
	3.4 Integrate 100% of high priority Amplitude events.	75%	75%	75%	75%				Implementation of Amplitude	Getting help from Amplitude to move forward Still discovering new things Should have environment with train-95 that we can send events to.