Mobile a/b testing and analytics status reports/August 21 2015

From MozillaWiki
Jump to: navigation, search

previous weeks report

Mobile A/B testing and analytics status report, August 21, 2015

Overall Project Health

This week: yellow - project is still in definition phase with active investigations. Once we have results, next steps and milestones/roadmap will be created.

Exec Summary

  • Scope: Enable stakeholders (mobile product marketing, mobile product management and mobile growth) access to actionable analytics data and to create a/b tests on our mobile products to allow insights and actions, as well as measure the impact of new features or modification on retention.
    • Statement of truth: a/b testing requires analytics in order to identify opportunities and measure results therefore are tightly coupled.
    • Initial proposals identified 3rd party tech solutions (Google Analytics and mixpanel for analytics, optimizely for a/b testing).
    • Revised plan is to understand the current data collection systems data, capabilities and limitations with an eye towards the feasibility of providing stakeholders access in the form of analaytics
    • current phase is investigation, the next phase will be defining the next step based on results. Current options include but are not limited to:
      • use existing stream
      • improve streams (eg, unified telemetry?)
      • leverage 3rd party solutions

Risks/Issues

Description of Risks/Issues State Owner Plan to Resolve/Mitigation Target Date
Understand what v3 captures (fhr) Open Alex Investigate 8/28
Understand what v3 captures (telemetry) Open Alex Investigate 8/28
Understand what v3 captures (Adjust) Open Alex Investigate 8/28
understand sample and limitations of data collection per channel/train Open Alex Investigate 8/28
Understand the deltas are between whats we gather today and what stakeholders need Open Alex Investigate 8/28
Understand the deltas on how stakeholders want to consume it (data exploration tools) Open Alex Investigate 8/28
Create user stories - engineering Open Engineering create items 8/28
Create user stories - product closed product create items 8/14
Create user stories - growth/marketing closed growth/marketing create items 8/14

Accomplished for Last Period

  • Assembled team
  • Acknowledgement of needs and problem statement
  • kick off meeting
  • user stories
  • initial legal review of 3rd party tools
  • Informal user story sign off as a means of vetting solutions
  • Agreed to review existing data collections systems

Planned for Next Period

  • Growth team:
    • Identify most important reports
    • See how we can streamline these reports with data team
    • Understand all the data available
  • Engineering
    • Engineering user stories
    • data stewards to review switch board
    • investigate feasability of adding a/b test versions to FHR for POC
  • POC effort/Reality check (Growth and Product)
    • Adavis: mentor/help product to prepare first a/b test
    • Prepare first onboarding a/b test with switchboard and telemetry
    • using nightly
    • Look into churn rate by experiment

Outstanding requests not yet road mapped into a release

Description State Owner Plan to Resolve/Mitigation Target Date
item item item item item
item item item item item

Important Links/References

Retrieved from "https://wiki.mozilla.org/index.php?title=Mobile_a/b_testing_and_analytics_status_reports/August_21_2015&oldid=1091309"