P2PULearningChallenges/Metrics: Difference between revisions
Jump to navigation
Jump to search
(Created page with "<h2>Final Metrics for Challenges on P2PU</h2> <h3>Evaluation Goals and Metrics for Challenges </h3> <pre class="_fck_mw_lspace"> The objective of these metrics is to assess w...") |
No edit summary |
||
| Line 1: | Line 1: | ||
<h2>Final Metrics for Challenges on P2PU</h2> | <h2>Final Metrics for Challenges on P2PU</h2> | ||
<h3>Evaluation Goals and Metrics for Challenges | <h3>Evaluation Goals and Metrics for Challenges</h3> | ||
</h3> | <ul><li>The objective of these metrics is to assess whether or not Challenges on P2PU are a good way to begin creating educational content for Mozilla projects. Both quantitative and qualitative methods are used to determine value for Mozilla, P2PU, and the user. </li | ||
< | ><li>The focus is on overall value - We have content and we are moving it into this challenge framework. How is that going? How many people are completing our content (within this framework). Conversion funnel. Completion rates. <br/><br | ||
</ | /></li | ||
< | ><li><b>Conversion rate</b><ul><li><b>Users who hit the site</b><ul><li>N = not logged-in users (data from google analytics) accessing:<ul><li>School of Webcraft home page</li | ||
</ | ><li>Challenge set home pages</li | ||
< | ><li>Challenge home pages</li | ||
</ | ><li>Challenge Task pages</li></ul | ||
< | ></li></ul | ||
</ | ></li | ||
< | ><li><b>Share of users who register but never start a challenge</b> <ul><li>Registered users only --> Visit a challenge related page but no challenge is started</li></ul | ||
< | ></li | ||
< | ><li><b>Share of users who start a challenge</b><ul><li><b>Share of users who complete a challenge</b><ul><li>Only users who started a challenge at least 3 days prior to reporting</li></ul | ||
</ | ></li | ||
< | ><li><b>Share who abandon a challenge</b><ul><li>Users who are inactive for 10 days</li | ||
</ | ><li>Users who manually "leave" a challenge</li></ul | ||
< | ></li></ul | ||
</ | ></li | ||
< | ><li><b>Share of users who complete more than one challenge (2, 3, 4, ...)</b></li | ||
</ | ><li><b>Share of users who complete a series (=set) of challenges</b></li | ||
< | ><li><b>Number of Challenges completed per learner</b><ul><li>N = Users who completed at least one challenge</li></ul | ||
</ | ></li></ul | ||
< | ></li | ||
</ | ><li><b>Badges</b><ul><li><b>Number of Badges issued over time (organized by badge type)</b></li | ||
< | ><li><b>Number of badges issued per learner (badge categories: making, understanding)</b></li></ul | ||
</ | ></li | ||
< | ><li><b>Mentors</b><ul><li><b>Number of Mentors over time (some measure of level of activity)</b><ul><li>Todo -> Check with John</li></ul | ||
</ | ></li></ul | ||
< | ></li | ||
</ | ><li><b>Qualitative Survey to ask people how they find the experience</b><ul><li>Laura to prepare<br/><br | ||
< | /></li></ul | ||
</ | ></li></ul | ||
< | ><h2>Hackasaurus Metrics</h2> | ||
</ | <h3>Understanding</h3> | ||
< | <ul><li>Number of Users who hit the site</li | ||
</ | ><li>Number of Badges issued over time</li | ||
< | ><li>Number of people signing up for our email/group lists/pledges/campaigns/twitter/bookmarks</li | ||
</ | ><li>Number of Hacktivity Kit downloads<br/><br | ||
< | /></li></ul | ||
</ | ><h3>Making</h3> | ||
< | <ul><li>Number of websites developed</li | ||
</ | ><li>People using our templates </li | ||
< | ><li>Number of hacked webpages</li | ||
</ | ><li>Number of Hackasaurus events </li | ||
< | ><li>Number of event participants</li | ||
</ | ><li>Goggle activations (bookmarklet uses) per day<br/><br | ||
< | /></li></ul | ||
</ | ><h3>Innovating</h3><br | ||
< | /><ul><li>patches (or forks) from volunteer contributors (dev and curriculum)</li | ||
</ | ><li>x # of hackasaurus "experiments" by community members</li | ||
< | ><li>localizations (Hacktivity Kit + web+ goggles)</li></ul | ||
</ | >*****************************************************************<br | ||
< | /><br | ||
</ | /><b>Project Components: </b><br | ||
< | /><br | ||
</ | /><b>Website:</b><br | ||
< | /><ul><li>Number of people visiting</li | ||
</ | ><li>XXX page views per day</li | ||
< | ><li>Number of people signing up for our email/group lists/pledges/campaigns/twitter/bookmarks</li | ||
</ | ><li>page views per visit</li | ||
><li>return visitors</li | |||
<h2>Hackasaurus Metrics</h2> | ><li>links clicked</li | ||
<h3>Understanding | ><li># of backlinks<br/><br | ||
</h3> | /></li></ul | ||
< | ><b>Goggles:</b><br | ||
</ | /><ul><li>XXX activations (bookmarklet uses) per day</li | ||
< | ><li>XXX patches (or forks) from volunteer contributors<br/><br | ||
</ | /></li></ul | ||
< | ><b>Hacktivity Kit:</b><br | ||
</ | /><ul><li>XXX # of downloads by the end of 2012/ Q1 - I'd collect this monthly (or weekly)</li | ||
< | ><li>x# of Hacktivity Kit localizations <ul><li>or 2x the number of localizations from 2011</li></ul | ||
</ | ></li | ||
<h3>Making | ><li>x # of feedback questionnaires completed<br/><br | ||
</h3> | /></li></ul | ||
< | ><b>P2PU Challenges</b><i>(this could also be for other things like parable or nav badge if not integrated into p2pu - "interactive challenges")</i><br | ||
</ | /><ul><li>x # of webpages created as a result of challenges</li | ||
< | ><li>Share of users who start a (Hackasaurus) challenge<ul><li>Share who abandon a challenge</li | ||
</ | ><li>Share of users who complete a challenge</li></ul | ||
< | ></li | ||
</ | ><li>Share of users who complete more than one challenge (2, 3, 4, ...)</li | ||
< | ><li>Share of users who complete a series of challenges</li | ||
</ | ><li>Number of Challenges completed per learner</li | ||
< | ><li>Number of Badges issued over time (organized by badge type)</li | ||
</ | ><li>Number of badges issued per learner (making and understanding badges)</li | ||
< | ><li>Number of Mentors over time (some measure of level of activity) <br/><br | ||
</ | /></li></ul | ||
<h3>Innovating | >Community<br | ||
</h3> | /><ul><li>x # of countries running hack jams</li | ||
< | ><li>x # of hackasaurus "experiments" by community members</li | ||
</ | ><li>x # of localizations</li | ||
< | ><li>x # of participants in Hackasaurus events<br/><br | ||
</ | /></li></ul | ||
< | ><h2>BRAINSTORMING METRICS:</h2> | ||
</ | <h3>Overall goals</h3> | ||
< | ><b>Evaluation Goals and Metrics for Challenges</b><br | ||
Website: | /><ul><li>The objective of these metrics is to assess whether or not Challenges on P2PU are a good way to begin creating educational content for Mozilla projects. Both quantitative and qualitative methods are used to determine value for Mozilla, P2PU, and the user. </li | ||
</ | ><li>Two sets of metrics:<ul><li>NOT FOCUS -> Internal - Once within a challenge - what are the metrics that tell us how well someone is doing, and how to make the challenge features / UX better. Click patterns. Drop off points. Etc. </li | ||
< | ><li>FOCUS -> Overall value - We have content and we are moving it into this challenge framework. How is that going? How many people are completing our content (within this framework). Conversion funnel. Completion rates. <br/><br | ||
</ | /></li></ul | ||
< | ></li></ul | ||
</ | ><h3>Framing:</h3> | ||
< | <ul><li>Focus metrics on state change - attitude, awareness, actions taken<br/><br | ||
</ | /></li></ul | ||
< | ><b>3 Levels of Impact for Metrics for MoFo (framing for our larger goals - see if we can re-frame some of the metrics below into Participation and Making/Learning buckets for consistency)</b><br | ||
</ | /><ul><li>Understanding - what increase in understanding/awareness of the open web and our programs (this is not as relevant for P2PU challenges specifically)</li></ul | ||
< | >CV: challenges provide a model for deep understanding, which is measured primarily in the ability of learners to reflect their knowledge. A tangible example of that is to create a tutorial for someone else on a topic.<br | ||
</ | /><ul><li>Participation - how many people are participating, in what ways, how deeply</li></ul | ||
< | >CV: an easy way to measure that is sequencing of comments: we do not care necessarily about how many comments have been made, what we care about as an indicator of participation is how many of these comments are an answer to a previous comment. Additionally things like sharing with external social graphs provd=ide an indicator > more on how to assess social assessment & participation here <a href="http://bit.ly/vloJcH">http://bit.ly/vloJcH</a><br | ||
</ | /><ul><li>Making - how are people improving their skills</li></ul | ||
< | >CV: not clear on this, how is this different from understanding?<br | ||
</ | /><ul><li>Innovation - how are people innovating (not as relevant?)</li></ul | ||
< | >CV: brilliant point- something to consider is measuring how many times for example a unique solution, such as a piece of code, is being re-used in other ways.<br | ||
</ | /><br | ||
< | />Can we currently measure all these on P2PU? What do the metrics look like for Hackasaurus (what do we have available to us there?)<br | ||
</ | />CV: no we cannot probably. but we can start considering ways we can measure participation (stealth assessment) as well as understanding (manual assessment)<br | ||
< | /><br | ||
</ | />PS: Main comment -> Need to reduce number of metrics / let's track general traffic and then pick a few important ones (max 3-5) to drill down<br | ||
< | /><br | ||
</ | /><br | ||
< | /><b>Goals</b><br | ||
</ | /><br | ||
< | /><b><i><u>Participation (beyond Reach)<s>Determine success of functional improvement</s></u></i></b><br | ||
< | /><br | ||
< | />How many people is Hackasaurus currently engaging? How many more might Challenges bring to the Hackasaurus Project? historic, current, and predictive views using quantitative data <br | ||
</ | /><br | ||
< | /><i>Method</i>: data analysis (Monitor for trends) – historical and current, weekly collection<br | ||
</ | /><br | ||
< | /><i>Metrics (increasing depth of participation):</i><br | ||
</ | /><ul><li>Access<ul><li>Basic demographics (from Google Analytics)</li | ||
< | ><li>Referrer stats (from Google Analytics)</li | ||
</ | ><li>Session Duration</li | ||
< | ><li>Number of total page views by language and country</li></ul | ||
< | ></li | ||
< | ><li><b>Conversion rate</b><ul><li><b>Users who hit the site</b></li | ||
</ | ><li><b>Share of users who register but never start a challenge</b></li | ||
< | ><li><b>Share of users who start a challenge</b><ul><li><b>Share who abandon a challenge</b></li | ||
</ | ><li><b>Share of users who complete a challenge</b></li></ul | ||
< | ></li | ||
</ | ><li><b>Share of users who complete more than one challenge (2, 3, 4, ...)</b></li | ||
< | ><li><b>Share of users who complete a series of challenges</b></li | ||
</ | ><li><b>Number of Challenges completed per learner</b></li | ||
< | ><li>Nice to have: Usage pattern over time (levels of activity / completion speed)<ul><li>Maybe hours / week over time</li></ul | ||
</ | ></li></ul | ||
< | ></li | ||
</ | ><li>Learning<ul><li>Number of Active Learners over time (ie those who complete a task once a week) </li | ||
< | ><li>Number of Challenge/Curriculum Completions over time (ie Learners who have completed all the Challenge/Curriculum tasks)</li | ||
</ | ><li>Share of users who complete at least one challenge</li | ||
< | ><li>Ratio of users who started a challenge and completed it</li | ||
</ | ><li>Click Stream</li></ul | ||
></li | |||
< | ><li><b>Badges</b><ul><li><b>Number of Badges issued over time (organized by badge type)</b></li | ||
< | ><li><b>Number of badges issued per learner (making and understanding badges)</b><ul><li>what about collaborating = learning behavior?</li></ul | ||
</ | ></li></ul | ||
< | ></li | ||
</ | ><li>Mentoring<ul><li><b>Number of Mentors over time (some measure of level of activity) </b><ul><li>This can probably only be done through a survey, because we let mentors structure their communication however they want - a lot of their interaction happens outside of p2pu.org</li></ul | ||
< | ></li | ||
</ | ><li>Conversation rates from participant to helper to mentor (deeper participation)</li | ||
< | ><li>sequence of comments > that is something we could measure</li | ||
</ | ><li>Number of Gurus (e.g. users who have ability to issue valuable badges) <ul><li>(does this exist?)- Yes, during the first pilot of Badges on P2PU, they seeded the community with gurus (ie "Has-A-Badge") assessors. @Chloe @Philipp - Do you know how many "gurus" are active on the site?</li></ul | ||
< | ></li></ul | ||
<h2>BRAINSTORMING METRICS</h2> | ></li | ||
><li>Peer Assessment (as a form of participation)<ul><li>Share of total users who participated in peer assessment (e.g. 20%)</li | |||
< | ><li>Share of peer to peer badges compared to overall badges (e.g. 15%)</li | ||
><li>Number of badges that lack sufficient reviews<br/><br | |||
/></li></ul | |||
></li></ul | |||
><b>Qualitative Survey to ask people how they find the experience</b><br | |||
/><br | |||
</ | /><br | ||
< | /><b><i><u>Making</u></i></b><br | ||
</ | /><br | ||
< | /><ul><li>Number of Badges (that are tied to making) issued over time</li | ||
</ | ><li>Number of Links to participants work (gathering external links will allow us to see what/if people are making)</li | ||
< | ><li>Handle through survey questions<br/><br | ||
< | /><br/><br | ||
< | /></li></ul | ||
</ | ><b><i><u>Innovating</u></i></b><br | ||
< | /><br | ||
</ | /><ul><li>Handle through survey questions<br/><br | ||
< | /></li></ul | ||
</ | >PS: I combined the following with Participation above<br | ||
< | /><br | ||
</ | /><br | ||
< | /><b><i><u><s>Community</s></u></i></b><br | ||
</ | /><br | ||
< | /><i><s>Metrics</s></i><br | ||
</ | /><ul><li><br/><br/><br | ||
/></li></ul | |||
< | ><b><i><u><s>3: Up Peer Assessment / e.g. </s></u></i></b><s>How well does the peer assessment work?</s><br | ||
< | /><i><s>Method</s></i><s>: data analysis (Monitor for trends) – historical and current, 1 week after launch, 4 weeks after launch, 3 months after launch</s><br | ||
</ | /><br | ||
/><i><s>Metrics</s></i><br | |||
</ | /><ul><li><s>Share of total users who participated in peer assessment (e.g. 20%)</s></li | ||
< | ><li><s>Share of peer to peer badges compared to overall badges (e.g. 15%)</s></li | ||
</ | ><li><s>Number of badges that lack sufficient reviews</s><br/><br | ||
/><br/><br | |||
< | /></li></ul | ||
< | >PS: Doesn't fit with Mozilla metrics framework:<br | ||
</ | /><br | ||
/><br | |||
/><b><i><u>4: User Survey - Determine perceived value to users</u></i></b><br | |||
CV: | /><i>Method</i>: survey 3 months after launch<br | ||
PS: | /><br | ||
Goals | /><i>Sample strata</i><br | ||
Participation (beyond Reach)Determine success of functional improvement | /><ul><li>Learners who abandoned a challenge (target problem areas)</li | ||
><li>Learners who never started a challenge</li | |||
Method: data analysis (Monitor for trends) – historical and current, weekly collection | ><li>Learners who completed a challenge<br/><br | ||
/></li></ul | |||
</ | ><i>Indicators / Variables</i><br | ||
< | /><ul><li>[future version] + contribution to getting a job / only for relevant challenges</li | ||
< | ><li>Contribution to personal satisfaction</li | ||
< | ><li>Contribution to social recognition</li | ||
</ | ><li>Focus on state changes - changes in attitude, awareness, actions taken</li | ||
< | ><li>Did you tell your friends about this?</li | ||
</ | ><li>Would you recommend it to a friend?</li | ||
< | ><li>.... need to do more work here<br/><br | ||
</ | /></li></ul | ||
< | ><b><i><u>5: Ongoing gathering of empirical information to improve individual challenges</u></i></b><br | ||
</ | /><i>Method</i>: Each learner is asked a single question after completing a challenge <br | ||
< | /><br | ||
</ | /><i>Metrics</i><br | ||
< | /><ul><li>+ Learner Satisfaction</li | ||
</ | ><li>+ Challenge structure success (target problem areas)<br/><br | ||
< | /></li></ul | ||
</ | >Possible Wording (need to wordsmith) - <br | ||
< | /><ul><li>Help us make this challenge better! <ul><li>Did you think this challenge was a little silly – just right - too boring</li | ||
</ | ><li>Did you think this challenge was too hard – just right – too easy / simple<br/><br | ||
< | /><br/><br | ||
</ | /></li></ul | ||
< | ></li></ul | ||
</ | ><b>OTHER NOTES</b><br | ||
< | /><b><i><u>ROI (Long Term)</u></i></b><br | ||
</ | />How have donations increased or decreased? How has participation spread?<br | ||
< | /><br | ||
</ | /><i>Method</i>: data analysis (Quantitative from Mozilla and P2PU user data, donation data) – Ongoing after launch, long term.<br | ||
< | /><br | ||
</ | /><i>Metrics</i><br | ||
< | /><ul><li>+ donation stream to p2pu and Mozilla</li | ||
< | ><li>+ number of new signups</li | ||
< | ><li>- Staff Hours</li | ||
</ | ><li>- Average Response Rate</li | ||
< | ><li>+ Total Likes, One ups or RT on social media messaging marked #challenges <br/><br | ||
< | /></li></ul | ||
< | ><i>Notes</i><br | ||
</ | /><ul><li>Suggestion is to postpone this and add to later phase (agree)</li | ||
< | ><li>Risk that adding option to donate into the challenge will change the user's learning experience (we really want to measure if the learning works at this point - adding donations may influence the outcomes)<ul><li>- Average Challenge Completion Time (currently located within the Challenge metrics)</li | ||
</ | ><li>- Average Task Completion Times</li | ||
< | ><li>+ Number of new challenges by the community</li></ul | ||
</ | ></li></ul | ||
< | > <br | ||
</ | /><b>Quantitative</b><br | ||
< | /><br | ||
</ | />Collecting these metrics will allow us to define meta metrics (ie number of learners vs non learners vs power learners or whatever)<br | ||
< | /><br | ||
</ | /><b><u>Across the Board:</u></b><br | ||
< | /><i>Metrics for Understanding and Awareness of Open Web and Mozilla Projects</i><br | ||
</ | /><ul><li>Basic demographics</li | ||
< | ><li>Number of total page views by language and country</li | ||
</ | ><li>Referrer stats</li | ||
< | ><li>Click Stream<br/><br | ||
</ | /></li></ul | ||
< | ><b>The Metrics Story:</b> Collecting the demographics of users will allow Mozilla to further focus their programs to specific target groups. Total number of page views by language and country furthers this definition. This will allow Mozilla to spend it's resources designing programs for people, programs that work. This will influence stats in participation depth, which will show progress towards the 10 Million Webmakers marker. Knowing which referrers are the most valuable will help streamline resources and eliminate erroneous spending on marketing and/or partnerships. Following the clickstream will help Mozilla understand what sorts of content inspire understanding, which will further Mozilla's ability to create content that pushes people to becoming webmakers.<br | ||
</ | /><br | ||
< | /><i>Participation Depth</i><br | ||
</ | /><ul><li>IP</li | ||
< | ><li>Session Duration & Clicks per Session</li | ||
</ | ><li>Think time</li | ||
< | ><li>Conversion rate<ul><li>Share of users who register but never <i>do</i> anything</li></ul | ||
</ | ></li></ul | ||
< | > <br | ||
</ | /><b>The Metrics Story: </b>Although IP logging is a raw metric, with thousands of users it will be valuable to see how deep into Mozilla programming users are going. By cross comparing IP logs between programs, Mozilla will have a better view of influence and partipation depth across the board. Collecting session durations and clicks per session and seeing an increase in these two metrics over time will further underline this viewpoint. Think time can be used to filter out users who simply browse. Decreasing negative conversion rates is important in showing strength of programming.<br | ||
< | /><br | ||
< | /><i>Skill Improvement</i><br | ||
< | /><ul><li>Number of Badges issued over time (organized by badge type)</li | ||
</ | ><li>Number of Links to participants work (gathering external links will allow us to see what/if people are making)<br/><br | ||
< | /></li></ul | ||
< | ><b>The Metrics Story:</b> The more badges that are issued and the more quality links that are submitted to Mozilla sites, the clearer the influence of Mozilla on skill improvement. <br | ||
< | /><br | ||
</ | /><b><u>Platform Specific</u></b><br | ||
< | /><br | ||
</ | /><b>P2PU</b><br | ||
< | /><i>Participation Depth</i><br | ||
</ | /><ul><li>Number of Mentors over time</li | ||
< | ><li>Conversation rates<ul><li>from participant to helper to mentor (deeper participation)</li | ||
><li>Number of Active Learners over time (ie those who complete a task once a week) </li | |||
</ | ><li>Share of users who complete at least one challenge</li | ||
< | ><li>Ratio of users who started a challenge and completed it</li></ul | ||
</ | ></li | ||
< | ><li>Share of total users who participated in peer assessment (e.g. 20%)</li | ||
</ | ><li>Share of peer to peer badges compared to overall badges (e.g. 15%)<br/><br | ||
< | /></li></ul | ||
</ | ><i>Skill Improvement</i><br | ||
< | /><ul><li>Number of Gurus (e.g. users who have ability to issue valuable badges) </li | ||
><li>Number of badges that lack sufficient reviews</li | |||
< | ><li>Number of Challenge/Curriculum Completions over time (ie Learners who have completed all the Challenge/Curriculum tasks)<br/><br | ||
</ | /></li></ul | ||
< | ><b><u>01.12.11 Metrics Meeting</u></b><br | ||
/><br | |||
/><b>Attendees</b><br | |||
</ | /><ul><li>Steph</li | ||
><li>Laura</li | |||
><li>Philipp</li | |||
><li>Chloe<br/><br | |||
< | /></li></ul | ||
</ | ><b>Dealing with Resources</b><br | ||
/><ul><li>Need to extend Arlton's contract to work on Challenges UX – involved 3.5 months ago. P2PU hired 2 people, 1 for Challenge content and 1 for UX. Arlton and Jamie Curle. <ul><li>Was getting the UX person the most efficient way to use the funds (Philipp says no). Discuss in longer call (TBD).</li></ul | |||
</ | ></li | ||
< | ><li>Need to figure out the longer term strategy for people</li | ||
</ | ><li>SoW – funded early badges work, SoW Community Management, Webmaking 101 Challenges. 50% p2pu funding (Zuzel, Chloe, John paid with p2pu funds) and 50% from SoW funding. </li | ||
< | ><li>By January, launching Hackasaurus on P2PU – get Challenges rocking the house.</li | ||
><li>Zuzel is superhuman -- really?<br/><br | |||
/></li></ul | |||
><b>Metrics Discussion</b><br | |||
</ | /><a href="https://etherpad.mozilla.org/challenges-evaluation-goals">https://etherpad.mozilla.org/challenges-evaluation-goals</a><br | ||
< | />Laura to set up metric meeting with Jess. Consistent set of metrics! Invite Steph <br | ||
< | />How do the p2pu metrics line up with the Hackasaurus metrics, which one are close enough aligned that we can then compare. Comparision chart.<br | ||
< | />Parse metrics – send out<br | ||
</ | /><br | ||
< | /><b>Agreements</b><br | ||
/>Challenge Fixes Board: Released Dec. 14<br | |||
< | />Implementation of Partner Account that allows Hackasaurus to run their challenges<br | ||
< | /><br | ||
< | /><b>Success looks like:</b><br | ||
/>The list of agreed upon must haves and launch by mid January | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
</ | |||
< | |||
</ | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
< | |||
</ | |||
< | |||
< | |||
</ | |||
< | |||
The list of agreed upon must haves and launch by mid January | |||
Latest revision as of 20:17, 28 December 2011
Final Metrics for Challenges on P2PU
Evaluation Goals and Metrics for Challenges
- The objective of these metrics is to assess whether or not Challenges on P2PU are a good way to begin creating educational content for Mozilla projects. Both quantitative and qualitative methods are used to determine value for Mozilla, P2PU, and the user.
- The focus is on overall value - We have content and we are moving it into this challenge framework. How is that going? How many people are completing our content (within this framework). Conversion funnel. Completion rates.
- Conversion rate
- Users who hit the site
- N = not logged-in users (data from google analytics) accessing:
- School of Webcraft home page
- Challenge set home pages
- Challenge home pages
- Challenge Task pages
- N = not logged-in users (data from google analytics) accessing:
- Share of users who register but never start a challenge
- Registered users only --> Visit a challenge related page but no challenge is started
- Share of users who start a challenge
- Share of users who complete a challenge
- Only users who started a challenge at least 3 days prior to reporting
- Share who abandon a challenge
- Users who are inactive for 10 days
- Users who manually "leave" a challenge
- Share of users who complete a challenge
- Share of users who complete more than one challenge (2, 3, 4, ...)
- Share of users who complete a series (=set) of challenges
- Number of Challenges completed per learner
- N = Users who completed at least one challenge
- Users who hit the site
- Badges
- Number of Badges issued over time (organized by badge type)
- Number of badges issued per learner (badge categories: making, understanding)
- Mentors
- Number of Mentors over time (some measure of level of activity)
- Todo -> Check with John
- Number of Mentors over time (some measure of level of activity)
- Qualitative Survey to ask people how they find the experience
- Laura to prepare
- Laura to prepare
Hackasaurus Metrics
Understanding
- Number of Users who hit the site
- Number of Badges issued over time
- Number of people signing up for our email/group lists/pledges/campaigns/twitter/bookmarks
- Number of Hacktivity Kit downloads
Making
- Number of websites developed
- People using our templates
- Number of hacked webpages
- Number of Hackasaurus events
- Number of event participants
- Goggle activations (bookmarklet uses) per day
Innovating
- patches (or forks) from volunteer contributors (dev and curriculum)
- x # of hackasaurus "experiments" by community members
- localizations (Hacktivity Kit + web+ goggles)
*****************************************************************
Project Components:
Website:
- Number of people visiting
- XXX page views per day
- Number of people signing up for our email/group lists/pledges/campaigns/twitter/bookmarks
- page views per visit
- return visitors
- links clicked
- # of backlinks
Goggles:
- XXX activations (bookmarklet uses) per day
- XXX patches (or forks) from volunteer contributors
Hacktivity Kit:
- XXX # of downloads by the end of 2012/ Q1 - I'd collect this monthly (or weekly)
- x# of Hacktivity Kit localizations
- or 2x the number of localizations from 2011
- x # of feedback questionnaires completed
P2PU Challenges(this could also be for other things like parable or nav badge if not integrated into p2pu - "interactive challenges")
- x # of webpages created as a result of challenges
- Share of users who start a (Hackasaurus) challenge
- Share who abandon a challenge
- Share of users who complete a challenge
- Share of users who complete more than one challenge (2, 3, 4, ...)
- Share of users who complete a series of challenges
- Number of Challenges completed per learner
- Number of Badges issued over time (organized by badge type)
- Number of badges issued per learner (making and understanding badges)
- Number of Mentors over time (some measure of level of activity)
Community
- x # of countries running hack jams
- x # of hackasaurus "experiments" by community members
- x # of localizations
- x # of participants in Hackasaurus events
BRAINSTORMING METRICS:
Overall goals
>Evaluation Goals and Metrics for Challenges
- The objective of these metrics is to assess whether or not Challenges on P2PU are a good way to begin creating educational content for Mozilla projects. Both quantitative and qualitative methods are used to determine value for Mozilla, P2PU, and the user.
- Two sets of metrics:
- NOT FOCUS -> Internal - Once within a challenge - what are the metrics that tell us how well someone is doing, and how to make the challenge features / UX better. Click patterns. Drop off points. Etc.
- FOCUS -> Overall value - We have content and we are moving it into this challenge framework. How is that going? How many people are completing our content (within this framework). Conversion funnel. Completion rates.
Framing:
- Focus metrics on state change - attitude, awareness, actions taken
3 Levels of Impact for Metrics for MoFo (framing for our larger goals - see if we can re-frame some of the metrics below into Participation and Making/Learning buckets for consistency)
- Understanding - what increase in understanding/awareness of the open web and our programs (this is not as relevant for P2PU challenges specifically)
CV: challenges provide a model for deep understanding, which is measured primarily in the ability of learners to reflect their knowledge. A tangible example of that is to create a tutorial for someone else on a topic.
- Participation - how many people are participating, in what ways, how deeply
CV: an easy way to measure that is sequencing of comments: we do not care necessarily about how many comments have been made, what we care about as an indicator of participation is how many of these comments are an answer to a previous comment. Additionally things like sharing with external social graphs provd=ide an indicator > more on how to assess social assessment & participation here <a href="http://bit.ly/vloJcH">http://bit.ly/vloJcH</a>
- Making - how are people improving their skills
CV: not clear on this, how is this different from understanding?
- Innovation - how are people innovating (not as relevant?)
CV: brilliant point- something to consider is measuring how many times for example a unique solution, such as a piece of code, is being re-used in other ways.
Can we currently measure all these on P2PU? What do the metrics look like for Hackasaurus (what do we have available to us there?)
CV: no we cannot probably. but we can start considering ways we can measure participation (stealth assessment) as well as understanding (manual assessment)
PS: Main comment -> Need to reduce number of metrics / let's track general traffic and then pick a few important ones (max 3-5) to drill down
Goals
Participation (beyond Reach)Determine success of functional improvement
How many people is Hackasaurus currently engaging? How many more might Challenges bring to the Hackasaurus Project? historic, current, and predictive views using quantitative data
Method: data analysis (Monitor for trends) – historical and current, weekly collection
Metrics (increasing depth of participation):
- Access
- Basic demographics (from Google Analytics)
- Referrer stats (from Google Analytics)
- Session Duration
- Number of total page views by language and country
- Conversion rate
- Users who hit the site
- Share of users who register but never start a challenge
- Share of users who start a challenge
- Share who abandon a challenge
- Share of users who complete a challenge
- Share of users who complete more than one challenge (2, 3, 4, ...)
- Share of users who complete a series of challenges
- Number of Challenges completed per learner
- Nice to have: Usage pattern over time (levels of activity / completion speed)
- Maybe hours / week over time
- Learning
- Number of Active Learners over time (ie those who complete a task once a week)
- Number of Challenge/Curriculum Completions over time (ie Learners who have completed all the Challenge/Curriculum tasks)
- Share of users who complete at least one challenge
- Ratio of users who started a challenge and completed it
- Click Stream
- Badges
- Number of Badges issued over time (organized by badge type)
- Number of badges issued per learner (making and understanding badges)
- what about collaborating = learning behavior?
- Mentoring
- Number of Mentors over time (some measure of level of activity)
- This can probably only be done through a survey, because we let mentors structure their communication however they want - a lot of their interaction happens outside of p2pu.org
- Conversation rates from participant to helper to mentor (deeper participation)
- sequence of comments > that is something we could measure
- Number of Gurus (e.g. users who have ability to issue valuable badges)
- (does this exist?)- Yes, during the first pilot of Badges on P2PU, they seeded the community with gurus (ie "Has-A-Badge") assessors. @Chloe @Philipp - Do you know how many "gurus" are active on the site?
- Number of Mentors over time (some measure of level of activity)
- Peer Assessment (as a form of participation)
- Share of total users who participated in peer assessment (e.g. 20%)
- Share of peer to peer badges compared to overall badges (e.g. 15%)
- Number of badges that lack sufficient reviews
Qualitative Survey to ask people how they find the experience
Making
- Number of Badges (that are tied to making) issued over time
- Number of Links to participants work (gathering external links will allow us to see what/if people are making)
- Handle through survey questions
Innovating
- Handle through survey questions
PS: I combined the following with Participation aboveCommunityMetrics
3: Up Peer Assessment / e.g. How well does the peer assessment work?Method: data analysis (Monitor for trends) – historical and current, 1 week after launch, 4 weeks after launch, 3 months after launchMetrics
Share of total users who participated in peer assessment (e.g. 20%)Share of peer to peer badges compared to overall badges (e.g. 15%)Number of badges that lack sufficient reviews
PS: Doesn't fit with Mozilla metrics framework:
4: User Survey - Determine perceived value to users
Method: survey 3 months after launch
Sample strata
- Learners who abandoned a challenge (target problem areas)
- Learners who never started a challenge
- Learners who completed a challenge
Indicators / Variables
- [future version] + contribution to getting a job / only for relevant challenges
- Contribution to personal satisfaction
- Contribution to social recognition
- Focus on state changes - changes in attitude, awareness, actions taken
- Did you tell your friends about this?
- Would you recommend it to a friend?
- .... need to do more work here
5: Ongoing gathering of empirical information to improve individual challenges
Method: Each learner is asked a single question after completing a challenge
Metrics
- + Learner Satisfaction
- + Challenge structure success (target problem areas)
Possible Wording (need to wordsmith) -
- Help us make this challenge better!
- Did you think this challenge was a little silly – just right - too boring
- Did you think this challenge was too hard – just right – too easy / simple
OTHER NOTES
ROI (Long Term)
How have donations increased or decreased? How has participation spread?
Method: data analysis (Quantitative from Mozilla and P2PU user data, donation data) – Ongoing after launch, long term.
Metrics
- + donation stream to p2pu and Mozilla
- + number of new signups
- - Staff Hours
- - Average Response Rate
- + Total Likes, One ups or RT on social media messaging marked #challenges
Notes
- Suggestion is to postpone this and add to later phase (agree)
- Risk that adding option to donate into the challenge will change the user's learning experience (we really want to measure if the learning works at this point - adding donations may influence the outcomes)
- - Average Challenge Completion Time (currently located within the Challenge metrics)
- - Average Task Completion Times
- + Number of new challenges by the community
Quantitative
Collecting these metrics will allow us to define meta metrics (ie number of learners vs non learners vs power learners or whatever)
Across the Board:
Metrics for Understanding and Awareness of Open Web and Mozilla Projects
- Basic demographics
- Number of total page views by language and country
- Referrer stats
- Click Stream
The Metrics Story: Collecting the demographics of users will allow Mozilla to further focus their programs to specific target groups. Total number of page views by language and country furthers this definition. This will allow Mozilla to spend it's resources designing programs for people, programs that work. This will influence stats in participation depth, which will show progress towards the 10 Million Webmakers marker. Knowing which referrers are the most valuable will help streamline resources and eliminate erroneous spending on marketing and/or partnerships. Following the clickstream will help Mozilla understand what sorts of content inspire understanding, which will further Mozilla's ability to create content that pushes people to becoming webmakers.
Participation Depth
- IP
- Session Duration & Clicks per Session
- Think time
- Conversion rate
- Share of users who register but never do anything
The Metrics Story: Although IP logging is a raw metric, with thousands of users it will be valuable to see how deep into Mozilla programming users are going. By cross comparing IP logs between programs, Mozilla will have a better view of influence and partipation depth across the board. Collecting session durations and clicks per session and seeing an increase in these two metrics over time will further underline this viewpoint. Think time can be used to filter out users who simply browse. Decreasing negative conversion rates is important in showing strength of programming.
Skill Improvement
- Number of Badges issued over time (organized by badge type)
- Number of Links to participants work (gathering external links will allow us to see what/if people are making)
The Metrics Story: The more badges that are issued and the more quality links that are submitted to Mozilla sites, the clearer the influence of Mozilla on skill improvement.
Platform Specific
P2PU
Participation Depth
- Number of Mentors over time
- Conversation rates
- from participant to helper to mentor (deeper participation)
- Number of Active Learners over time (ie those who complete a task once a week)
- Share of users who complete at least one challenge
- Ratio of users who started a challenge and completed it
- Share of total users who participated in peer assessment (e.g. 20%)
- Share of peer to peer badges compared to overall badges (e.g. 15%)
Skill Improvement
- Number of Gurus (e.g. users who have ability to issue valuable badges)
- Number of badges that lack sufficient reviews
- Number of Challenge/Curriculum Completions over time (ie Learners who have completed all the Challenge/Curriculum tasks)
01.12.11 Metrics Meeting
Attendees
- Steph
- Laura
- Philipp
- Chloe
Dealing with Resources
- Need to extend Arlton's contract to work on Challenges UX – involved 3.5 months ago. P2PU hired 2 people, 1 for Challenge content and 1 for UX. Arlton and Jamie Curle.
- Was getting the UX person the most efficient way to use the funds (Philipp says no). Discuss in longer call (TBD).
- Need to figure out the longer term strategy for people
- SoW – funded early badges work, SoW Community Management, Webmaking 101 Challenges. 50% p2pu funding (Zuzel, Chloe, John paid with p2pu funds) and 50% from SoW funding.
- By January, launching Hackasaurus on P2PU – get Challenges rocking the house.
- Zuzel is superhuman -- really?
Metrics Discussion
<a href="https://etherpad.mozilla.org/challenges-evaluation-goals">https://etherpad.mozilla.org/challenges-evaluation-goals</a>
Laura to set up metric meeting with Jess. Consistent set of metrics! Invite Steph
How do the p2pu metrics line up with the Hackasaurus metrics, which one are close enough aligned that we can then compare. Comparision chart.
Parse metrics – send out
Agreements
Challenge Fixes Board: Released Dec. 14
Implementation of Partner Account that allows Hackasaurus to run their challenges
Success looks like:
The list of agreed upon must haves and launch by mid January