Mozilla Community Surveys/Completed/SUMO
Contents
Introduction
The second survey completed as part of the MCS project examined perceptions of support.mozilla.org (SUMO).
Publicly available summaries of parts of the analysis are available on the MCS blog (post one, post two, post three).
Research Questions
1. To get a better understanding on people’s perception of the quality of support
2. To understand and measure how Mozilla is doing at support in the different locales
3. To find out how can we make Mozilla support more useful
Instrument
We asked four questions in total, three of which directly related to our objectives.
Question 1: How would you rate the support that community provides for Mozilla products in your language? (1- lowest quality, 5- highest quality)
- Mozilla overall
- Firefox
- Thunderbird
- Calendar
- SeaMonkey
- Camino
- Add-ons
Question 2: How accurate are the following statements for your community? (1 - least accurate, 5 - most accurate, plus "Don't know")
In my community…
- most of the relevant information can be found in my locale
- the support sites have a structure that is easy to read and navigate
- users ask questions in the right channels
- users rarely ask the same question more than once
- the support sites are up-to-date
- the support sites cover well the most common issues
- the support sites load quickly in my browser
- the support sites make use of screencasts and tutorial videos to help our users
- new issues/problems with Mozilla software are quickly added to support sites
Question 3: What resources make or would make the user-to-user support in your language more effective? (1: least important to 5: most important)
- Tutorials/How-tos
- Troubleshooting articles
- Support available in my language
- Support available from a single site
- Screenshots
- Screencasts/videos
- Specific versions of support depending on operating system (Windows/Mac/Linux)
- Forums
- Live Chat
Question 4: What’s your involvement in the Mozilla project:
- I’m a user of a Mozilla product
- I follow the news from the project
- I help other users in the discussion boards
- I write documentation
- I localize documentation
- I localize Mozilla products
- I develop Mozilla code
Respondents could choose more than one option. These variables were used to create a new variable with three values:
- End-users — those who responded that they were users of Mozilla products, but didn’t select any other answer
- Community members — those who responded that they used Mozilla products and followed the news on the Mozilla project
- Active community members — those who respond that they were localizers, developers, users helping other users, and authors of the support articles for Mozilla products.
Analysis Plan
Research Question One: Overall Perceptions of Quality of Support
We address this question simply by reporting on the typical values and spread of responses to Questions 1, 2 and 3. These variables are 5 point rating scales and hence it is acceptable (in social science) to treat them as being of the interval level of measurement. This means that
- it is appropriate to report means, standard deviations and confidence intervals for each of the variables involved.
- if we become interested in testing whether one variable has a higher mean than another we can use a paired-sample t test.
Research Question Two: Differences in Perceptions By Locale
There are only six locales with a large enough sample size for statistically valid comparisons to be made. As we will be comparing means between six groups one-way ANOVA is the appropriate technique to use. Given the similiarity of sample sizes (not different by an order of magnitude or more) Tukey's HSD post-hoc tests will be used to explore significant ANOVA results.
Research Question Three: Making SUMO More Useful
The same techniques that we used for RQ1 are applicable here.
Results
RQ1 & RQ3: Overall Perceptions
Average Values of All Variables
N | Mean | Median | Mode | ||
---|---|---|---|---|---|
Valid | Missing | Valid | Missing | Valid | |
locale | 1558 | 0 | |||
country | 1558 | 0 | |||
Mozilla overall | 1274 | 284 | 4.04 | 4.00 | 4 |
Firefox | 1454 | 104 | 4.35 | 5.00 | 5 |
Thunderbird | 1165 | 393 | 3.96 | 4.00 | 4 |
Calendar | 599 | 959 | 3.34 | 3.00 | 4 |
SeaMonkey | 527 | 1031 | 3.39 | 3.00 | 3(a) |
Camino | 416 | 1142 | 3.22 | 3.00 | 3 |
Add-ons | 1245 | 313 | 3.74 | 4.00 | 4 |
Most of the relevant support is in my language | 1453 | 105 | 3.87 | 4.00 | 4 |
Support in English is sufficient | 1486 | 72 | 2.04 | 2.00 | 1 |
Navigation on the support sites is easy | 1445 | 113 | 3.71 | 4.00 | 4 |
Users ask questions in the right channels | 1195 | 363 | 3.34 | 3.00 | 4 |
Users rarely ask the same question more than once | 1177 | 381 | 2.70 | 3.00 | 2 |
The support sites are up-to-date | 1344 | 214 | 3.80 | 4.00 | 4 |
Most common issues are covered well | 1353 | 205 | 3.83 | 4.00 | 4 |
The support sites load quickly | 1394 | 164 | 4.04 | 4.00 | 4 |
The support sites make use of video | 1050 | 508 | 2.78 | 3.00 | 3 |
New issues are quickly addressed | 1265 | 293 | 3.74 | 4.00 | 4 |
Tutorials / How-tos | 1558 | 0 | 3.91 | 4.00 | 5 |
Troubleshooting articles | 1558 | 0 | 4.11 | 4.00 | 5 |
Availibility from a single site | 1558 | 0 | 3.90 | 4.00 | 5 |
Screenshots | 1558 | 0 | 3.65 | 4.00 | 4 |
Screencasts / videos | 1558 | 0 | 3.10 | 3.00 | 3 |
OS specific information | 1558 | 0 | 3.74 | 4.00 | 5 |
Forums | 1558 | 0 | 3.81 | 4.00 | 4 |
LiveChat (embedded) | 1558 | 0 | 2.76 | 3.00 | 3 |
VoIP | 1558 | 0 | 2.39 | 2.00 | 1 |
IM | 1558 | 0 | 2.91 | 3.00 | 1 |
A user of a Mozilla product | 1558 | 0 | .98 | 1.00 | 1 |
a Multiple modes exist. The smallest value is shown |
<object type="image/svg+xml" name="omap" data="RPlots.svg" width="350" height="176">
Frequencies
Locale
Frequency | Valid Percent | ||
---|---|---|---|
Valid | ca | 4 | .3 |
cs | 1 | .1 | |
da | 5 | .3 | |
de | 228 | 14.6 | |
en | 67 | 4.3 | |
es | 57 | 3.7 | |
fr | 283 | 18.2 | |
hu | 101 | 6.5 | |
it | 158 | 10.1 | |
pl | 187 | 12.0 | |
pt | 35 | 2.2 | |
ro | 5 | .3 | |
ru | 406 | 26.1 | |
sq | 2 | .1 | |
sv | 7 | .4 | |
uk | 12 | .8 | |
Total | 1558 | 100.0 |
User Profile
Frequency | Percent | ||
---|---|---|---|
End user | 289 | 18.5 | |
Community Member | 768 | 49.3 | |
Active Community Member | 501 | 32.2 | |
Total | 1558 | 100.0 |
RQ2: Differences in Perceptions by Locale
Perceptions of aspects of support were significantly different (at the 1% level) by locale for all variables except “the site loads quickly” and support for Camino.
The full results of statistical tests are shown here.
The table below shows how many time each locale rated a product the lowest, and how many time each rated a product the highest. Note that because of the overlapping feature of multiple comparisons some the counts do not sum to the number of products.
Locale | Low Count | High Count |
---|---|---|
de | 5 | 1 |
fr | 5 | 1 |
hu | 3 | 5 |
ru | 4 | 5 |
pl | 2 | 5 |
it | 2 | 5 |
It seems clear from this summary that German and French respondents generally rate the products lower than do respondents from other locales; conversely Polish and Italian respondents rate the products higher.
Summary
RQ1: Overall Evaluations
Products
- The average ratings of all products are on the positive side of neutral
- Firefox is rated the highest, and higher than all others.
- Mozilla overall, Thunderbird and Add-ons are rated next highest, and no different from each other
- Calendar, Seamonkey and Camino are rated lowest and no different from each other
SUMO
The highest rating (most accurate statements) variables are, with no difference between them:
- Most of the relevant support is in my language
- Navigation on the support sites is easy
- The support sites are up-to-date
- Most common issues are covered well
- The support sites load quickly
- New issues are quickly addressed
Then:
- Users ask questions in the right channels
Then:
- The support sites make use of video
- Users rarely ask the same question more than once
Finally, the lowest (least accurate statement):
- Support in English is sufficient
The last three variables are on the “inaccurate” side of the scale.
RQ2: Differences in Evaluations by Locale
The results are a classic example of the applicability of the phrase “Lies, damned lies and statistics”. There are two possible (and not mutually exclusive explanations for this result:
- For the locale-specific needs of respondents, Mozilla products serve their customers better in some locales than in others (Managerial implication: put extra effort into locale-specific services and products in French and German locales)
- Because of cultural factors, people in some locales tend to rate 'everything higher or lower than do people in other locales (Managerial implication: do nothing, or at least do nothing until the researchers have adjusted the results to take into account cultural factors)
RQ3: Improving SUMO
The most highly desired changes are:
- Troubleshooting articles
Followed by:
- Tutorials / How-tos
- Availibility from a single site
- OS specific information
- Forums
- Screenshots
Then:
- Screencasts / videos
Then:
- LiveChat (embedded)
- IM
Finally:
- VoIP
- IM
The last three are on the “least important” side of neutral.