Mozilla Community Surveys/Completed/SUMO

Introduction

The second survey completed as part of the MCS project examined perceptions of support.mozilla.org (SUMO).

Publicly available summaries of parts of the analysis are available on the MCS blog (post one, post two, post three).

Research Questions

1. To get a better understanding on people’s perception of the quality of support

2. To understand and measure how Mozilla is doing at support in the different locales

3. To find out how can we make Mozilla support more useful

Instrument

We asked four questions in total, three of which directly related to our objectives.

Question 1: How would you rate the support that community provides for Mozilla products in your language? (1- lowest quality, 5- highest quality)

• Mozilla overall
• Firefox
• Thunderbird
• Calendar
• SeaMonkey
• Camino
• Add-ons

Question 2: How accurate are the following statements for your community? (1 - least accurate, 5 - most accurate, plus "Don't know")

In my community…

• most of the relevant information can be found in my locale
• the support sites have a structure that is easy to read and navigate
• users ask questions in the right channels
• users rarely ask the same question more than once
• the support sites are up-to-date
• the support sites cover well the most common issues
• the support sites load quickly in my browser
• the support sites make use of screencasts and tutorial videos to help our users
• new issues/problems with Mozilla software are quickly added to support sites

Question 3: What resources make or would make the user-to-user support in your language more effective? (1: least important to 5: most important)

• Tutorials/How-tos
• Troubleshooting articles
• Support available in my language
• Support available from a single site
• Screenshots
• Screencasts/videos
• Specific versions of support depending on operating system (Windows/Mac/Linux)
• Forums
• Live Chat

Question 4: What’s your involvement in the Mozilla project:

• I’m a user of a Mozilla product
• I follow the news from the project
• I help other users in the discussion boards
• I write documentation
• I localize documentation
• I localize Mozilla products
• I develop Mozilla code

Respondents could choose more than one option. These variables were used to create a new variable with three values:

• End-users — those who responded that they were users of Mozilla products, but didn’t select any other answer

• Community members — those who responded that they used Mozilla products and followed the news on the Mozilla project

• Active community members — those who respond that they were localizers, developers, users helping other users, and authors of the support articles for Mozilla products.

Analysis Plan

Research Question One: Overall Perceptions of Quality of Support

We address this question simply by reporting on the typical values and spread of responses to Questions 1, 2 and 3. These variables are 5 point rating scales and hence it is acceptable (in social science) to treat them as being of the interval level of measurement. This means that

• it is appropriate to report means, standard deviations and confidence intervals for each of the variables involved.

• if we become interested in testing whether one variable has a higher mean than another we can use a paired-sample t test.

Research Question Two: Differences in Perceptions By Locale

There are only six locales with a large enough sample size for statistically valid comparisons to be made. As we will be comparing means between six groups one-way ANOVA is the appropriate technique to use. Given the similiarity of sample sizes (not different by an order of magnitude or more) Tukey's HSD post-hoc tests will be used to explore significant ANOVA results.

Research Question Three: Making SUMO More Useful

The same techniques that we used for RQ1 are applicable here.

Results

RQ1 & RQ3: Overall Perceptions

Average Values of All Variables

	N		Mean	Median	Mode
	Valid	Missing	Valid	Missing	Valid
locale	1558	0
country	1558	0
Mozilla overall	1274	284	4.04	4.00	4
Firefox	1454	104	4.35	5.00	5
Thunderbird	1165	393	3.96	4.00	4
Calendar	599	959	3.34	3.00	4
SeaMonkey	527	1031	3.39	3.00	3(a)
Camino	416	1142	3.22	3.00	3
Add-ons	1245	313	3.74	4.00	4
Most of the relevant support is in my language	1453	105	3.87	4.00	4
Support in English is sufficient	1486	72	2.04	2.00	1
Navigation on the support sites is easy	1445	113	3.71	4.00	4
Users ask questions in the right channels	1195	363	3.34	3.00	4
Users rarely ask the same question more than once	1177	381	2.70	3.00	2
The support sites are up-to-date	1344	214	3.80	4.00	4
Most common issues are covered well	1353	205	3.83	4.00	4
The support sites load quickly	1394	164	4.04	4.00	4
The support sites make use of video	1050	508	2.78	3.00	3
New issues are quickly addressed	1265	293	3.74	4.00	4
Tutorials / How-tos	1558	0	3.91	4.00	5
Troubleshooting articles	1558	0	4.11	4.00	5
Availibility from a single site	1558	0	3.90	4.00	5
Screenshots	1558	0	3.65	4.00	4
Screencasts / videos	1558	0	3.10	3.00	3
OS specific information	1558	0	3.74	4.00	5
Forums	1558	0	3.81	4.00	4
LiveChat (embedded)	1558	0	2.76	3.00	3
VoIP	1558	0	2.39	2.00	1
IM	1558	0	2.91	3.00	1
A user of a Mozilla product	1558	0	.98	1.00	1
a Multiple modes exist. The smallest value is shown

<object type="image/svg+xml" name="omap" data="RPlots.svg" width="350" height="176">

Frequencies

Locale

locale

		Frequency	Valid Percent
Valid	ca	4	.3
	cs	1	.1
	da	5	.3
	de	228	14.6
	en	67	4.3
	es	57	3.7
	fr	283	18.2
	hu	101	6.5
	it	158	10.1
	pl	187	12.0
	pt	35	2.2
	ro	5	.3
	ru	406	26.1
	sq	2	.1
	sv	7	.4
	uk	12	.8
	Total	1558	100.0

User Profile

User Profile

		Frequency	Percent
	End user	289	18.5
	Community Member	768	49.3
	Active Community Member	501	32.2
	Total	1558	100.0

RQ2: Differences in Perceptions by Locale

Perceptions of aspects of support were significantly different (at the 1% level) by locale for all variables except “the site loads quickly” and support for Camino.

The full results of statistical tests are shown here.

The table below shows how many time each locale rated a product the lowest, and how many time each rated a product the highest. Note that because of the overlapping feature of multiple comparisons some the counts do not sum to the number of products.

Locale	Low Count	High Count
de	5	1
fr	5	1
hu	3	5
ru	4	5
pl	2	5
it	2	5

It seems clear from this summary that German and French respondents generally rate the products lower than do respondents from other locales; conversely Polish and Italian respondents rate the products higher.

Summary

RQ1: Overall Evaluations

Products

• The average ratings of all products are on the positive side of neutral
• Firefox is rated the highest, and higher than all others.
• Mozilla overall, Thunderbird and Add-ons are rated next highest, and no different from each other
• Calendar, Seamonkey and Camino are rated lowest and no different from each other

SUMO

The highest rating (most accurate statements) variables are, with no difference between them:

• Most of the relevant support is in my language
• Navigation on the support sites is easy
• The support sites are up-to-date
• Most common issues are covered well
• The support sites load quickly
• New issues are quickly addressed

Then:

• Users ask questions in the right channels

Then:

• The support sites make use of video
• Users rarely ask the same question more than once

Finally, the lowest (least accurate statement):

• Support in English is sufficient

The last three variables are on the “inaccurate” side of the scale.

RQ2: Differences in Evaluations by Locale

The results are a classic example of the applicability of the phrase “Lies, damned lies and statistics”. There are two possible (and not mutually exclusive explanations for this result:

• For the locale-specific needs of respondents, Mozilla products serve their customers better in some locales than in others (Managerial implication: put extra effort into locale-specific services and products in French and German locales)

• Because of cultural factors, people in some locales tend to rate 'everything higher or lower than do people in other locales (Managerial implication: do nothing, or at least do nothing until the researchers have adjusted the results to take into account cultural factors)

RQ3: Improving SUMO

The most highly desired changes are:

• Troubleshooting articles

Followed by:

• Tutorials / How-tos
• Availibility from a single site
• OS specific information
• Forums
• Screenshots

Then:

• Screencasts / videos

Then:

• LiveChat (embedded)
• IM

Finally:

• VoIP
• IM

The last three are on the “least important” side of neutral.