Mozilla Community Surveys/Completed/SUMO

Introduction

The second survey completed as part of the MCS project examined perceptions of support.mozilla.org (SUMO).

Publicly available summaries of parts of the analysis are available on the MCS blog (post one, post two, post three).

Research Questions

1. To get a better understanding on people’s perception of the quality of support

2. To understand and measure how Mozilla is doing at support in the different locales

3. To find out how can we make Mozilla support more useful

Instrument

We asked four questions in total, three of which directly related to our objectives.

Question 1: How would you rate the support that community provides for Mozilla products in your language? (1- lowest quality, 5- highest quality)

• Mozilla overall
• Firefox
• Thunderbird
• Calendar
• SeaMonkey
• Camino
• Add-ons

Question 2: How accurate are the following statements for your community? (1 - least accurate, 5 - most accurate, plus "Don't know")

In my community…

• most of the relevant information can be found in my locale
• the support sites have a structure that is easy to read and navigate
• users ask questions in the right channels
• users rarely ask the same question more than once
• the support sites are up-to-date
• the support sites cover well the most common issues
• the support sites load quickly in my browser
• the support sites make use of screencasts and tutorial videos to help our users
• new issues/problems with Mozilla software are quickly added to support sites

Question 3: What resources make or would make the user-to-user support in your language more effective? (1: least important to 5: most important)

• Tutorials/How-tos
• Troubleshooting articles
• Support available in my language
• Support available from a single site
• Screenshots
• Screencasts/videos
• Specific versions of support depending on operating system (Windows/Mac/Linux)
• Forums
• Live Chat

Question 4: What’s your involvement in the Mozilla project:

• I’m a user of a Mozilla product
• I follow the news from the project
• I help other users in the discussion boards
• I write documentation
• I localize documentation
• I localize Mozilla products
• I develop Mozilla code

Respondents could choose more than one option. These variables were used to create a new variable with three values:

• End-users — those who responded that they were users of Mozilla products, but didn’t select any other answer

• Community members — those who responded that they used Mozilla products and followed the news on the Mozilla project

• Active community members — those who respond that they were localizers, developers, users helping other users, and authors of the support articles for Mozilla products.

Analysis Plan

Research Question One: Overall Perceptions of Quality of Support

We address this question simply by reporting on the typical values and spread of responses to Questions 1, 2 and 3. These variables are 5 point rating scales and hence it is acceptable (in social science) to treat them as being of the interval level of measurement. This means that

• it is appropriate to report means, standard deviations and confidence intervals for each of the variables involved.

• if we become interested in testing whether one variable has a higher mean than another we can use a paired-sample t test.

Research Question Two: Differences in Perceptions By Locale

There are only six locales with a large enough sample size for statistically valid comparisons to be made. As we will be comparing means between six groups one-way ANOVA is the appropriate technique to use. Given the similiarity of sample sizes (not different by an order of magnitude or more) Tukey's HSD post-hoc tests will be used to explore significant ANOVA results.

Research Question Three: Making SUMO More Useful

The same techniques that we used for RQ1 are applicable here.

Results

RQ1: Overall Perceptions

Average Values of All Variables

	N		Mean	Median	Mode
	Valid	Missing	Valid	Missing	Valid
locale	1558	0
country	1558	0
Mozilla overall	1274	284	4.04	4.00	4
Firefox	1454	104	4.35	5.00	5
Thunderbird	1165	393	3.96	4.00	4
Calendar	599	959	3.34	3.00	4
SeaMonkey	527	1031	3.39	3.00	3(a)
Camino	416	1142	3.22	3.00	3
Add-ons	1245	313	3.74	4.00	4
Most of the relevant support is in my language	1453	105	3.87	4.00	4
Support in English is sufficient	1486	72	2.04	2.00	1
Navigation on the support sites is easy	1445	113	3.71	4.00	4
Users ask questions in the right channels	1195	363	3.34	3.00	4
Users rarely ask the same question more than once	1177	381	2.70	3.00	2
The support sites are up-to-date	1344	214	3.80	4.00	4
Most common issues are covered well	1353	205	3.83	4.00	4
The support sites load quickly	1394	164	4.04	4.00	4
The support sites make use of video	1050	508	2.78	3.00	3
New issues are quickly addressed	1265	293	3.74	4.00	4
Tutorials / How-tos	1558	0	3.91	4.00	5
Troubleshooting articles	1558	0	4.11	4.00	5
Availibility from a single site	1558	0	3.90	4.00	5
Screenshots	1558	0	3.65	4.00	4
Screencasts / videos	1558	0	3.10	3.00	3
OS specific information	1558	0	3.74	4.00	5
Forums	1558	0	3.81	4.00	4
LiveChat (embedded)	1558	0	2.76	3.00	3
VoIP	1558	0	2.39	2.00	1
IM	1558	0	2.91	3.00	1
A user of a Mozilla product	1558	0	.98	1.00	1
Follows the news on Mozilla	1558	0	.79	1.00	1
Helps other users in the discussion boards	1558	0	.27	.00	0
Writes documentation	1558	0	.06	.00	0
Localizes documentation	1558	0	.06	.00	0
Localizes Mozilla product(s)	1558	0	.08	.00	0
Develops the Mozilla code	1558	0	.04	.00	0
A user	1558	0	.1855	.0000	.00
A community member	1558	0	.4929	.0000	.00
An active community member	1558	0	.3216	.0000	.00
User_profile	1558	0	2.1361	2.0000	2.00
a Multiple modes exist. The smallest value is shown

<object type="image/svg+xml" name="omap" data="RPlots.svg" width="350" height="176">

Frequencies

Locale

locale

		Frequency	Valid Percent
Valid	ca	4	.3
	cs	1	.1
	da	5	.3
	de	228	14.6
	en	67	4.3
	es	57	3.7
	fr	283	18.2
	hu	101	6.5
	it	158	10.1
	pl	187	12.0
	pt	35	2.2
	ro	5	.3
	ru	406	26.1
	sq	2	.1
	sv	7	.4
	uk	12	.8
	Total	1558	100.0

User Profile

User Profile

		Frequency	Percent
	End user	289	18.5
	Community Member	768	49.3
	Active Community Member	501	32.2
	Total	1558	100.0

RQ2: Differences in Perceptions by Locale

Perceptions of aspects of support were significantly different (at the 1% level) by locale for all variables except “the site loads quickly” and support for Camino.

Homogenous Subsets

How to interpret these tables: Each column contains a set of groups that are not significantly different from each other. Some groups appear in two columns, this means that they are the intersection of two (or more) overlapping sets of groups.

Mozilla overall
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	181	3.83
fr	243	3.88
ru	282	4.11	4.11
hu	90	4.11	4.11
pl	167		4.22
it	148		4.22
Sig.		.052	.838
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 161.978.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Firefox
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
de	211	4.16
fr	271	4.20	4.20
ru	373		4.43	4.43
pl	171		4.45	4.45
it	154			4.47
hu	95			4.58
Sig.		.998	.068	.573
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 176.584.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Thunderbird
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	168	3.82
fr	225	3.85
ru	299	4.05	4.05
it	123	4.06	4.06
pl	129	4.06	4.06
hu	77		4.25
Sig.		.313	.574
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 140.811.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Calendar
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	84	3.11
fr	114	3.16	3.16
ru	138	3.39	3.39
pl	61	3.56	3.56
it	65	3.63	3.63
hu	40		3.70
Sig.		.088	.069
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 70.837.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

SeaMonkey
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	69	3.03
fr	75	3.15
ru	153	3.50	3.50
hu	36	3.53	3.53
pl	61	3.57	3.57
it	51		3.78
Sig.		.128	.776
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 61.136.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Add-ons
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
fr	236	3.42
hu	79	3.61	3.61
de	179	3.77	3.77
ru	310		3.85
pl	147		3.93
it	136		3.96
Sig.		.096	.085
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 150.513.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Most of the relevant support is in my language
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
pl	176	3.69
de	203	3.81	3.81
fr	269	3.84	3.84
hu	94	3.90	3.90
ru	386	3.96	3.96
it	149		4.07
Sig.		.141	.170
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 175.102.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Support in English is sufficient
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
ru	396	1.66
fr	275	1.95	1.95
pl	179		2.04
it	152		2.08
de	209		2.08
hu	96		2.26
Sig.		.119	.083
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 178.952.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Navigation on the support sites is easy
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
pl	177	3.55
de	206	3.57
fr	263	3.66	3.66
ru	383	3.77	3.77
hu	90	3.83	3.83
it	149		3.92
Sig.		.071	.121
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 172.712.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Users ask questions in the right channels
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	1
de	191	3.24
hu	68	3.25
pl	156	3.27
ru	322	3.28
fr	202	3.49
it	112	3.55
Sig.		.053
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 138.452.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Users rarely ask the same question more than once
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
fr	201	2.43
ru	320	2.52
it	113	2.58	2.58
pl	153	2.69	2.69
de	178		2.92
hu	66			3.85
Sig.		.312	.094	1.000
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 135.578.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

The support sites are up-to-date
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
fr	234	3.67
pl	169	3.68	3.68
de	197	3.70	3.70
hu	79	3.77	3.77	3.77
ru	356		3.98	3.98
it	141			4.02
Sig.		.930	.055	.169
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 158.620.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Most common issues are covered well
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
pl	169	3.50
de	197	3.79	3.79
hu	78		3.81	3.81
fr	241		3.82	3.82
ru	366		3.93	3.93
it	135			4.09
Sig.		.069	.712	.073
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 157.469.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

The support sites make use of video
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
ru	263	2.38
de	167	2.71	2.71
fr	194		2.90	2.90
pl	145		2.94	2.94
it	104			3.16
hu	46			3.26
Sig.		.213	.580	.140
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 112.790.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

New issues are quickly addressed
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
de	192	3.57
pl	157	3.59
ru	333	3.75	3.75
fr	231	3.79	3.79	3.79
hu	71		4.01	4.01
it	126			4.07
Sig.		.377	.175	.125
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 146.589.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Tutorials / How-tos
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	228	3.61
pl	187	3.76	3.76
ru	406	3.87	3.87
fr	283		4.02
it	158		4.04
hu	101		4.08
Sig.		.209	.087
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Troubleshooting articles
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
hu	101	3.90
fr	283	3.99	3.99
pl	187	4.01	4.01
de	228	4.07	4.07
it	158		4.22
ru	406		4.26
Sig.		.500	.076
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Availibility from a single site
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
de	228	3.47
fr	283	3.80	3.80
it	158		3.82
hu	101		4.00	4.00
ru	406		4.10	4.10
pl	187			4.17
Sig.		.063	.094	.680
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Screenshots
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
ru	406	3.42
hu	101	3.46
it	158	3.54	3.54
de	228	3.75	3.75
pl	187		3.81
fr	283		3.85
Sig.		.057	.087
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Screencasts / videos
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
ru	406	2.64
it	158		3.07
de	228		3.10
hu	101		3.28
fr	283		3.31
pl	187		3.43
Sig.		1.000	.076
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

OS specific information
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	1
ru	406	3.56
de	228	3.61	3.61
pl	187	3.80	3.80	3.80
fr	283	3.83	3.83	3.83
it	158		3.94	3.94
hu	101			4.07
Sig.		.241	.085	.233
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

Forums
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
hu	101	3.62
fr	283	3.70	3.70
pl	187	3.73	3.73
it	158	3.80	3.80
de	228	3.81	3.81
ru	406		3.99
Sig.		.542	.115
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

LiveChat (embedded)
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
ru	406	2.60
it	158	2.65	2.65
fr	283	2.69	2.69
de	228	2.73	2.73
pl	187	2.97	2.97
hu	101		2.99
Sig.		.069	.128
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

VoIP
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	3	4	1
ru	406	2.06
fr	283	2.20	2.20
it	158	2.44	2.44	2.44
de	228		2.46	2.46
pl	187			2.82	2.82
hu	101				2.88
Sig.		.070	.391	.062	.997
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

IM
Tukey HSD

locale_code	N	Subset for alpha = .05
	1	2	1
de	228	2.64
fr	283	2.69
it	158	2.85
ru	406	2.96	2.96
hu	101	3.01	3.01
pl	187		3.29
Sig.		.114	.185
Means for groups in homogeneous subsets are displayed.
a Uses Harmonic Mean Sample Size = 187.733.
b The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.