Support/Personal Support 2010Q2

From MozillaWiki
Jump to: navigation, search

This month, I set out to assess the relative effectiveness and reach of each method of personal support that SUMO is offering - the forum, live chat, and Twitter. Chatting with a user, as expected, takes a lot more time than responding to a forum post. Many helpers, however, find live chat to be more engaging and effective. We need to find why that is – and more importantly how we can extend the active live chat community to other areas of SUMO.

I started by asking some of our top contributors about their perceptions on the different means of support in an informal survey. The survey responders were mostly live chat contributors, mainly because these people are online more often and easier to reach. Later, using the ideas from the survey, I measured actual effectiveness based on time spent and questions answered.

Informal interviews/surveys

To begin the process of assessing our effectiveness, I decided to conduct an “informal survey” with interested community members. I shared a document containing several questions with each respondent, then had quick discussions on IRC or Skype as people started to fill in answers. The questions were open ended, asking how each person got started with SUMO, what they find rewarding and challenging with contributing, which types of questions they try to answer, and how much time they usually spend each week helping with each method of support. I stressed that it should be though of more than a discussion than a survey, since I really wanted to get ideas and open ended feedback about what about live chat works – and what doesn't.

I first asked each user to compare the three methods of support with regard to effectiveness. Most of the people I surveyed – which were mainly live chat contributors – said that live chat was the most effective way to give support. 100% of the responses mentioned enjoying “thank you”'s from users, which happen much more often in Live Chat than in the forum. This result was rather surprising, since a few of the people I surveyed also said that they aren't helping very often due to live chat's hours or that live chat takes too much time.

When I narrowed the survey to ask what makes each method effective, the most popular responses were the gratefulness of users and the instant response times in live chat. Newer chat contributors mentioned that they like being able to get help from more senior contributors instantly, while more veteran chat contributors mentioned reasons such as multitasking with different users at the same time, the ability to perform more advanced troubleshooting steps, and being able to see exactly what other other helpers are doing. The advantages mentioned by the few forum respondees I got were that the forum doesn't require a time commitment, and that more questions can be answered with less effort.

I next asked what we could improve to make each method more effective. The problems identified in Live Chat were requiring Spark to help, not being open often enough, and that some chats can take up to an hour to solve. For the forum, everyone who helped there noted that most users don't reply – making contributors feel that effort is being wasted.


Data (in progress)

This quarter, I've been working to identify ways we can better measure the relative effectiveness of each method of support. To better analyze the perceptions of the community, I've been measuring data about how the same users are contributing to each method of support. Now that we have unified SUMO accounts for both the forum and live chat, we can evaluate the performance of live chat and the forum with the same set of contributors.

  • Each user helped about the same number of questions each time they helped, whether in the forum or live chat

[histogram: forum sessions vs. livechat sessions, all contributors]

  • Live Chat interactions were much more in depth, with chat times ranging from 30 seconds to 3 hours.
  • “churn” among active contributors is much higher in higher in live chat than the forum, where the most active contributors tend to stay the same from week to week. (Likely due to higher time requirements)

[Graph: Helpers “gained” and “lost” per week who answered at least 5 questions] [Stacked bar graph: Individual helper participation per week]


Discussion points and observations

  • Instant gratification is really important: all live chat helpers mentioned this in the survey
  • Participation in Live Chat tends to go up in weeks where there are fewer chat requests.
  • In Live Chat, helpers can see each other work in real-time, which seems to build a stronger sense of community.
  • The fact that many users never get back to their e-mails or forum threads is the main turn-off for live chat helpers using the forum
  • People seem to really like the live chat workflow, especially for users that ask unclear questions or when advanced troubleshooting is needed. (Canned responses, multitasking, instant replies)
    • Can we bring some of these aspects to the forum?
    • Live Chat tends to be most useful for users needing in-depth explanations or those with issues under investigation
      • Can we focus live chat to serve more of these users?