Support/IssueTrackerPRD

Latest proposal

Overview

The Sumo issue tracker project aims to build a new system to track user issues that we encounter on SUMO in a centralized location to simplify metrics gathering and make it easier for helpers to communicate findings to each other.

What it replaces

The current system for metrics collection and knowledge sharing uses a lot of manual labor from zzxc and Cww and works like this:

1. Helpers answer questions
   • Cww and zzxc help answer tough questions or questions that helpers are unfamiliar with using their collected knowledge
   • Helpers have to search Google or mozillazine or various other sources if they encounter a problem they can't answer and we're not around to answer questions. Or they must turn away the user. Some solutions found may be out of date or not fully relevant.
2. Cww and zzxc read a sampling of reports each week, with over 2000 forum threads and 300 live chats, this is a very time consuming task.
3. Cww and zzxc create metrics and file bugs as needed for new issues or new KB articles
4. Cww and zzxc have weekly discussions to make sure we're on the same page, sharing information informally.
5. Cww reports to QA & dev
6. Cww or zzxc also has to "lead" the investigation of any thorny issues that are found
7. zzxc develops troubleshooting guides for specific issues
8. When a new issue needs documentation, or if a helper is trying to find information on a problem for a user, a search can turn up dozens of threads and hits and each of these must be read through to find the desired information.

This suffers the following issues:

• Metrics collection is a very labor-intensive step that's separate from answering questions. At least once a week, Cww and zzxc need to read through hundreds of forum threads/chat sessions to gather data for that week's metrics and figure out the most common issues. Helpers who actually answer questions do no sort or classify the issue as they answer them. Since we can only take a sampling of issues and can't ask followup questions of the users, the metrics are rough and lack specifics.
• Helpers must learn about new issues from zzxc or Cww, not each other. For LiveChat, this means that helpers have a hard time opening without zzxc. For forums, many helpers do not follow the weekly common issues or learn from each other, having to rediscover solutions that others may have found or give outdated advice. This also means training new helpers is a personalized and highly labor intensive process. We don't even do it on the forums and it takes quite a lot of hours on Live Chat.
• KB integration is poor. Cww and zzxc also have to track all the KB articles out there and what they cover so we can help helpers find the right article for a given problem.
• Without a comprehensive index of forum threads, chat logs, contributors writing documentation or QA/devs looking for report cases need to do crude word-based searches and filter through a large number of reports.

What we're building

Having a centralized location which groups threads/chat logs by issue and associates them with solutions and/or KB articles would alleviate each of these concerns.

• Since threads/chats will be marked as specific issues while the helper is answering them, we'll already have preliminary metrics by the end of the week. Cww and zzxc can then spend their time with tricky issues or new issues which really need our attention or working with dev and QA teams.
• Since issues will be associated with solutions and/or troubleshooting steps and/or KB articles, helpers can see which solutions worked for other users with the same or similar issues and can leave notes for future helpers, building a collective pool of information rather relying on two people.
• New KB articles will be associated into the index so that helpers finding a specific issue in the database will see right away that there's a KB article written for it.
• Issues will be linked using a multitude of ways to each other (tags, via a system that tracks symptoms, by version number) so that it's very easy to find related issues if a specific one is unable to solve a problem or when writing documentation or doing followup work.

In terms of detailed workflow, we imagine that it would work something like this:

1. Helpers see symptom/user report, copy ID to issue tracker
   • In the future, we hope to have tighter integration with the SUMO interface (as in you'd have the tracker UI on the same page as the forum thread UI or in the LiveChat client) but for its first iteration, the tracker will be independent and will require a single copy and paste action.
2. Issue tracker fills out most of the issue report form via AJAX and a single query to the SUMO databases
3. Helper uses dynamic issue tracker search fields to locate troubleshooting instructions and relevant issue number and solutions
4. Helpers use solutions linked from issue to solve user's question, copying and pasting instructions or by typing them out in his own words.
5. Helper clicks OK to file new issue report, linking the log of the chat or forum thread to the issue for searching down the line.
   • If helper doesn't use the issue tracker or file a report, anyone with access to the thread or the chat log can go back to do it, allowing gaps to be filled. The tracker will take care of duplicates.
6. Helpers leave comments on issues to help future helpers -- if needed
7. Difficult issues or new issues or exceedingly common issues are flagged and triaged by support team or experienced helpers
8. Cww and zzxc generate regular reports (automatically) from the data to report to qa/devs/rest of team
9. Many other reporting options exist so contributors looking to write missing articles or other Mozilla community members looking for more information will be able to quickly find it.

Scope of project

We're hoping the tracker will be comprehensive in that every single forum thread and chat makes it into the tracker. There will probably need to be a few catch-all issues (for non-Firefox issues, for issues that require a tutorial rather than troubleshooting, for extremely rare or bizarre cases, for cases of incomplete information). Detailed tracking will probably be limited to somewhat common, Firefox issues and focus primarily on troubleshooting. This is purely because the other cases are difficult to get unified collective information on. As far as followup, each issue has suggested "steps" which are either steps to get more info for devs, known workaround/solution/extension to disable, or a link to a KB article or a bug number to send for more detailed followup. Unlike bugzilla, the issue tracker will NOT be tracking patches or detailed technical information or even steps to reproduce (other than what's reported by users).

Possible drawbacks

1. The largest concern is that this will make answering questions slower as we're asking contributors to search this database for the relevant issue before answering. Therefore, we'll need tight coupling with the forums and livechat so that this process can be as streamlined as possible. Also, we hope that since the database will be providing help to the contributor in the form of suggested troubleshooting steps and canned responses that can be copied over, we'll be saving helpers time in terms of the amount of searching they have to do. Since it will make it easier for relatively inexperienced helpers to contribute (just search, and copy and paste), a slight reduction in an individual helper's productivity may be offset by having more contributors overall.
2. There is quite a resource investment getting this off the ground. In addition to coding, the initial database needs to populated with issues.

Timeline (proposed)

Since the tracker has a number of goals and parts, we've put together two preliminary timelines for the project depending on priorities.

Tag-focused

An easy way to get started with a tiki-wiki compatible interface and having helpers classify issues is by adding tags to forum threads that can be set by contributors. With some suggested tags (for common symptoms, for different Firefox versions), we can make it easier to track issues and help contributors find related threads. This will form the seed data for the larger database and tracker and will also provide some feedback as to how contributors respond to having to mark each thread. The advantage to starting with this approach first is it gets contributor involvement right away and we can take their feedback on usage.

The goals for each milestone would be as follows:

• Scope M1: tag support for forum threads and live chat sessions (including Spark support for tags, roll out our own Spark builds), search for posts and pages associated with a tag (eg. by clicking it), allow trusted contributors access to chat
• Scope M1.5: gather feedback from helpers, evaluate needs for future development, add polish, fix bugs, get more metrics
• Scope M2: Integrated tag database used by both Openfire and Tikiwiki
• Scope M3: some tags automatic when pasting KB links, etc (version numbers, plugins, OS, etc), CSAT part 2 (karma, helpers' name stored in CSAT, etc), making tag addition easy (tag completion/search)
• Scope M?: expand tag database, tags with special meanings (issue tags, symptom tags), automatic metrics, integration with knowledge base and issue/solution database
• Scope M?: advanced interface for using issue tracker to solve issues, easy report generation, finding all related posts/chat logs for a specific issue
• Other considerations: sanitation of personal information in live chat logs

And the likely resource requirements.

• Resources M1 (20 hours by Matthew, 4 hours by Cheng, 5 hours by Webdev)
  • Forum tags: 5-6 hours of work by webdev
    • Need a new field for inputting tags, saved in a basic database table on tikiwiki
  • Tag support in Spark: 15 hours, by Matthew (implementing, creating Spark builds, debugging, explaining to helpers, helper documentation)
  • Allow trusted contributors access to chat logs, with logging and automatic e-mail address sanitation: 4 hours, by Matthew
  • Building database queries, setting up on tools server: 4 hours by Cheng and/or Matthew
  • Add 1 hour to Cheng's weekly metrics load
• Resources M1.5 (12 hours by Cheng, 12 hours by webdev)
  • (No major changes, focusing on user experience and defining future requirements)
  • Evaluate results, gather feedback, file bugs, develop metrics: 12 hours by Cheng
  • Fixing bugs, adding polish, features (based on feedback) to make tags more useful: 12 hours (Matthew and/or webdev)
• Resources M2 (~30 hours all Matthew and/or webdev)
  • Database for tag storage, linkage with live chat and forum, communication with Openfire and Tikiwiki: at least 4 hours by Matthew for planning/specifications, ~5 hours by webdev for tikiwiki integration, ~20 hours implementation/testing (Matthew and/or webdev)
• Resources M3
  • Automatic tags when pasting links, etc: 4 hours by Matthew, 4 hours by webdev
  • Tag autocompletion, using database from M2: ?
  • Integration with CSAT Part 2 (karma, etc): 5 hours by Matthew, 5 hours by webdev

Timeline: M1 can be Q1, M1.5 throughout Q2, M2 not before Q3 - after tag system is working and it's clear we should move on

Database-focused

Alternately, the first goal can be oriented around setting up a database that tracks all forum threads and livechat logs. The advantage to doing this approach first is that zzxc and Cww can use it right away when they're sorting issues manually every week. So after implementing a database and before the UI is implemented on the tiki-wiki side, we can already get better metrics using an interface written in another language. The advantage of starting with this approach first is that we'll get good metrics right away (since it's only being used by Cww and zzxc) and that since it's not integrated into tikiwiki right away, we don't need to add to the sumodev timeline.

• Scope M1: Finalize issue tracker database schema (including solutions, issues, symptoms, etc), integrate into bare-bones interface for sumo team to use, add ability to group forum threads and live chat logs into issues, automatic metric queries
• Scope M1.5: Analyze feedback from metrics, create more useful metrics, add features to database interface, integrate into sumo metrics dashboard
• Scope M2: Open tracker up to interested contributors, get feedback on what works for them and what doesn't. (Implement autocompletion, security features, UI improvements)
• Scope M3: Integrate with sumo (Issues have comments, integration into sumo search, tags in chats/forum)
• Scope M3.5: Fix bugs, add polish, etc

• Resources M1: 40 hours, by Cheng and/or Matthew (15 hours by each coding, 5 hours by each planning)
• Resources M1.5: 4 hours by Matthew fixing bugs, 5-6 hours by Cheng collecting feedbacks and creating metrics
• Resources M2: 5 hours by Cheng and/or Matthew, 15 hours of webdev (some by Matthew)
  • Security improvements: webdev, 5 hours
  • UI improvements: 5 hours by Cheng and/or Matthew
  • Autocompletion of fields, make as automatic as possible: 10 hours (webdev and/or Matthew)
• Resources M3: an entire sumo milestone (40-50 hours webdev)

• Timeline: M1 in Q1, M1.5 end of Q1/beginning of Q2

---

Rough Notes

Data we currently collect for each issue/would like to collect

• Status of fixing each issue (Bugzilla)- link to bug for every issue
• Status of documenting each issue
• Suggested instructions to fix or work around
  • Diagnostic questions to ask (dynamic content)
• Data we need to collect to resolve each issue
• Person assigned as the lead for each issue
• Links to existing documentation that covers each issue
• Samples and frequency data from each issue, from both forum and live chat
• Tags to organize issues
• Link issues together, link issues to symptoms, link issues to solutions, etc (solutions & symptoms are records)
• Operating system(s), Firefox version(s), URL, CC list (via watches)
• Future: Language field, links between issues in different languages

Problems with our current collection of data

• Not unified: live chat and forum data collected at different times and listed on different pages
• Live Chat data requires a lot of manual work, including manually reading chat logs and adding information to a page
• Lag of up to a week as dumps are downloaded and processed
• No ability to quickly query for similar problems
• Information can only be read in weekly reports, manual queries (eg. all issues with symptom X) can't be run by interested people

Alternate Proposed Workflow (Future):

1. Tracker uses a script to pull all forum threads and details from tikiwiki
2. Helpers use issue tracker and can answer posts from there, adding comments and/or details as needed.
3. As above.

Features we'd want (% = Future)

• Integration with tikiwiki forums and livechat
• Eventual ability for sumo contributors to add information from their own chats or forum replies %
• Good query tool to allow anyone to produce a custom report
• Integration with CSAT: list number of forum replies solved, potentially solved, unsolved, etc for each issue %
  • How successful is sumo in solving this issue in KB, forums, and live chat?
• Each issue viewable with a Bugzilla-like interface, with the ability for contributors to comment directly on an issue
• Attachments, with security parameters to allow only trusted users (admins) to download %
• XML output and/or sumobot integration %

Possible Database schema

• Symptoms table, each symptom contains textual description, "AppliesTo", tags, URL links (for article bugs and KB articles, newsgroup discussion, anything else), cc list, text field wiki markable for general instructions
  • Future: automated walkthrough based on decisions
• Relationships table with "cloud" for related symptoms. Or linkage entirely done by tags
• Solutions table, name ("newprofile"), description ("Make a new Firefox profile"), details ("1) close Firefox...")
  • Dynamic content for common instructions, used in details
• Issue table
  • Detailed description field
  • Links to other issue tracker items (symptoms, solutions, tags, reports)
  • Automated counters for metrics based on reports
• Report table (automatic)
  • Index: By autocounter and random hash
  • Type: (forum, live chat, kb)
  • Index: (chat id, forum thread id, kb report id - currently a user poll object)
  • Status: (solved, potentially solved, unsolved)
  • List of plugins, Firefox version, OS, extensions, build id
  • Tags: (example: newissue, critical, security)
    • If you are replying to a forum thread and you are a contributor, you see a tags field under "Submit"
  • Text field (could contain question asked, editable at will)
  • Random hash from CSAT
• Links tables: make the links described above possible
  • Table 1: reports to issues (catchall issues)
  • Table 2: links issues to symptoms, solutions, cc members

Other things to consider

• Catchall issues, eg. "Firefox Crashes" for each major symptom
• Catchall solutions: "basic troubleshooting",
  • "Catch many" solutions: "Make a new profile"
• Need to have dynamic content in various fields
• CSAT part 2 - Need to consider anonymous users, etc