Drumbeat/p2pu/Assessment and Accreditation/Webcraft Assessments - detailed/Methodology v1

Determining competencies of interest – a methodology

Background

For the P2PU/Mozilla School of Webcraft project, we need to identify a suite of competencies (skills, habits, knowledge, etc) that are of particular interest to employers and collaborators who need to evaluate the potential for someone to be a good professional fit for their needs. In actuality, this problem is ubiquitous for most disciplines, especially when it comes to non-knowledge-based abilities, but we are focused on the School of Webcraft here.

We have talked to experts in the field, and we plan to continue doing so. But there is a need for a more comprehensive, replicable methodology for determining the "desirable qualities" of someone in an open-source programming environment. The following approach has not been tested, to our knowledge, but seems worthwhile, and may prove to be generally applicable.

Methodology

1. Using a professional social network (such as LinkedIn), identify to classes of people (there is likely to be some overlap): 1) people who identify as web developers, and 2) people who manage or hire web developers.
2. In each case, browse the following items, if present:
   1. Recommendations from other people.
   2. Recommendations for other people.
   3. Job ads they have posted (that are relevant).
   4. Descriptions of themselves.
   5. Any other information that seems relevant.
3. Cut and paste into a separate document or spreadsheet those parts of the browsed items that refer to specific skills and habits, which should (as a rule) be positive.
   1. This process will result in a database of sorts composed of text snippets which refer to skills and habits that are considered desirable by web developers and those who employ them.
4. The resulting database can be analyzed by hand, or by text-analysis software, to extract common words and phrases that refer to desired competencies. This process will produce a new database.
5. This final database serves as the raw material for any additional consideration of the listed attributes. It will likely need some additional sorting and aggregating, as well as some evaluation from experts and the broader community as a form of validation for the results.

Comments

We believe this process is robust and replicable. In aggregating the individual texts, the origin URLs and other relevant data should be captured as well so that others can perform their own analyses of the information if they wish.

There may be ways to streamline this even more, for example by querying the social network itself for certain types of words and phrases (the reverse of what is suggested above) and then noting the relative incidence of the different text-items.

If we really want to demonstrate the potential for this approach, we should recruit several different people to perform parallel analyses (i.e., pursue the same questions on the same system, but independently) so that we can evaluate the extent to which the resulting conclusions differ from each other. Some diversity is fine, but if this methodology is sufficiently objective, we should see broadly similar results for each of the efforts.

Data

First analysis of LinkedIn data - UK company ads

We generated a spreadsheet of ad copy within LinkedIn for many different web developer positions posted by companies. The file is available here.

Using the AntConc 3.2.1 program, we generated a word list based on frequencies of use within the ads. There are other types of analyses we could perform as well, but it makes more sense to first explore the raw data for patterns and then analyze specific text snippets based on those analyses. The top words (in descending order to a minimum of 3 hits) associated with desirable "habits" were:

1. Team (8 hits) - as in a team player or working as part of a team.
2. Agile (7 hits) - as in agile software development.
3. Creative (5 hits) - as in strong, flexible, creative flair.
4. Writing, written, and verbal (5,3,3 hits) - as in excellent and proven writing skills or excellent written and verbal communication skills.
5. Communication (4 hits) - as in excellent communication skills.
6. Analytical (3 hits) - as in strong analytical skills.
7. Attention (3 hits) - as in attention to detail.
8. Attitude (3 hits) - as in positive attitude.
9. Designing (3 hits) - as in experience designing and optimizing code.
10. Fast (3 hits) - as in working in a fast paced environment.
11. Thinker (3 hits) - as in logical thinker or critical thinker.