Drumbeat/p2pu/Assessment and Accreditation/Webcraft Assessments - detailed/Methodology v1: Difference between revisions

← Older edit

Drumbeat/p2pu/Assessment and Accreditation/Webcraft Assessments - detailed/Methodology v1 (view source)

Revision as of 18:22, 29 July 2010

3,660 bytes added , 29 July 2010

→‎Determining competencies of interest – a methodology

Ahrashb

191

edits

@@ Line 1: / Line 1: @@
-== Determining competencies of interest – a methodology ==
+== Determining skills of interest – a methodology ==
 ''Background''
-For the P2PU/Mozilla School of Webcraft project, we need to identify a suite of competencies (skills, habits, knowledge, etc) that are of particular interest to employers and collaborators who need to evaluate the potential for someone to be a good professional fit for their needs. In actuality, this problem is ubiquitous for most disciplines, especially when it comes to non-knowledge-based abilities, but we are focused on the School of Webcraft here.
+For the P2PU/Mozilla School of Webcraft project, we need to identify a suite of skills (competencies, habits, knowledge, etc) that are of particular interest to employers and collaborators who need to evaluate the potential for someone to be a good professional fit for their needs. In actuality, this problem is ubiquitous for most disciplines, especially when it comes to non-knowledge-based abilities, but we are focused on the School of Webcraft here.
 We have talked to experts in the field, and we plan to continue doing so. But there is a need for a more comprehensive, replicable methodology for determining the "desirable qualities" of someone in an open-source programming environment. The following approach has not been tested, to our knowledge, but seems worthwhile, and may prove to be generally applicable.
@@ Line 28: / Line 28: @@
 If we really want to demonstrate the potential for this approach, we should recruit several different people to perform parallel analyses (i.e., pursue the same questions on the same system, but independently) so that we can evaluate the extent to which the resulting conclusions differ from each other. Some diversity is fine, but if this methodology is sufficiently objective, we should see broadly similar results for each of the efforts.
+== Data ==
+=== First analysis of LinkedIn data - UK company ads ===
+We generated a spreadsheet of ad copy within LinkedIn for many different web developer positions posted by companies. The file is available here: [[File:P2PU_work%28uk%29.odt]].
+Using the [http://www.antlab.sci.waseda.ac.jp/software.html AntConc 3.2.1] program, we generated a word list based on frequencies of use within the ads. There are other types of analyses we could perform as well, but it makes more sense to first explore the raw data for patterns and then analyze specific text snippets based on those analyses. The top words (in descending order to a minimum of 3 hits) associated with desirable "habits" were:
+# Team (8  hits) - as in a ''team player'' or ''working as part of a team.''
+# Agile (7  hits) - as in ''agile software development.''
+# Creative (5 hits) - as in ''strong, flexible, creative flair.''
+# Writing, written, and verbal (5,3,3 hits) - as in ''excellent and proven writing skills'' or ''excellent written and verbal communication skills.''
+# Communication (4 hits) - as in ''excellent communication skills.''
+# Analytical (3 hits) - as in ''strong analytical skills.''
+# Attention (3 hits) - as in ''attention to detail.''
+# Attitude (3 hits) - as in ''positive attitude.''
+# Designing (3 hits) - as in ''experience designing and optimizing code.''
+# Fast (3 hits) - as in ''working in a fast paced environment.''
+# Thinker (3 hits) - as in ''logical thinker'' or ''critical thinker.''
+A quick review of this list reveals no real surprises in terms of the types of skills and habits that are in demand by employers of web developers. Most employers want creative and analytical thinkers who work well on teams, have strong communication (written and verbal) skills, and can work quickly and flexibly. The bigger challenge, of course, is figuring out ways to authentically identify these traits in the employee pool.
+We will be performing a companion analysis using recommendations from colleagues which are more likely to contain words and phrases that are less standardized and perhaps more revealing. As the dataset grows, we will also be able to group and outline typical words and phrases into larger categories and gain some analytical power as a result, especially for better understanding exactly which types of web developer positions favor certain skills and habits over others.
+=== Second analysis of LinkedIn data - personal recommendations in LinkedIn ===
+[http://wiki.mozilla.org/images/5/5f/SOW_recommendation_data_29_July_2010.odt This spreadsheet] contains snippets from personal recommendations that people wrote for each other and posted to LinkedIn. All of the people here have some responsibility for web development.
+Again using the [http://www.antlab.sci.waseda.ac.jp/software.html AntConc 3.2.1] program, we generated a word list based on frequencies of use within the recommendations. The top words (in descending order to a minimum of 3 hits) associated with desirable "habits" were:
+# Team (4  hits) - as in a ''team player'' or ''working as part of a team.''
+# Leader (3  hits) - as in ''strong group leader.''
+# Communicator (3 hits) - as in ''great communicator.''
+These were the only words in common for the raw descriptions to the minimum standard of 3 hits. Obviously, we could use more data. But this small list is also reflective of the vastly greater diversity in the terminology people use to describe each other, as opposed to the semi-standardized terminology that you find in job descriptions (see above). It would be worth taking some time to browse the descriptions manually and do some grouping and outlining.