User:Clouserw/AMO/Validator:v2

Version 1 of our add-on validator has been a great success and really proven the value of an automated validation system. This year we need to take it to the next level with some new features, finishing some rough edges, and making the code rock solid.

Version 1 stuff:

• original spec
• current help docs

Version 2

Idea List

• Written in Python
• Written in such a way as it's portable, e.g. ./validate.py addon.xpi
• Supports offline processing via gearman so we don't hold up server threads waiting on processing and we don't hit memory/time limits.
  • Progress should be checked via ajax
  • We should be able to see how many add-ons are in the queue at any time. If it's ever a large number we'll get more processing power.
• Jetpack verification support
• Better L10n support (there are a couple old bugs about this, and we need to update the external)
• Add-ons for unit tests can be built on the fly. Jbalogh has a short script that this can be based on.
• Levels of flagging? Like, a minimum level and a super picky level. Or would this just be too confusing and everyone would always only use one?
• Real time virus scanning

Jorge suggests:

• Be smart about caching: only clear validation results cache when the file is modified or the validator is modified. Maybe clearing the validation cache could be added as a step for production pushes.
• Give priority to add-on uploads (by authors) in the validation queue.
• Give severity weight to warnings and sort them by severity.
• Show help text next to the warnings, instead of a separate page.
• Recognize common validation patterns and make it easier to add them. For example:
  • [ file mask (*.js), what to look for (/eval/), severity (high), grouping ].
  • (the grouping would be for grouping similar results together and showing the link to the validation page)
• Easy to download and distribute (to developers).
  • note that we offer the validation tool to add-ons that don't host on AMO already
• Smarter JS library recognition. Use common internal patterns instead of just file names.
• Nils' validator, written in Python. We should definitely coordinate with him and see how we can integrate his ideas.

Ian suggests:

• It seems like Redis is going to come into the mix in Zamboni, and there's already Cucumber, so one of those would be appropriate for the queuing.
• Presumably in stand-along mode everything is just done immediately, no queuing (i.e., the queuing is really a separate system).
• Jetpack has a bunch of Python tools for building stuff, I don't know if they have validation too.
• I would flag everything and only filter in the UI.
  • As a result, it might be good to use some structured output from the validate script (also maybe more easily extendable by unit tests?). Maybe just use the XUnit output format.