User:Bhashem/AddonAuditTools

Here is my braindump of what some add-on audit tools might do. It's based on the AMO add-on review process.

Overall Goal: Ensure that only high-quality, secure add-ons are hosted on AMO

• Provide AMO Editors tools to help them evaluate add-on quality
• Provide tools to add-on authors so that they "lint before they submit"

General Techniques

• A: Use structural analysis to uncover errors and characterize add-ons
• B: Use language static analysis for syntactic and semantic checking of an add-on
• C: Use run-time analysis to characterize what an add-on is doing?

A: Structural Analysis

• Review the File/Directory Structure
  • Is it a valid XPI or JAR? (for themes)
  • Is jar packaging OK? (missing basic files, etc...)
  • Are there duplicate files/directories (common mistake is to include chrome/myadd.jar + all the files at the same level - they are never used)
  • Do there appear to be binary components or executables?
  • Does it appear to have OS platform specific files?
  • Is there orphaned content/skins/chrome/etc?

B: Static Analysis

Extension Type-Specific rules

• Is only allowed content included?
  • For Dictionaries - e.g., no JS files!
  • For Themes
  • For Locale Packs

File type: Install.rdf

• Syntactically correct RDF
• Semantic errors - e.g. invalid version ranges, invalid RDF references, unknown app GUIDs, invalid add-on GUID formats, etc...

File type: Chrome.manifest

• Syntax errors
• Are there dangling references in chrome.manifest?

File type: .js

• Passes Jslint with selected options
• Namespacing concerns
• Does it make use of XHR?
• Does it load content in chrome?
• Does it generate JavaScript errors on invocation/functional use?
• Does it use eval for evil?
• Does it load remote JS?
• Does it load remote XUL?

File type: .xul

• MFinkle's XUL Validator

File type: .css

• TBD

C: Run-time Analysis

• Use some sort of specially instrumented Firebug add-on or a JS Shell
• Does it leak memory?
• Does it load remote JS?
• Does it load remote XUL?
• Does it load non-https content as chrome?