Javascript:SpiderMonkey:ES6 Testing

Objective

Summary: Properly test all new features and aspects of ES6.

ES6, the next iteration of ECMAScript, introduces many new features, new methods, new algorithms, and occasionally makes changes to existing methods and algorithms. Implementing these changes correctly will require a considerable number of tests for all additions and changes.

Historically most Mozilla JS tests have been self-written and work only in SpiderMonkey test harnesses. There is an effort at ECMA, test262, to introduce a standardized test format, harness, and so on, so that tests that are written can be run by any browser/engine. We should make an effort to write more tests in standardized format, so that tests we write can be used by other engines, improving the web for everyone -- even if they don't use SpiderMonkey or a Mozilla-based browser.

People

Accountable: Naveed
Responsible: Waldo, jorendorff
Consulted:
Informed: Product Marketing

Steps

Enumerate the features/details/changes in ES6

This simply requires running through ES6 drafts, as they're published, and noting things that will need testing.

• The ordinary/exotic object distinction (relevant to proxies, arrays, and the exotic objects mentioned in 8.4/8.5)
• proper parsing of template literals, wrt the InputElementTemplateTail goal symbol -- ASI
• Unicode format-control characters within template literals -- line terminators as well
• identifier goal symbols seem to have changed some -- to use the Unicode properties ID_Start and ID_Continue (and also _/$/<ZWJ>/<ZWNJ>) -- may need testing
• new keywords: class, continue, const, export, import, super (possibly all reserved before, tests needed for their new semantics, and places that still forbid them)
• likely-new keyword yield
• regular expression literals syntax rules appear different -- maybe something to do here, maybe not
• "8.1.6.4 Well-Known Symbols and Intrinsics": are these directly observable, or only indirectly through spec algorithms' uses of them?
• 8.2.4.3 GetThisValue(V): how is this used, where, maybe needs tests (if not just some sort of refactoring of |this| keyword handling, not sure yet)
• the Property Descriptor type now has an optional [[Origin]] field -- possibly/likely used for property descriptors returned from proxy hooks -- needs testing
• new [[HasOwnProperty]] hook -- previously this was simply derived from [[GetOwnProperty]]
• new [[Enumerate]] and [[OwnPropertyKeys]] hooks with well-defined semantics for for(var i in o) and Object.getOwnPropertyNames/keys -- visible in proxies
• functions (ordinary function objects, 8.3.15.1) have a bunch more internal data properties to track -- affecting semantics in various ways; realms are perhaps the most important one (corresponding to the multiple-globals problem in web browsers)
• |new| on a function whose [[Prototype]] has been changed to null/undefined -- semantics of this are still partially up in the air
• there seems to be a requirement that (function() { "use strict"; }).caller === null per 8.3.15.3 which might be a change from ES5 -- needs investigation, testing
• [[BuiltinBrand]] is new and probably affects every place that currently does exact-class checks (or exact-class-modulo-other-globals checks)
• String exotic objects: new internal ops defined for them, may require more testing for out-of-range -- also for negative indexes, which in the current spec seem to be passed through abs() (!)

New features:

• template literals (nee quasis)
  • templates with zero or more substitution parts
• binary integer literals
• floating point hex literals (maybe? not added yet, might be)
  • \u{} Unicode escape sequences* \u{HexDigits} is a syntax error if HexDigits > 1114111 (0x10FFFF)
  • \u{} with a surrogate-pair number, other reserved code points
• Binary Data proposal (8.2.1)
  • data block type
• super
  • implicates Super References (8.2.4), and the super keyword -- something to do with calling superclass methods, and similar things; haven't grokked details yet
• ToPrimitive is now explicitly hookable in objects (doesn't use toString/valueOf properties), using a well-known symbol/intrinsic
• scriptable Proxy objects (8.5) (direct proxies)

Import the test262 suite into Mozilla, and run it on tinderbox

There's relatively little impetus for us to write test262 tests if we can't run them as often as we run all our existing tests. It should be a high priority to set up a process through which test262 tests are imported into our tree, run on each new change, and regressions flagged so that the relevant patches can be reverted.

Examine the test262 test format/harness to identify any limitations in it

We frequently add new features and functionality to the JS shell for testing purposes. We add methods to help verify implementation internals, where adding directly to the shell makes obvious sense. But we also add functionality to make it easier to test particular bits of entirely-standard functionality -- to permit evaluating multiple global scripts in a single test, for example. We've added these features because, in our harness, this was the easiest way to test a feature. test262 doubtless has similar limitations, and as these tests must be cross-engine, it won't be possible to use Mozilla-specific helper functions.

We need to examine the test262 harness and test format and identify these limitations, and consider how they might be resolved or worked around.

Issues

Importing third-party tests into our tree, that we don't necessarily pass (particularly as tests for new features are introduced), requires a mechanism for marking third-party tests as failing. Ideally, this must be possible without editing the imported tests, or at least with as few edits to the imported tests as possible. It's not clear what this mechanism will look like, but it's critical that we have it.

Risks

test262's test format may not be adequate to test all behavior required by ES6. Tests that must run in global code, for example, may not necessarily fit into the test262 format. (This is purely my recollection from having looked into this a few years ago -- things may have changed since.) Tests which require executing multiple JS scripts (that is, through multiple <script> elements in the browser) also may pose issues. Other instances may arise over time. Dealing with these harness limitations will require adding new functionality to the harness, and may delay properly testing some new features.