Javascript:SpiderMonkey:ES6 Testing

Objective

Summary: Properly test all new features and aspects of ES6.

ES6, the next iteration of ECMAScript, introduces many new features, new methods, new algorithms, and occasionally makes changes to existing methods and algorithms. Implementing these changes correctly will require a considerable number of tests for all additions and changes.

Historically most Mozilla JS tests have been self-written and work only in SpiderMonkey test harnesses. There is an effort at ECMA, test262, to introduce a standardized test format, harness, and so on, so that tests that are written can be run by any browser/engine. We should make an effort to write more tests in standardized format, so that tests we write can be used by other engines, improving the web for everyone -- even if they don't use SpiderMonkey or a Mozilla-based browser.

Accountable: Naveed
Responsible: Waldo, jorendorff
Consulted:
Informed: Product Marketing

Milestones

Status	Task	Deliverable	Assigned	Estimate	Expected	Actual
done	Get test262 running on Tinderbox	Test262 import script works, imports test262 tests, manually run periodically to get updates	Waldo
in progress	Implement @negative support for imported test262 tests, bug 496923	Most of the test262 skips in js/src/tests/jstests.list are removed	Waldo	1 week	20130823
in progress	Import the ch15 test262 tests	js/src/tests/test262/ch15 exists, the tests are run on Tinderbox	Waldo	1 week	20130823
in progress	Process for upstreaming test262 contributions	Have at least one test262 contribution in js/src/tests/test262/to-upstream/ from someone not involved in the test262 harness work	Waldo	3 days
in progress	Fix remaining issues preventing us from running any test262 tests	Only actually-failing tests skipped	Waldo	1 week	20130823
pending	Evaluate potential test262 format changes, simplifications to ease contributions	Iron out test-format concerns we have that make it hard to run the tests with our shell or in the browser; scope unknown, but synchronous script inclusion from script really should go	Waldo	1 week
in progress	Draw up a list of the ES6 features/changes to be tested	Spec fully run through, all new portions listed with points of interest called out	Waldo	1 week

Import the test262 suite into Mozilla, and run it on tinderbox

Owner: jwalden Status: done

There's relatively little impetus for us to write test262 tests if we can't run them as often as we run all our existing tests. It should be a high priority to set up a process through which test262 tests are imported into our tree, run on each new change, and regressions flagged so that the relevant patches can be reverted.

Fix remaining issues preventing us from running any test262 tests

Owner: jwalden
Status: in progress
Estimate: 1 week

The initial test262 import was relatively brain-dead: import everything, skip tests that fail (validly or not). We need to go back and fix up our test harnesses, or the test262 import script, to run those additional tests.

Issues to fix:

• @negative support
• others?

Examine the test262 test format/harness to identify any limitations in it

Owner: jwalden
Status: incomplete
Estimate: 1 week

We frequently add new features and functionality to the JS shell for testing purposes. We add methods to help verify implementation internals, where adding directly to the shell makes obvious sense. But we also add functionality to make it easier to test particular bits of entirely-standard functionality -- to permit evaluating multiple global scripts in a single test, for example. We've added these features because, in our harness, this was the easiest way to test a feature. test262 doubtless has similar limitations, and as these tests must be cross-engine, it won't be possible to use Mozilla-specific helper functions.

We need to examine the test262 harness and test format and identify these limitations, and consider how they might be resolved or worked around. Note that as test262 is a shared resource, any changes we request will have to be discussed with and reviewed by non-Mozilla people to consider their effects on others, an extra bit of process that may make this take longer than it would if it were totally under our control.

Process for upstreaming test262 contributions

Owner: jwalden
Status: incomplete
Estimate: 3 days

Contributions to test262, going through test262, probably have too much friction to get developers to make them in the course of usual bug-fixing/regression-fixing. We should set up a js/src/tests/test262/to-upstream/ directory into which test262-format tests can be dumped. Then, whoever runs the update script can be responsible for moving them to a reasonable location, etc. and contributing them to test262. When they're landed upstream, we can run the update script, pick up the new tests, and remove the to-upstream ones.

This mostly requires adding the new to-upstream directory and writing and disseminating documentation about how to write test262 tests.

Enumerate the features/details/changes in ES6

Owner: jwalden
Status: incomplete
Estimate: 4 days

This simply requires running through ES6 drafts, as they're published, and noting things that will need testing. The following list was produced by a read-through of the ES6 spec, so it likely misses systemic interactions that need testing.

Ordinary/exotic objects

Owner: unowned
Status: incomplete
Estimate: 2 days

• need to run through 8.4/8.5 with a fine-toothed comb looking for all the various differences
• some differences should already be tested, because they were implicit (or explicit in different form) in ES5 and earlier

Template literals

Owner: unowned
Status: incomplete
Estimate: unknown; depends on their being implemented, and on the implementation complexities encountered, to know what needs the most test work

• the InputElementTemplateTail goal symbol has interesting interactions with ASI, that must be investigated
• template literals generally need to be implemented, and basic semantics tested
• testing for specific interesting characters in template literals
  • Unicode format-control characters within template literals
  • line terminators (Unicode and non-Unicode both)
• templates with zero or more substitution parts need particular testing

Binary/octal numeric literals

Owner: jwalden
Status: tested, but not in test262
Estimate: 1 day

• implemented and tested in bug 894026
• tests need to be ported to test262 format and upstreamed

Non-BMP Unicode escapes

Owner: unowned
Status: incomplete
Estimate: 1 day

• \u{}
• \u{HexDigits} is a syntax error if HexDigits > 1114111 (0x10FFFF)
• \u{} with a surrogate-pair number
• \u{} with other reserved and special code points (BOM, the reserved bits around U+FFE0 or so)

Identifier syntax changes

Owner: unowned
Status: incomplete
Estimate: 1 day

• identifier goal symbols seem to have changed some -- to use the Unicode properties ID_Start and ID_Continue (and also _/$/<ZWJ>/<ZWNJ>)

New keywords

Owner: unowned
Status: incomplete
Estimate: 1 day

• new keywords: class, continue, const, export, import, super (possibly all reserved before, tests needed for their new semantics, and places that still forbid them)
• yield contextual keyword (being worked on by wingo)
• we already have tests to which keyword-checking stuff can be added (although they're not in test262 format)

Regular expression syntax

Owner: unowned
Status: incomplete
Estimate: unknown, depends on complexity of syntax changes

• apparent changes here -- unclear at a glance what needs doing here

this

Owner: unowned
Status: incomplete
Estimate: 1 day, appears relatively short

• 8.2.4.3 GetThisValue(V): how is this used, where, maybe needs tests (if not just some sort of refactoring of this keyword handling, not sure yet)

Property descriptors

Owner: unowned
Status: incomplete
Estimate: unknown

• the Property Descriptor type now has an optional [[Origin]] field -- possibly/likely used for property descriptors returned from proxy hooks -- needs testing
• implementing this has interactions with the ongoing MOP refactoring work; JSPropertyDescriptor and/or PropDesc need changes for this

New [[HasOwnProperty]] hook

Owner: jwalden
Status: incomplete
Estimate: unknown

• previously this was implicitly derived from [[GetOwnProperty]]
• implementing this has interactions with MOP refactoring
• jwalden has partial patch for that refactoring, bug 826587
• jwalden also has partial patch for the has-own-property hook, bug 862848 and/or bug 905168

New inheritance hooks

Owner: jwalden
Status: incomplete
Estimate: 2 days, plus implementation time

• needed for prototype-setting and getting to work with proxies
• we have this, to some extent, already

Enumeration changes

Owner: unowned
Status: incomplete
Estimate: 1 week

• new [[Enumerate]] and [[OwnPropertyKeys]] hooks with well-defined semantics for for(var i in o) and Object.getOwnPropertyNames/keys -- visible in proxies
• enumeration affects practically everything, so a bit of work necessary here

Function internal properties

Owner: unowned
Status: incomplete
Estimate: unknown

• functions (ordinary function objects, 8.3.15.1) have a bunch more internal data properties to track -- affecting semantics in various ways; realms are perhaps the most important one (corresponding to the multiple-globals problem in web browsers)
• |new| on a function whose [[Prototype]] has been changed to null/undefined -- semantics of this are still partially up in the air
• there seems to be a requirement that (function() { "use strict"; }).caller === null per 8.3.15.3 which might be a change from ES5 -- needs investigation, testing, possible bug-reporting

[[BuiltinBrand]]

Owner: unowned
Status: incomplete
Estimate: unknown

• used for exact-class checks from ES5 and before
• likely affects every place that currently does exact-class checks (or exact-class-modulo-other-globals checks)
• most recent ES6 draft doesn't discuss this anywhere except in overview documentation: highly under-specified right now

String objects

Owner: unowned
Status: incomplete
Estimate: unknown

• String exotic objects: new internal ops defined for them, may require more testing for out-of-range -- also for negative indexes, which in the current spec seem to be passed through abs() (!, may be a bug)

Symbol exotic objects

Owner: unowned
Status: incomplete
Estimate: unknown

• spec still in flux
• specific testing needed for "8.1.6.4 Well-Known Symbols and Intrinsics": are these directly observable, or only indirectly through spec algorithms' uses of them?
• all the hooks in 8.4.4 must be exercised, probably most simply through the Reflect.* methods that were (are?) mooted at one time for ES6

Integer Indexed exotic objects

Owner: unowned
Status: incomplete
Estimate: unknown

• looks like a catch-all for typed array views, approximately

The super keyword

Owner: unowned
Status: incomplete
Estimate: unknown, semantics not well-understood yet

• used to implement "Super References (8.2.4)" -- something to do with calling superclass methods, and similar things
• will need extensive testing with all our various function/generator syntaxes
• will need testing with various methods of calling functions

BinaryData proposal

Owner: unowned
Status: incomplete
Estimate: unknown

• implemented in bug 578700, ongoing bugfixing/fuzz-testing work before it'll be fully enabled
  • data block type
• need to loop in nsm on this

Scriptable proxies

Owner: unowned
Status: incomplete
Estimate: unknown

• possibly done a bit by ejpbruel?

Other things

• ToPrimitive is now explicitly hookable in objects (doesn't use toString/valueOf properties), using a well-known symbol/intrinsic

Issues

Importing third-party tests into our tree, that we don't necessarily pass (particularly as tests for new features are introduced), requires a mechanism for marking third-party tests as failing. Ideally, this must be possible without editing the imported tests, or at least with as few edits to the imported tests as possible. It's not clear what this mechanism will look like, but it's critical that we have it.

Risks

test262's test format may be inadequate to test all behavior required by ES6. Tests that must run in global code, for example, may not necessarily fit into the test262 format. (This is purely my recollection from having looked into this a few years ago -- things may have changed since.) Tests which require executing multiple JS scripts (that is, through multiple <script> elements in the browser) also may pose issues. Other instances may arise over time. Dealing with these harness limitations will require adding new functionality to the harness, and may delay properly testing some new features.

ES6 is a moving target: specified features may be removed, new features may be added, changes may be made at any time. Keeping tests atop the latest spec revisions will be an ongoing challenge until ES6 is finalized (currently targeted at the end of 2014).

The test262 suite is quite large. Simply adding all test262 tests to the "J" test runs on Tinderbox will make "J" builds take a lot longer to run. It may be necessary to split test262 into its own set of builds, so that test262 can be run in parallel with the existing "J" tests. (The goal is to minimize changeset-push-to-results latency. Historically this has been accomplished mostly by running tests in parallel.) It currently appears we can import a substantial number of tests into the existing "J" builds, without unduly lengthening the time it takes to run them, so this challenge doesn't need to be immediately faced to make progress toward running test262 tests.

Resources

• test262 repository (tests, harnesses, etc.)
• test262 test format
• etc.
• ...