User:Waldo/Internationalization API: Difference between revisions

Every operation is performed in terms of locales, specified as [http://tools.ietf.org/html/bcp47#section-2.1 language tags]: <code>en-US</code>, <code>nan-Hant-TW</code>, <code>und</code>, and so on.  The main components of a language tag are the language and optionally a script, region, and variations that might exist within these.  An extension component may follow, permitting inclusion of extra structured data (usually to contextualize a use of the language tag).  Finally, an optional private-use component may include arbitrary data (this is for the use of webpages -- not for SpiderMonkey's use).  All components are alphanumeric and case-insensitive ASCII.  The components are joined into a language tag using hyphens; individual components can be distinguished by length and internal syntax (length, prefix, etc.).  The precise details of language tag structure are quite complex, and they include a list of irregular forms for legacy compatibility reasons.  See [http://tools.ietf.org/html/bcp47#section-2.1 BCP 47] for all the gory details.

Locale-sensitive date formatting in ES5 admits only a single implementation-defined format with a fixed set of components.  ECMA-402 enhances date formatting to allow the selection of components to be customized -- month, day, and year, for example..  The way in which these components will be displayed in the final format string is locale-dependent, as different locales write out dates in different ways.  (For example, a date might be formatted as "September 24, 2012" for en-US and as "24 Sept. 2012" for fr-FR.)  Moreover, various styles may be chosen for the components included in the format: "narrow", "short", "long", "2-digit", and "numeric".  (Exactly which of these styles are supported and what they look like is also implementation-dependent.)  These styles feed into the final computation of an appropriate pattern to use to generate a final string.

SpiderMonkey includes significant support for the Internationalization API.  The fundamental primitives used to implement the API are provided by an in-tree imported copy of ICU.  This is an optional component of a SpiderMonkey build; support may be turned on using the <code>--enable-intl-api</code> configuration option.  Most of SpiderMonkey's Internationalization code gets built even when the API is disabled, to prevent bitrot; ICU interfaces of note are stubbed out in this configuration with methods that do nothing but assert.  The most important differences in a SpiderMonkey build without Internationalization are that the <code>Intl</code> object isn't added to global objects and the legacy <code>toLocale*String</code> methods are implemented using SpiderMonkey's old <code>JSLocaleCallbacks</code> interface.  Features and capabilities of the Internationalization API itself are implemented in both C++ and in self-hosted JavaScript that accesses ICU functionality through various intrinsic functions in the self-hosting global.

Internationalization in various cases requires keeping around large data tables: to record the minor units of supported currency codes, to record language tag mappings, and so on. Some of this data lives in {{source|js/src/builtin/IntlData.js}} and is generated by {{source|js/src/builtin/make_intl_data.py}}.  This script downloads original (large) plaintext databases, parses them, and extracts in the proper format the data used by Internationalization.  Updating this static data &mdash; which should happen any time the underlying databases receive an update &mdash; should be as simple as rerunning the script. '''XXX Link to the mailing lists to track to learn when updates occur!'''

Self-hosted code calls into various intrinsics to access ICU functionality.  The full list of Internationalization intrinsics is (necessarily, at the moment -- this will probably change eventually) in {{source|js/src/vm/SelfHosting.cpp}}, but the intrinsics themselves are implemented in {{source|js/src/builtin/Intl.cpp}}.

Tests live in {{source|js/src/tests/test402}}, an unmodified import of the ECMA-402 test suite. The tests are run during the normal jstests/jsreftest suite. '''XXX Is this true yet?'''

Internationalization tests are treated as a third-party import. Contributions to them should go through '''XXX how to contribute'''.

As we may not have fully-correctly implemented ECMA-402 at any point, or a bug might be found before a test is committed, we require a mechanism to mark an ECMA-402 test as failing without requiring that marking be upstreamed. '''XXX what is this mechanism?'''

In builds with ECMA-402 support disabled, these tests are skipped. '''XXX how?'''

ECMA-402 currently exposes <code>Intl.Collator</code>, <code>Intl.DateTimeFormatter</code>, and <code>Intl.NumberFormatter</code> objects.  The spec also permits initializing an existing object as one of these, for a small wrinkle.  The fundamental ICU data structures providing the relevant functionality are <code>UCollator*</code>, <code>UNumberFormat*</code>, and <code>UDateFormat*</code>, opaque pointers all.  Instances are created using <code>u{col,num,date}_open</code>, passing in appropriate arguments.  For objects ''created'' by the constructor, the pointer is stored in a reserved slot as a private value.  For objects merely ''initialized'' by the constructor, the ICU data structures must be (inefficiently!) created anew every time.  (This difference should not be observable, except through performance-timing, because the only structures consulted to create the ICU structure are internal ones , operations on which aren't observable.)

Internals objects are objects with null <code><nowiki>[[Prototype]]</nowiki></code>, with properties corresponding to the other internal properties on the object, named naturally -- "calendar", "initializedDateTimeFormat", and so on (no brackets).  Accessing any internal property is simply a matter of doing <code>internals.calendar</code>: this is safe because, with the <code><nowiki>[[Prototype]]</nowiki></code> nulled out, property accesses can't touch any script-visible state.  Internal properties are added and set during the initialization process.  They are lazily consulted to construct an ICU structure when collation/formatting/etc. actually occurs in the <code>js::intl_CompareStrings</code>, <code>js::intl_FormatNumber</code>, and <code>js::intl_FormatDateTime</code> functions.  (Although not ''directly'' there, but rather in sub-methods called when the ICU structure isn't cached, or when the object was initialized as an <code>Intl</code> object but wasn't actually one -- see again the "inefficiently" bit above.)

 

 

 

 

ICU has major releases once or twice a year, and minor releases as needed. Releases are announced on the [https://lists.sourceforge.net/lists/listinfo/icu-announce icu-announce mailing list]. Each release includes the latest versions of the CLDR locale data, the IANA time zone database, and the ISO 4217 currency data, so it's generally worth it for Mozilla to update its copy each time. As of April 2013, upgrades are unfortunately blocked by [http://bugs.icu-project.org/trac/ticket/10043 ICU bug 10043].

Bugs in ICU should be reported into the [http://bugs.icu-project.org/trac/ ICU bug database]. Bug fixes can be [http://site.icu-project.org/processes/contribute contributed]; as of April 2013, one contribution is in progress.

The IANA language subtag registry is updated around 4 times a year. Releases are announced on the [https://mm.icann.org/mailman/listinfo/ietf-languages-announcements ietf-languages-announcements mailing list]. Updates are usually not urgent. Changes require updating ICU and, if language tag mappings are involved, the mapping tables in IntlData.js.

* Changes to time zone names (new names or renames) require updating the time zone data within ICU and at the same time the time zone name mappings in IntlData.js (once bug 837961 has been implemented).

==== Currency changes ====

The ISO 4217 currency code list is updated around twice a year. Releases are announced in a [http://www.currency-iso.org/en/home/amendments/newsletter.html newsletter]. Changes might be urgent, although politicians seem to understand the impact of changing currencies better than that of changing time zones. Changes require updating ICU and, if the minor unit value for a currency was/is different from 2, the currencyDigits table in Intl.js (IntlData.js once bug 843758 is fixed).

=== Other random details ===

....anything?...