IDN Display Algorithm - Revision history

Gerv: More explanation

2017-04-17T18:30:56Z

More explanation

← Older revision		Revision as of 18:30, 17 April 2017
Line 1:		Line 1:
	This page explains how Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.		This page explains how Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode (i.e. normal) or Punycode (i.e. gobbledigook) form.

	Implementing this plan was covered by {{bug\|722299}}.		Implementing this plan was covered by {{bug\|722299}}.
Line 11:		Line 11:
	===Previous Algorithm===		===Previous Algorithm===

	Our previous algorithm was to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We checked the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator had to apply directly (they cannot be nominated by another person), and on several occasions we required policy updates or implementation as a condition of getting in.		We updated how Firefox dealt with this in 2012. Our previous algorithm was to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We checked the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator had to apply directly (they cannot be nominated by another person), and on several occasions we required policy updates or implementation as a condition of getting in.

	We also had a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters. This still exists.		We also had a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters. This still exists.

Gerv: Update to show this is now what we do

2017-04-17T13:46:33Z

Update to show this is now what we do

← Older revision		Revision as of 13:46, 17 April 2017
Line 1:		Line 1:
	This page explains ~~the plan for changing the mechanism by which~~ Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.		This page explains how Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.

	Implementing this plan is covered by {{bug\|722299}}.		Implementing this plan was covered by {{bug\|722299}}.

	==Background==		==Background==
Line 9:		Line 9:
	If we just display any possible IDN label, we open ourselves up to [http://en.wikipedia.org/wiki/IDN_homograph_attack IDN homograph attacks], where one identical-looking domain can spoof another. So we have to have some mechanism to decide which ones to display and which ones to not display, which does not involve comparing the domain in question against every other single domain which exists (which is impossible).		If we just display any possible IDN label, we open ourselves up to [http://en.wikipedia.org/wiki/IDN_homograph_attack IDN homograph attacks], where one identical-looking domain can spoof another. So we have to have some mechanism to decide which ones to display and which ones to not display, which does not involve comparing the domain in question against every other single domain which exists (which is impossible).

	===~~Current~~ Algorithm===		===Previous Algorithm===

	Our ~~current~~ algorithm is to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We ~~check~~ the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator ~~must~~ apply directly (they cannot be nominated by another person), and on several occasions we ~~have~~ required policy updates or implementation as a condition of getting in.		Our previous algorithm was to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We checked the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator had to apply directly (they cannot be nominated by another person), and on several occasions we required policy updates or implementation as a condition of getting in.

	We also ~~have~~ a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters.		We also had a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters. This still exists.

	===~~Need For Change~~===		===Why We Changed===

	~~This~~ strategy ~~provides~~ pretty good user protection, and it ~~provides~~ consistency -		The old strategy provided pretty good user protection, and it provided consistency -
	every Firefox everywhere works the same. However, it ~~does mean~~		every Firefox everywhere works the same. However, it meant
	that IDNs do not work at all in many TLDs, because the registry (for whatever reason)		that IDNs did not work at all in many TLDs, because the registry (for whatever reason)
	~~has~~ not applied for inclusion, or because we ~~do not~~ think they ~~have~~ sufficiently		had not applied for inclusion, or because we didn't think they had sufficiently
	strong protections in place. In addition, ICANN is about to open		strong protections in place. In addition, ICANN was about to open
	a [http://newgtlds.icann.org/en/program-status/application-results/strings-1200utc-13jun12-en large number of new TLDs]. So either maintaining a whitelist is going to become		a [http://newgtlds.icann.org/en/program-status/application-results/strings-1200utc-13jun12-en large number of new TLDs]. So either maintaining a whitelist was going to become
	burdensome, or the list ~~will~~ become wildly out of date and we ~~will~~ not		burdensome, or the list was going to become wildly out of date and we would not
	be serving our users.		be serving our users.

	==~~=Other Browsers=~~==		==The New Idea==

	~~The Chromium IDN page has a [http://www.chromium.org/developers/design-documents/idn-in-google-chrome good summary]~~		Instead, we now augment our whitelist with something based on ascertaining whether all the characters in a label
	~~of the policies of Chrome/Chromium and the other browsers. Unfortunately~~, ~~no consensus has emerged on how to do~~
	~~this. Those other mechanisms were considered, but many of them depend on the configuration of the user's~~
	~~computer (e.g. installed languages), and this does not give site owners any confidence that their IDN will be correctly displayed for all their visitors (and no way of telling if it's not).~~

	~~==Proposal==~~

	~~The plan is to~~ augment our whitelist with something based on ascertaining whether all the characters in a label
	all come from the same script, or are from one of a limited and defined number of allowable combinations. The		all come from the same script, or are from one of a limited and defined number of allowable combinations. The
	hope is that any intra-script near-homographs will be recognisable to people who understand that script.		hope is that any intra-script near-homographs will be recognisable to people who understand that script.

	We ~~will~~ retain the whitelist as well, because a) removing it might break		We retain the whitelist as well, because a) removing it might break
	some domains which worked previously, and b) if a registry submits a		some domains which worked previously, and b) if a registry submits a
	good policy, we have the ability to give them more freedom than the default restrictions do.		good policy, we have the ability to give them more freedom than the default restrictions do.
	So an IDN ~~would be~~ shown as Unicode if the TLD was on the whitelist or, if not, if it		So an IDN is shown as Unicode if the TLD was on the whitelist or, if not, if it
	met the criteria above.		met the criteria above.

	==Algorithm==		==Algorithm==

	If a TLD is in the whitelist, we ~~will~~ unconditionally display Unicode. If it is not, the following		If a TLD is in the whitelist, we unconditionally display Unicode. If it is not, the following
	algorithm ~~will apply~~.		algorithm applies.

	[http://www.unicode.org/reports/tr39/#Restriction_Level_Detection Unicode Technical Report 39]		[http://www.unicode.org/reports/tr39/#Restriction_Level_Detection Unicode Technical Report 39]
Line 85:		Line 78:
	===Possible Issues and Open Questions===		===Possible Issues and Open Questions===

	The following issues are open, but ~~should~~ not block initial implementation.		The following issues are still open, but were not considered important enough to block initial implementation.

	Further suggestions from TR #39:		Further suggestions from TR #39:
Line 100:		Line 93:
	===Downsides===		===Downsides===

	This system ~~would permit~~ whole-script confusables		This system permits whole-script confusables
	(All-Latin "scope.tld" vs all-Cyrillic "ѕсоре.tld"). However, so do the		(All-Latin "scope.tld" vs all-Cyrillic "ѕсоре.tld"). However, so do the
	solutions of the other browsers, and it has not proved to be a		solutions of the other browsers, and it has not proved to be a
	significant problem so far. If there is a problem, every browser is equally affected.		significant problem so far. If there is a problem, every browser is equally affected.

	~~If problems arose in the future (e.g. whole-script, or homographs between a particular~~		Our response to this issue is that in the end, it is up to registries
	~~single script and Latin), our~~ response ~~would be~~ that in the end, it is up to registries
	to make sure that their customers		to make sure that their customers
	cannot rip each other off. Browsers can put some technical restrictions in place,		cannot rip each other off. Browsers can put some technical restrictions in place,
Line 113:		Line 105:
	only people in a position to implement the proper checking here. For our part,		only people in a position to implement the proper checking here. For our part,
	we want to make sure we don't treat non-Latin scripts as second-class citizens.		we want to make sure we don't treat non-Latin scripts as second-class citizens.

	~~==Transition==~~

	~~In between adopting this plan and shipping a Firefox with~~
	~~the restrictions implemented, we will admit into the whitelist any~~
	~~TLD whose anti-spoofing policies <i>at registration time</i> were at least as strong as~~
	~~those outlined above.~~

Ckoehler: Reverted edits by Karirkerja (talk) to last revision by Gerv

2014-08-13T00:53:38Z

Reverted edits by Karirkerja (talk) to last revision by Gerv

← Older revision		Revision as of 00:53, 13 August 2014
Line 119:		Line 119:
	the restrictions implemented, we will admit into the whitelist any		the restrictions implemented, we will admit into the whitelist any
	TLD whose anti-spoofing policies <i>at registration time</i> were at least as strong as		TLD whose anti-spoofing policies <i>at registration time</i> were at least as strong as
	those outlined above ~~[http://www~~.~~karirwanita.com a]~~		those outlined above.

Karirkerja: /* Transition */

2014-08-13T00:11:20Z

Transition

← Older revision		Revision as of 00:11, 13 August 2014
Line 119:		Line 119:
	the restrictions implemented, we will admit into the whitelist any		the restrictions implemented, we will admit into the whitelist any
	TLD whose anti-spoofing policies <i>at registration time</i> were at least as strong as		TLD whose anti-spoofing policies <i>at registration time</i> were at least as strong as
	those outlined above.		those outlined above [http://www.karirwanita.com a]

Gerv at 16:32, 18 March 2014

2014-03-18T16:32:16Z

← Older revision		Revision as of 16:32, 18 March 2014
Line 80:		Line 80:

	* Display as Punycode labels which use more than one numbering system		* Display as Punycode labels which use more than one numbering system
	* Display as Punycode labels which contain both simplified-only and traditional-only Chinese characters		* <strike>Display as Punycode labels which contain both simplified-only and traditional-only Chinese characters</strike> -- {{bug\|857481}}
	* Display as Punycode labels which have sequences of the same nonspacing mark		* Display as Punycode labels which have sequences of the same nonspacing mark

Gerv: /* Algorithm */

2013-02-22T14:53:55Z

Algorithm

← Older revision		Revision as of 14:53, 22 February 2013
Line 74:		Line 74:
	to provide data about characters which are only used in a few (but more than one) script.		to provide data about characters which are only used in a few (but more than one) script.
	The sample code in #39 should be updated to include that, so handling such cases."		The sample code in #39 should be updated to include that, so handling such cases."
	This data is now available, but not yet in the Firefox platform. In the mean time, Common and Inherited		This data is now available, but not yet in the Firefox platform ({{bug\|844106}}). In the mean time, Common and Inherited
	characters are permitted without restriction.		characters are permitted without restriction.

Gerv at 14:47, 22 February 2013

2013-02-22T14:47:53Z

Gerv: /* Algorithm */

2013-02-22T14:43:41Z

Algorithm

← Older revision		Revision as of 14:43, 22 February 2013
Line 50:		Line 50:
	algorithm will apply.		algorithm will apply.

	[http://www.unicode.org/reports/~~tr36~~/~~proposed.html~~#~~Security_Levels_and_Alerts~~ Unicode Technical Report 36],		[http://www.unicode.org/reports/tr39/#Restriction_Level_Detection Unicode Technical Report 39]
	defines a "Moderately Restrictive" profile. It says the following (with edits for clarity, after discussion with the author):		defines a "Moderately Restrictive" profile. It says the following (with edits for clarity, after discussion with the author):

Line 110:		Line 110:
	a level playing field for non-Latin scripts on the web. The registries are the		a level playing field for non-Latin scripts on the web. The registries are the
	only people in a position to implement the proper checking here. For our part,		only people in a position to implement the proper checking here. For our part,
	we want to make sure we don't treat non-Latin scripts as second-class citizens.		we want to make sure we don't treat non-Latin scripts as second-class citizens.

	==Transition==		==Transition==

Gerv at 09:35, 29 June 2012

2012-06-29T09:35:45Z

← Older revision		Revision as of 09:35, 29 June 2012
Line 1:		Line 1:
	This page explains the plan for changing the mechanism by which Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.		This page explains the plan for changing the mechanism by which Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.

			Implementing this plan is covered by {{bug\|722299}}.

	==Background==		==Background==
Line 11:		Line 13:
	Our current algorithm is to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We check the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator must apply directly (they cannot be nominated by another person), and on several occasions we have required policy updates or implementation as a condition of getting in.		Our current algorithm is to display as Unicode all IDN labels within TLDs on our [http://www.mozilla.org/projects/security/tld-idn-policy-list.html whitelist], and display as Punycode otherwise. We check the anti-spoofing policies of a registry before adding their TLD to the whitelist. The TLD operator must apply directly (they cannot be nominated by another person), and on several occasions we have required policy updates or implementation as a condition of getting in.

	We also have a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters. ~~(XXX Do we need to update this to remove some of those, like ZWJ/ZWNJ, for IDNA2008?)~~		We also have a character blacklist - characters we will never display under any circumstances. This includes those which could be used to spoof the separators "/" and ".", and invisible characters.

	===Need For Change===		===Need For Change===
Line 49:		Line 51:

	[http://www.unicode.org/reports/tr36/proposed.html#Security_Levels_and_Alerts Unicode Technical Report 36],		[http://www.unicode.org/reports/tr36/proposed.html#Security_Levels_and_Alerts Unicode Technical Report 36],
	defines a "Moderately Restrictive" profile. It says the following (with edits for clarity):		defines a "Moderately Restrictive" profile. It says the following (with edits for clarity, after discussion with the author):

	<blockquote>		<blockquote>
Line 73:		Line 75:
	characters are permitted without restriction.		characters are permitted without restriction.

	Additional checks:		Additional checks, as suggested by TR #39 section 5:

	* Display as Punycode labels which use more than one numbering system ~~(we would need a list of numbering systems in Unicode)~~		* Display as Punycode labels which use more than one numbering system
	* Display as Punycode labels which contain both simplified-only and traditional-only Chinese characters~~, using the Unihan data in the Unicode Character Database (should be < 16k of data for a simple binary test)~~		* Display as Punycode labels which contain both simplified-only and traditional-only Chinese characters
	* Display as Punycode labels which have sequences of the same nonspacing mark ~~(we would need a list of, or the name of a class containing, all such marks)~~		* Display as Punycode labels which have sequences of the same nonspacing mark

	===Possible Issues and Open Questions===		===Possible Issues and Open Questions===
Line 83:		Line 85:
	The following issues are open, but should not block initial implementation.		The following issues are open, but should not block initial implementation.

	~~Suggestion~~ from TR#39:		Further suggestions from TR #39:

	* Check to see that all the characters are in the sets of exemplar characters for at least one language in the Unicode Common Locale Data Repository. [XXX What does this mean? -- Gerv]		* Check to see that all the characters are in the sets of exemplar characters for at least one language in the Unicode Common Locale Data Repository. [XXX What does this mean? -- Gerv]
Line 89:		Line 91:
	Also:		Also:

	* Should we document our character hard-blacklist as part of this exercise? It's already visible in the prefs. Are any characters in it legal in IDNA2008 anyway?		* Should we document our character hard-blacklist as part of this exercise? It's already visible in the prefs. Are any characters in it legal in IDNA2008 anyway? We should review it to make sure we aren't disallowing characters now allowed by IDNA 2008.
	* Do we want to allow the user to choose between multiple "restriction levels", or have a hidden pref? There are significant downsides to allowing this...		* Do we want to allow the user to choose between multiple "restriction levels", or have a hidden pref? There are significant downsides to allowing this...
	* Do we ever want to display errors other than just by using Punycode? I suggest not...		* Do we ever want to display errors other than just by using Punycode? I suggest not...

Gerv at 16:28, 4 February 2012

2012-02-04T16:28:01Z

← Older revision		Revision as of 16:28, 4 February 2012
Line 1:		Line 1:
	This page explains the plan for changing the mechanism by which Firefox decides whether to display a given IDN ~~domain~~ label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.		This page explains the plan for changing the mechanism by which Firefox decides whether to display a given IDN label (a domain name is made up of one or more labels, separated by dots) in its Unicode or Punycode form.

	==Background==		==Background==
Line 5:		Line 5:
	===The Problem===		===The Problem===

	If we just display any possible IDN ~~domain~~ label, we open ourselves up to [http://en.wikipedia.org/wiki/IDN_homograph_attack IDN homograph attacks], where one identical-looking domain can spoof another. So we have to have some mechanism to decide which ones to display and which ones to not display, which does not involve comparing the domain in question against every other single domain which exists (which is impossible).		If we just display any possible IDN label, we open ourselves up to [http://en.wikipedia.org/wiki/IDN_homograph_attack IDN homograph attacks], where one identical-looking domain can spoof another. So we have to have some mechanism to decide which ones to display and which ones to not display, which does not involve comparing the domain in question against every other single domain which exists (which is impossible).

	===Current Algorithm===		===Current Algorithm===
Line 29:		Line 29:
	of the policies of Chrome/Chromium and the other browsers. Unfortunately, no consensus has emerged on how to do		of the policies of Chrome/Chromium and the other browsers. Unfortunately, no consensus has emerged on how to do
	this. Those other mechanisms were considered, but many of them depend on the configuration of the user's		this. Those other mechanisms were considered, but many of them depend on the configuration of the user's
	computer (e.g. installed languages), and this does not give site owners any confidence that their IDN ~~domain~~		computer (e.g. installed languages), and this does not give site owners any confidence that their IDN will be correctly displayed for all their visitors (and no way of telling if it's not).
	~~name~~ will be correctly displayed for all their visitors (and no way of telling if it's not).

	==Proposal==		==Proposal==
Line 41:		Line 40:
	some domains which worked previously, and b) if a registry submits a		some domains which worked previously, and b) if a registry submits a
	good policy, we have the ability to give them more freedom than the default restrictions do.		good policy, we have the ability to give them more freedom than the default restrictions do.
	So an IDN ~~domain~~ would be shown as Unicode if the TLD was on the whitelist or, if not, if it		So an IDN would be shown as Unicode if the TLD was on the whitelist or, if not, if it
	met the criteria above.		met the criteria above.

Line 68:		Line 67:
	a definition for how we detect whether a string is "single script". Some Common or Inherited characters		a definition for how we detect whether a string is "single script". Some Common or Inherited characters
	are only used in a small number (but more than one) script. Mark Davis writes:		are only used in a small number (but more than one) script. Mark Davis writes:
	"The Unicode Consortium in U6.1 (due out soon) is adding the property Script_Extensions,		"The Unicode Consortium in U6.1 (due out soon) is adding the property [http://unicode.org/Public/6.1.0/ucd/ScriptExtensions.txt Script_Extensions],
	to provide data about characters which are only used in a few (but more than one) script.		to provide data about characters which are only used in a few (but more than one) script.
	The sample code in #39 should be updated to include that, so handling such cases." We should		The sample code in #39 should be updated to include that, so handling such cases." We should

← Older revision		Revision as of 14:47, 22 February 2013
Line 22:		Line 22:
	has not applied for inclusion, or because we do not think they have sufficiently		has not applied for inclusion, or because we do not think they have sufficiently
	strong protections in place. In addition, ICANN is about to open		strong protections in place. In addition, ICANN is about to open
	a [http://newgtlds.icann.org/ large number of new TLDs]. So either maintaining a whitelist is going to become		a [http://newgtlds.icann.org/en/program-status/application-results/strings-1200utc-13jun12-en large number of new TLDs]. So either maintaining a whitelist is going to become
	burdensome, or the list will become wildly out of date and we will not		burdensome, or the list will become wildly out of date and we will not
	be serving our users.		be serving our users.
Line 63:		Line 63:
	* Common + Inherited + Latin + Han + Bopomofo; or		* Common + Inherited + Latin + Han + Bopomofo; or
	* Common + Inherited + Latin + Han + Hangul; or		* Common + Inherited + Latin + Han + Hangul; or
	* Common + Inherited + Latin + any single other script except Cyrillic, Greek~~, or Cherokee~~		* Common + Inherited + Latin + any single other "Recommended" or "Aspirational" script except Cyrillic or Greek
	</blockquote>		</blockquote>

	[http://www.unicode.org/reports/tr39/#Mixed_Script_Detection Unicode Technical Report 39] gives		[http://www.unicode.org/reports/tr39/#Mixed_Script_Detection Unicode Technical Report 39] gives
	a definition for how we detect whether a string is "single script". Some Common or Inherited characters		a definition for how we detect whether a string is "single script".

			Some Common or Inherited characters
	are only used in a small number (but more than one) script. Mark Davis writes:		are only used in a small number (but more than one) script. Mark Davis writes:
	"The Unicode Consortium in U6.1 (due out soon) is adding the property [http://unicode.org/Public/6.1.0/ucd/ScriptExtensions.txt Script_Extensions],		"The Unicode Consortium in U6.1 (due out soon) is adding the property [http://unicode.org/Public/6.1.0/ucd/ScriptExtensions.txt Script_Extensions],
	to provide data about characters which are only used in a few (but more than one) script.		to provide data about characters which are only used in a few (but more than one) script.
	The sample code in #39 should be updated to include that, so handling such cases." ~~We should~~		The sample code in #39 should be updated to include that, so handling such cases."
	~~take this enhancement when the~~ data ~~becomes~~ available; in the mean time, Common and Inherited		This data is now available, but not yet in the Firefox platform. In the mean time, Common and Inherited
	characters are permitted without restriction.		characters are permitted without restriction.

	~~Additional~~ checks, as suggested by TR #39 ~~section~~ 5:		We also implement additional checks, as suggested by TR #39 sections 5.3 and 5.4:

	* Display as Punycode labels which use more than one numbering system		* Display as Punycode labels which use more than one numbering system