ServerJS/Encodings

Rationale

For Streams, we need encodings support. There also should be a low-level API available for this.

There is some discussion on the mailing list (see <http://groups.google.com/group/serverjs/browse_thread/thread/6365b2a54615a134>) and here, there is a summary of these efforts.

Encoding Names

The encoding names should be among those supported by ICONV, which seem to be a superset of http://www.iana.org/assignments/character-sets.

The following encodings are required:

• US-ASCII
• UTF-8
• UTF-16
• ISO-8859-1

Encoding names must be case insensitive

API

OK, so probably this should be a module:

 var enc = require('encodings')

Simple methods

For convenience, there should be these easy methods for converting between encodings:

string = enc.convertToString(sourceEncoding, byteStringOrArray)

Converts a ByteString or a ByteArray to a Javascript string.

byteString = enc.convertFromString(targetEncoding, string)

Converts a Javascript string to a ByteString.

byteString = enc.convert(sourceEncoding, targetEncoding, byteStringOrArray)

Converts a ByteString or a ByteArray to a ByteString.

Checking for available encodings

enc.supports(encodingName)

Checks if encodingName is supported and return true if so, false otherwise.

enc.listEncodings([encodingCheckerFunction or regex])

encodingCheckerFunction takes the encoding name as a parameter and returns true-ish if the encoding should be listed. Regexes should also be supported. If the parameter is missing, returns all supported encodings.

Class: Converter

There also should be a class enc.Converter for more advanced conversion (transcoding).

[Constructor] Converter(from, to)

Where from and to are the encoding names.

[Method] push(byteStringOrArray[, outputByteArray])

Convert input from a ByteString or ByteArray. Those parts of byteStringOrArray that could not be converted (for multi-byte encodings) are stored in a buffer. If outputByteArray is passed, the results are appended to outputByteArray.

If outputByteArray was passed, returns outputByteArray, otherwise returns (as a ByteString) as much output as could be converted.

[Method] close()

Close the stream. Throws an exception if there was a conversion error (specifically, a partial multibyte character).

Returns nothing and takes no parameters.

TODO: Which exception to throw on error?

Example:

 Converter = require('encodings').Converter
 converter = new Converter('iso-8859-1', 'utf-32')
 output = converter.push(input) // input is a ByteString, and output too
 converter.close()

Another example:

 converter = new Converter('utf-32', 'utf-8')
 output = new ByteArray()
 while (input = readSomeByteFromSomewhere()) {
         converter.push(input, output)
 }
 converter.close()
 // output is the complete conversion of all the input chunks concatenated now

(See ServerJS/Encodings/OldClass for another API.)