Jsctypes/api

js-ctypes is a library for calling C/C++ functions from JavaScript without having to write or generate any C/C++ "glue code".

js-ctypes is already in mozilla-central, but the API is subject to change. This page contains design proposals for the eventual js-ctypes API.

Proposal 1

1. opening a library and declaring a function

Cu.import("ctypes"); // imports the global ctypes object

// searches the path and opens "libmylib.so" on linux,
// "libmylib.dylib" on mac, and "mylib.dll" on windows
let mylib = ctypes.open("mylib", ctypes.SEARCH);

// declares the C prototype int32_t myfunc(int32_t)
// Int32 implies ctypes.Int32, shortened for brevity
let myfunc = mylib.declare("myfunc", DEFAULT_ABI, Int32(), Int32());

let ret = myfunc(2); // calls myfunc

Note that for simple types (integers and strings), we will autoconvert the argument at call time - there's no need to pass in an Int32 object. The consumer should never need to instantiate such an object explicitly, unless they're using it to back a pointer - in which case we require explicit, strong typing. See later for examples.

2. declaring and passing a simple type (by object)

let myfunc = mylib.declare("myfunc", DEFAULT, Int32, Int32);
let i = new Int32(); // instantiates an Int32 object with default value 0
let ret = myfunc(i);

An Int32 object, like all other type objects in ctypes, can be used for type specification when passed as an object, as above. declare() can look at the prototype JSObject* of its argument, and use this as a canonical JSObject representing the type, a pointer to which can be used for simple type equality comparisons. (This will work for user-defined types such as structs also - see later - though for pointer types we need to dig down to the underlying type.)

Int32() can have two modes depending on whether JS_IsConstructing(cx) is JS_TRUE ("new Int32()") or JS_FALSE ("Int32()"). Used as a function, we could perform a type conversion with range checking, for instance:

let n = Int32(4); // JSVAL_IS_INT(n) == JS_TRUE
n = Int32(4e16); // RangeError - out of bounds
n = Int32.max; // 2^31 - 1
// etc

For the new constructor, the resulting object stores three pieces of information internally in reserved slots. |new Int32()| creates a JSObject which allocates sizeof(int32_t) and stores that pointer in a private slot. It also stores its type, as a JSObject* pointing to the canonical Int32 prototype, and can store a parent JSObject* in case it refers to an Int32 that happens to be part of another object. Thus the slot layout of i above would be

i object:
  slot 1 (parent): JSObject* -> NULL (no parent object)
  slot 2 (type) : JSObject* -> Int32 prototype
  slot 3 (value) : void* -> binary blob from malloc(sizeof(int32_t))

Do we need to provide an explicit set() method, to allow for efficient modification? For instance,

i.set(5); // cheaper than i = new Int32(5);

3. declaring and passing a pointer

// C prototype: int32_t myfunc(int32_t* p)
let myfunc = mylib.declare("myfunc", DEFAULT_ABI, Int32, Pointer(Int32));
let p = new Pointer(new Int32()); // instantiates an int and a pointer
let ret = myfunc(p); // the int is an outparam
let i = p.contents(); // i = *p (by reference)
let a = p.address(); // 0x...

// same thing, but with a named integer
let i = new Int32();
let p = new Pointer(i);
let ret = myfunc(p); // modifies i

// same thing, but with a pointer temporary
let i = new Int32();
let ret = myfunc(new Pointer(i)); // modifies i

// other examples
let q = new Pointer(); // instantiate a null pointer to a void type
q = new Pointer(5); // TypeError - require a ctypes type

Internally, a pointer requires a backing object (unless it's a null pointer). In the examples, the Pointer JSObject holds a reference to the Int32 JSObject for rooting purposes, and is laid out similarly to an Int32 object:

p object:
  slot 1 (parent): JSObject* -> Int32 backing object
  slot 2 (type) : JSObject* -> Pointer prototype
  slot 3 (value) : void* -> pointer to binary int32_t blob inside backing object

4. declaring a pointer to opaque struct

const FILE = ctypes.Struct(); // creates a Struct() type with no allocated binary storage, and no fields to access
let fopen = mylib.declare("fopen", DEFAULT_ABI, Pointer(FILE), String);
let file = fopen("foo"); // creates a new Pointer() object
file.contents(); // will throw - type is unknown
file.address(); // ok

5. declaring a struct

// C prototype: struct s_t { int32_t a; int64_t b };
const s_t = Struct([{ a: Int32 }, { b: Int64 }]);
let myfunc = mylib.declare("myfunc", DEFAULT_ABI, Int32, s_t);

let s = new s_t(10, 20);

This creates an s_t object which allocates binary space for both fields, creates getters and setters to access the binary fields via their offset, assigns the values 10 and 20 to the fields, and whose prototype is s_t:

s object:
  slot 1 (parent): JSObject* -> NULL
  slot 2 (type) : JSObject* -> s_t prototype
  slot 3 (value) : void* -> pointer to binary blob from malloc()
  slot 4 (fields): array of data for each field:
    { JSObject* parent; JSObject* type; ptrdiff_t offset; }

The array of field information allows each field to be dependent on another JSObject (only for the case where the field is a pointer), have an associated type, and have an offset into the binary blob for ease of access.

let c = s.b; // invokes the getter for |b| to create an Int64 object like so:

c object:
  slot 1 (parent): JSObject* -> s backing object
  slot 2 (type) : JSObject* -> Int64 prototype
  slot 3 (value) : void* -> pointer to binary int64_t blob inside backing object

let i = myfunc(s); // checks the type of s by JSObject* prototype equality

6. pointers to struct fields

let p = new Pointer(s.b);

Once the Int64 representing s.b is constructed, the Pointer object references it directly:

p object:
  slot 1 (parent): JSObject* -> Int64 backing object (which, in turn, is backed by s)
  slot 2 (type) : JSObject* -> Pointer prototype
  slot 3 (value) : void* -> pointer to binary int64_t blob inside backing object

7. nested structs

const u_t = Struct([{ x: Int64 }, { y: s_t }]);
let u = new u_t(5e4, s); // copies data from s into u.y - no references

let u_field = u.y; // creates an s_t object that points directly to the offset of u.y within u.

const v_t = Struct([{ x: Pointer(s_t) }, { y: Pointer(s_t) }]);
let v = new v_t(new Pointer(s), new Pointer(s));

In this case, the fields array will each have their respective Pointer as the parent object, and both will point to the s binary blob.

Proposal 2

Types

A type maps JS values to C/C++ values and vice versa. They're used when declaring functions. They can also be used to create and populate C/C++ data structures entirely from JS.

The types provided by ctypes

ctypes provides the following builtin types:

ctypes.int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, float32_t, float64_t - Primitive types that behave the same way on all platforms (with the usual caveat that every platform has slightly different floating-point behavior, in corner cases, and there's nothing we can realistically do about it).

ctypes.bool, short, unsigned_short, int, unsigned, unsigned_int, long, unsigned_long, float, double - Types that behave like the corresponding C types. Some or all of these might be aliases for the primitive types listed above. As in C, unsigned is always an alias for unsigned_int.

(TBD: Do we want char types? how should they behave?)

ctypes.string, ustring - String types. The C/C++ type for ctypes.string is const char *. C/C++ values of this type must be either null or pointers to null-terminated strings. ctypes.ustring is the same, but for const jschar *; that is, the code units of the string are uint16_t.

<ocde>ctypes.void_t - The special C type void. This can be used as a return value type. (void is a keyword in JavaScript.)

Starting from those builtin types, ctypes can create additional types:

ctypes.PointerType - Constructor for pointer types. new ctypes.PointerType(T), where T is a type, creates the type "pointer to T". Also new ctypes.PointerType(name), where name is a string, creates an opaque pointer type.

ctypes.ArrayType - Constructor for array types. new ctypes.ArrayType(T, n), where T is a type and n is a nonnegative integer, creates the type T[n]. If n is omitted, this creates an array type with unspecified length.

ctypes.StructType - Constructor for struct types. new ctypes.StructType(name, fields) creates a new struct with the given name" and fields. fields is an array of field descriptors. If fields contains a field descriptor with a type t such that t.size is undefined, a TypeError is thrown.

(TODO: Finish specifying field descriptors.)

Examples:

const DWORD = ctypes.uint32_t;
const HANDLE = new ctypes.PointerType("HANDLE");

const FILE = new ctypes.PointerType("FILE *");
const IOBuf = new ctypes.ArrayType(ctypes.uint8_t, 4096);

const struct_tm = new ctypes.Struct('tm', [[ctypes.int, 'tm_sec'], ...]);

Properties of types

All the fields described here are read-only.

All types have these properties:

t.size - The C/C++ sizeof the type, in bytes.

If t is an array type with unspecified length, t.size is undefined.

ctypes.void_t.size is undefined.

t.name - A string, the type's name. It's intended that in ordinary use, this will be a C/C++ type expression, but it's not really meant to be machine-readable in all cases.

For primitive types this is just the name of the corresponding C/C++ type, e.g. ctypes.int32_t.name == "int32_t" and ctypes.void_t == "void". But some of the builtin types are aliases for other types, so it might be that ctypes.unsigned_long.name == "uint32_t" (or something else). (Open issue: Is that too astonishing?)

For struct types and opaque pointer types, this is simply the string that was passed to the constructor; e.g. FILE.name == "FILE *" and struct_tm.name == "tm". For other pointer types and array types this should try to generate valid C/C++ type expressions, which isn't exactly trivial.

(Open issue: This conflicts with the usual meaning of .name for functions, and types are functions.)

Pointer types also have:

t.targetType - The pointed-to type, or null if t is an opaque pointer type.

Struct types also have:

t.fields - A sealed array of field descriptors, details TBD.

Array types also have:

t.elementType - The type of the elements of an array of this type. E.g. IOBuf.elementType === ctypes.uint8_t.

t.length - The number of elements, a nonnegative integer.

Minutiae:

The [[Class]] of a ctypes type is "Function". (Open issue: Subject to possible implementation difficulties.)

The .prototype properties of the type-constructors ctypes.{Pointer,Struct,Array,}Type and of each ctypes type are read-only and permanent.

Types have a hierarchy of prototype objects. The prototype of ctypes.Type.prototype is Function.prototype. The prototype of ctypes.{Array,Struct,Pointer}Type.prototype and of all the builtin types except for the string types is ctypes.Type.prototype. The prototype of an array type is ctypes.ArrayType.prototype. The prototype of a struct type is ctypes.StructType.prototype. The prototype of a string type or pointer type is ctypes.PointerType.prototype.

ctypes.Type.prototype.constructor === ctypes.Type

ctypes.ArrayType.prototype.constructor === ctypes.ArrayType

ctypes.StructType.prototype.constructor === ctypes.StructType

ctypes.PointerType.prototype.constructor === ctypes.PointerType

(TODO: Specify toString().)

Calling types

js-ctypes types are JavaScript constructors. That is, they are functions, and they can be called in various different ways. (js-ctypes buffers and references are to be described in later sections.)

new t or new t() - Without arguments, these allocate a new buffer of t.size bytes, populate it with zeroes, and return a new ctypes reference to the complete object in that buffer.

If t.size is undefined, this throws a TypeError.

t() - Return ConvertToJS(new t). (ConvertToJS is defined below. The result is that ctypes.bool() === false, for number types t() === 0, for string types t() === "", and for pointer types t() === null on platforms where the null pointer is all zeroes. For all other types the result is the same as new t.)

new t(val) - Convert val to type t according to the explicit conversion rules below, throwing a TypeError if the conversion is impossible. Create a new buffer and reference as above, populating the new buffer with the converted value instead of zeroing it out. (When val is a reference of type t, the behavior is like malloc followed by memcpy.)

As a special case, if t is an array type of unspecified length and val is a nonnegative integer, allocate a new buffer of size val * t.elementType.size. Populate it with zeroes. Return a reference to the new array.

t(val) - Return ConvertToJS(new t(val)).

The special type ctypes.void_t throws a TypeError if called with new, but ctypes.void_t() and ctypes.void_t(x) are allowed. Both return undefined.

References

js-ctypes references are like C++ references: they have a type, they refer to a C++ value, their address can be taken, and they can be assigned to. However, the JavaScript syntax is very different from the C++ syntax.

(TODO)

ref.assign(val) - Convert val to the type of ref using the implicit conversion rules. Store the converted value in the buffer location referred to by ref.

Struct references have getters and setters for each struct member:

cstruct.member - Let R be a reference to the struct member. Return ConvertToJS(R).

cstruct.member = value - The value is converted to the type of the member using the implicit conversion rules. The converted value is stored in the buffer.

Likewise, array references have getters and setters for each element. Arrays additionally have a length property.

Note that these getters and setters are only present for integers i in the range 0 ≤ i < carray.length. (Open issue: can we arrange to throw an exception if i is out of range?)

carray[i] - Let R be a reference to the element at index i. Return ConvertToJS(R).

carray[i] = val - Convert val to the type of the array element using the implicit conversion rules and store the result in the buffer.

carray.length - Read-only. The length of the array.

(TODO: Provide a way to get the type of a reference.)

(TODO: Figure out if the type of new FooArray(30) is FooArray or ArrayType(Foo, 30).)

(TODO: Possibly, a way to get a reference that acts like a view on a window of an array. E.g. carray.slice(start, stop). Then you could .assign one region of memory to another, effectively memcpy-ing.)

Minutiae:

The [[Class]] of a ctypes reference is "Reference".

The prototype of a reference is the same as its type's .prototype property.

(Implementation notes: A ctypes reference is a JSObject; it has a reserved slot that points to its type, a reserved slot that points to the backing buffer, and a pointer to the actual referenced location within the buffer. Since the data pointer might not be aligned to 2 bytes, PRIVATE_TO_JSVAL is insufficient; a custom JSClass.trace hook will be needed. A ctypes buffer is a separate JSObject that has a pointer to a malloc'd buffer where the C++ data is stored. It doesn't have a pointer to a ctypes type. It has a finalizer that frees the buffer; references that point into the buffer keep the buffer alive, thanks to the reserved slot.)

Pointers

js-ctypes pointers are very simple JavaScript objects that represent C/C++ pointers. Like C/C++ pointers, js-ctypes pointers represent a memory address. They may point to valid memory, but they may also point off the end of an array, to memory that has been freed, to uninitialized or unmapped memory, or to data of a different type.

Like C/C++ pointers, js-ctypes pointers never protect the data they point to from garbage collection.

It is hard to use (non-opaque) pointers safely, so js-ctypes is designed to support as many APIs as possible without requiring the use of pointers. For example, if a C/C++ function takes a parameter that is a pointer to a struct, you can just pass it a struct, and ctypes will quietly take its address. (The implicit conversion rules will handle this.)

These functions produce pointers:

ctypes.addressOf(ref) - Return a pointer to the object referenced by ref. If ref is not a ctypes reference, throw a TypeError.

(The rest of these strike me as targets of opportunity. Only certain unusual C APIs will need them.)

ctypes.addressOfField(ref, name) - Return a pointer to the named field of the struct referenced by ref. If ref is not a reference to a struct, or the struct does not have a field with the given name, throw a TypeError.

ctypes.addressOfElement(ref, i) - Return a pointer to element i of the array referenced by ref. If ref is not a reference to an array, or i is not a valid index into the array, throw a TypeError.

ctypes.castPointer(t, ptr) - Return a pointer of type t with the same bit-value as ptr. If t is not a pointer type or ptr is neither a pointer nor an integer, throw a TypeError.

ctypes.pointerAdd(ptr, nelements) - Like the C expression ptr + nelements. Return a pointer of the same type as ptr, adjusted by nelements * targetType.size bytes. If ptr is not a pointer or nelements is not an integer, throw a TypeError.

In js-ctypes, as in C/C++, pointers are totally unchecked. They may refer to valid memory or not. They may refer to memory that has been freed. So there is no guaranteed-safe way to dereference a pointer. However, if the application knows that the pointer is valid, it can access the pointed-to data using this function:

ctypes.pointerToUnsafeReference(ptr) - If ptr is not a pointer, or is a null pointer, throw a TypeError. Otherwise return a reference of type t pointing to the same location as ptr. The new reference is safe to use only as long as ptr is a valid pointer.

Pointers have the following method:

ptr.toString() - Return a string of the form "(type) 0xhexdigits" where type is the name of ptr's target type and hexdigits consists of lowercase hexidecimal digits and is exactly 8 characters on 32-bit platforms and 16 characters on 64-bit platforms.

Minutiae: The [[Class]] of a pointer is "Pointer". ctypes.Pointer is a function that takes two arguments, a pointer type and a pointer or number, and returns a new pointer object. Its .prototype method is read-only. ctypes.Pointer.prototype is a pointer. Its value is NULL. ctypes.Pointer.prototype.constructor === ctypes.Pointer. The prototype of ctypes.Pointer.prototype is Object.prototype. The prototype of every other pointer is ctypes.Pointer.prototype.

Conversions

The implicit conversion rules are applied whenever a JavaScript value of any kind is passed to a parameter of a ctypes-declared function, passed to ref.assign(val), or assigned to an array element or struct member via a reference, as in arrayref[i] = val or structref.member = val. These rules are intended to lose precision only when there is no reasonable alternative. They generally do not coerce values of one type to another type.

(TODO: precise rules.)

The explicit conversion rules are applied when a JavaScript value is passed as a parameter when calling a type, as in t(val) or new t(val). These rules are a bit more aggressive.

(TODO: precise rules.)

ConvertToJS(x) - This function is used to convert a ctypes reference or a C++ return value to a JavaScript value. The intent is to return a primitive value or ctypes pointer whenever possible, and a ctypes reference otherwise. The precise rules are:

• If the value is of type void, return undefined.

• If the value is of type bool, return the corresponding JavaScript boolean.

• If the value is of a number type, return the corresponding JavaScript number. (In the case of 64-bit integer types, this can result in a loss of precision.)

• If the value is a null pointer (of a pointer or string type), return null.

• If the value is of a string type and non-null, return a JavaScript string.

• If the value is of any other pointer type and non-null, return a ctypes pointer with the appropriate type and value.

• Otherwise the value is of an array or struct type. If the argument x is a ctypes reference, return it. Otherwise allocate a new buffer of the appropriate size, populate it with the C++ value x, and return a ctypes reference to the complete object in the new buffer.