Labs/Jetpack/Packaging System Notes

Notes on packaging systems for Jetpack

Nearly every change to a standard feed corresponded with a change in the Ubiquity platform itself; this coupling meant that the feed we "streamed" was identical to the version of the feed that came with Ubiquity itself. This is basically a specific case of dependency hell, which streaming functionality has a dangerous tendency to facilitate.
Streaming functionality introduces issues of server infrastructure and scalability into the system that weren't really present before.

David Ascher's advice (paraphrased): packaging systems are hard. ActiveState spent years on a system that went nowhere, and most of the packaging systems in existence have significant flaws.

Given this, we should try to make the system as simple as possible, though this will probably mean also making it limited in some ways. At the very least, we should probably be explicit from the beginning about what the packaging system won't do, and make sure that everyone's okay with these limitations.

Possible solutions

Note that these solutions aren't mutually exclusive; some can be combined with others.

Since the volume of code at the beginning of the ecosystem will be fairly small, we may want to consider simply including the entire package repository in the Jetpack extension itself, and pushing out new versions of Jetpack frequently.

Alternatively, we can include (and stream on a regular basis) and index of the SHA1 hashes of every package/chrome booster that we've reviewed and approved; this allows individual Jetpacks to simply self-host the chrome boosters they use--either in ZIP files or as separate files on the server--and the Jetpack extension can perform validation to ensure that all chrome code being used is pre-approved, raising big red flags if not.

Narwhal has a package format/system called tusk that's pretty straightforward and uses JSON metadata and a standard directory structure. We could potentially expand the JSON metadata to include Jetpack-specific information, such as the kind of context SecurableModules need to be executed in (chrome vs. content, etc.). There's already a number of tusk packages out there that we can potentially reuse. Tusk itself is written in JS so we might even be able to reuse the code here. Bespin may be using this package format as well, or a superset of it.

I suspect that SecurableModules should be loaded in content-space unless otherwise specified in package.json.