MDN/Archives/Kuma/Scripting

Overview

The mission of the Kuma wiki is to replace MindTouch. One of the advanced features of MindTouch is DekiScript, a Lua-based scripting environment for wiki authors.

We'd like to replace this feature with something secure, capable, modern, and convenient. We'd also like it if we could carry over the existing body of DekiScript with minimal manual changes. Let's see what's feasible.

DekiScript highlights

• Lua-based dynamic scripting language
• XML/HTML literals for easier construction of well-formed markup
• Inline { { script expressions } } in wiki page content
• Long-form Template: scripts callable with parameters from inline expressions
• Local API with access to Wiki data queries (eg. custom tables of contents, etc)
• Network API with access to external web services (eg. RSS feeds, bugzilla, etc)
• L10N-aware constructs and alternate content
  • Example: Template:XULAttrInheritedWide
  • eg. span lang="en" class="lang lang-en", span lang="de" class="lang lang-de", span lang="*" class="lang lang-*"
• Safe for use by wiki content authors
  • (Not entirely sure what makes it safe, need more research)
    • Seems like long-form Template: scripts are editable only by elevated-permission users?
    • Normal content editors can make calls to Template: scripts, but not employ all scripting features?

Code samples

Here's some inline script excerpted from en/Firefox_9_for_developers:

    <li>The <code>value</code> attribute of {{HTMLElement("li")}} now can be
    negative as specified in HTML5. Previously negative values were converted
    to 0.</li>

Here's the source of Template:HTMLElement, a commonly used script:

    /* accepts as input one required parameter: HTML element to create a xref to */
    var uri = uri.parts(Page.uri);
    var lang = string.tolower(uri.path[0]);
    if (string.contains(lang, "project") || string.contains(lang, "Project")) {
      let lang = string.substr(lang, 8);
    }
    /* fall back to page.language on a user page */
    else if (string.StartsWith(lang, "user:")) {
      let lang = page.language;
    }
    var name = $0;
    var sectionname = "Element";

    if (!string.compare("es", string.tolower(lang))) {
      let sectionname = "Elemento";
    }

    if (args.title) {
      let name = args.title;
    }
    var dest = lang .. '/' .. 'HTML/' .. sectionname .. '/' .. name;
    var destEng = 'en/HTML/Element/' .. name;
    if (wiki.pageExists(dest)) { /* the page exists */
      <code>(web.link(wiki.uri(dest), '<' .. name .. '>'))</code>;
    } else if (lang == 'zh_tw' && wiki.pageExists(destEng)){
      /* the MozTW community consider links to English pages better than red ones. 
         I'll write about this to mozilla.dev.mdc later */   
      <code>(web.link(wiki.uri(destEng), '<' .. name .. '>'))</code>;
    } else { /* the page doesn't exist */
      var targeturi = "https://developer.mozilla.org/Article_not_found?uri=" .. dest;
      <code><a rel=('internal') href=(targeturi) class=('new')>web.text('<' .. name .. '>')</a></code>
    }

Miscellaneous notes

• Would like a relatively boring solution that matches core competencies
  • Components should have active existing communities
  • Try to stay close to dev team core-competencies
  • Try to make it sensible to host for care & feeding by IT

• Switch from Lua to JavaScript basis
  • JavaScript is more of a webdev core-competency
  • JS is pretty close to Lua
  • node.js has an active community, works out-of-the-box
    • Maybe consider SpiderNode, but only if we get benefits beyond not-invented-here purity

• Or, do something with Lua?
  • Wikipedia seems fond of Lua for their templates

• HTTP proxy service vs embedding JS
  • Embedding JS in a Python app seems experimental, and outside webdev team core-competency
  • HTTP is a webdev core-compentency, and has loose-coupling and RESTafarian benefits
  • Taming a code execution environment in its own process seems safer and more easily scaled

• Taming semi- or untrusted JavaScript code
  • Built-in node.js sandboxing features
  • Sandbox to wrap JS execution in node.js?
    • Seems to have some more features than built-in node.js sandboxing
    • timeouts, restricted method access, graceful errors

• Use standalone embedded JS templates instead of literals embedded in script?
  • Employs code-in-markup instead of markup-in-code, and avoids the need for fancy compiler hijinx
  • This is being used in other projects

• Might js-xml-literals come in handy?
  • I failed on my first attempt to install, so haven't seen this in action.
  • This seems experimental; no one else really seems to be using it yet

• Should we get a compiler / language expert in on this for opinions?
  • This smells like dangerous excitement

• Could we port DekiScript directly? (It's open source, right?)

Proposed solution #1

The basic concept:

• Kuma (Django/Python) serves as the wiki/CMS for both general content and template scripts
• Document views are proxied through a node.js-based service that evaluates and executes inline macros.
• Support inline parameterized macros that invoke template scripts.
  • Macros are usable by all content editors
  • Macros are neutered function calls, basically, and not full JS expressions
  • Macros are handled by a custom parser, and definitely not just eval()'ed
• Support template scripts that are evaluated in a node.js sandbox as EJS templates.
  • Template scripts could be made editable only by elevated-privilege editors

Considering this from a high-level view of off-the-shelf parts that could be glued together:

• Kuma wiki CMS (Django/Python)
• Sandboxed server-side JavaScript filter proxy (node.js)
• EJS templates
• Convenience methods to perform wiki content queries and utility functions
  • Something like what DekiScript offers for wiki functions and variables
• Plentiful and intelligent HTTP-based caching

Prototype

lorchard has started work on a prototype in node.js:

• https://github.com/lmorchard/kumascript

Flow sketch

Here is a rough sketch of the process and network flows involved in this proposed solution:

More notes:

• If the caching works right, we should be able to almost never load & compile templates unless they're edited. (Cache invalidation? HEAD requests before GET requests?)
• When a Kuma wiki page is edited, the last-cached KumaScript response is discarded.
• To support preview functionality, we can accept HTTP POST of arbitrary page source at the start of the KumaScript flow. Not cached, which is a feature.

Code samples (imaginary)

We can limit inline expressions to invoking long-form templates, rather than free-form scripting. Usage still looks familiar with respect to content "in the wild", though:

    <li>The <code>value</code> attribute of {{HTMLElement("li")}} now can be
    negative as specified in HTML5. Previously negative values were converted
    to 0.</li>

Long-form templates become embedded JS templates, something like this:

<%
    /* accepts as input one required parameter: HTML element to create a xref to */
    var uri = uri.parts(Page.uri);
    var lang = string.tolower(uri.path[0]);
    if (string.contains(lang, "project") || string.contains(lang, "Project")) {
      let lang = string.substr(lang, 8);
    }
    /* fall back to page.language on a user page */
    else if (string.StartsWith(lang, "user:")) {
      let lang = page.language;
    }
    var name = arguments[0];
    var sectionname = "Element";

    if (!string.compare("es", string.tolower(lang))) {
      sectionname = "Elemento";
    }

    if (args.title) {
      name = args.title;
    }
    var dest = lang + '/' + 'HTML/' + sectionname + '/' + name;
    var destEng = 'en/HTML/Element/' + name;
    if (wiki.pageExists(dest)) { /* the page exists */
      %> <code><%- web.link(wiki.uri(dest), '<' + name + '>')) %></code> <%
    } else if (lang == 'zh_tw' && wiki.pageExists(destEng)){
      /* the MozTW community consider links to English pages better than red ones. 
         I'll write about this to mozilla.dev.mdc later */   
      %> <code><%- web.link(wiki.uri(destEng), '<' + name + '>')) %></code> <%;
    } else { /* the page doesn't exist */
      var targeturi = "https://developer.mozilla.org/Article_not_found?uri=" .. dest;
      %> <code><a rel="internal" href="<%= targeturi %>" class="new"><%- web.text('<' + name + '>') %></a></code> <%
    }
%>

The above presupposes an API similar to what's exposed to DekiScript, is non-functional, and mainly serves as a thought experiment in comparative syntax.

Sandboxed JavaScript execution

• Can this be done in a way that restricts file, network, memory, and CPU usage?
  • Anything else dangerous and in need of restriction?
  • How restrictive do we need to be? (ie. is trusting a certain set of authors enough?)
  • How restrictive is DekiScript?

• Options inside node.js
  • node.js has sandboxing out-of-the-box
  • There's also Sandbox

• Options for host running node.js
  • No filesystem access at all (chroot?)
    • and/or LXC
  • Whitelisted network access (eg. firewall rules? limit base URLs of services?)
  • Limited execution time (eg. kill the process after 30000 msec?)
  • Limited memory usage (eg. kill the process after 10MB consumed?)
  • Auto-disable script if abuse detected (eg. penalty box for X minutes?)