SummerOfCode/2012/Meemoo

From MozillaWiki
Jump to: navigation, search
Meemoo is "the Arduino for Web"

Meemoo (Hackable Web Apps)

Meemoo is a framework for hackable web apps by Forrest Oliphant and an incredible tool for web making. My main goal in the project is to extend Meemoo to make modules creation easier. Based on a collaborative live code editor I want people creating new modules for hackable web apps inside their own browsers, without external tools.

Give Meemoo a try! Go http://meemoo.org/iframework

Fork Meemoo! Go http://github.com/meemoo

Weekly Report

A daily log of my activities around Meemoo GSoC 2012 project is at: http://etherpad.mozilla.org/meemoo2012. Here is a resume of those activities week by week.

Before GSoC period

My first contact with Meemoo was by Alex McLean, who presented Meemoo as an interesting livecoding environment on Web. This was around middle of March and after some days I knew Meemoo was part of Moz GSoC.

At March 29 through April 23 I was in contact with Forrest. First of all I created a hello-world module (now we are calling this external modules by "NodeBox-Iframe") to understand how modules communicates inside Meemoo. I was surprised how easy was that! This first contact with Meemoo generated this tutorial: https://github.com/meemoo/iframework/wiki/Creating-a-Module.

I was amazed with Meemoo and started to write my proposal to Moz GSoC. I read about Moz focus on "creating webmakers" by Mark Surman, Dave Humphrey and others mozillians, so I wanted to join the initiative, and Meemoo seems to be an amazing tool around that. In this meantime I studied Backbone.js, window.postMessage and all the Meemoo's architecture: iframework (the environment) and meemoo.js (the embeddable module lib).

In this period I reported some issues on iframework and created a bunch of simple modules and apps:

I also fixed an issue around actions history:

And at April 23 I was crossing my fingers :-)

Community bonding period

Week 1 (Apr 23 - Apr 30)

For this period I originally proposed: 

   April 23 - May 21 (4 weeks): discussing with my mentor about the exact features and 
   means of implementation around Meemoo, syncing with my mentor works and doing 
   some prototypes.

I started thinking about the live code editor (how to implement that? as a module? how to evaluate the generated code?) and decided to go with an iframe editor using Ace: [1] and [2].

Week 2 (May 1 - May 7)

I discussed some ideas around the live code editor with Forrest and at the end of week Forrest pointed me to Meteor. I studied Meteor planning to use it to make the live code editor collaborative. And with Forrest we planned to use Meteor as a way to make the whole Meemoo (iframework) collaborative! I created a test app using Meteor and feel surprised. I just didn't liked all the magic needed to generate DOM, but we can use something in the middle to make the "view source" work.

Week 3 (May 8 - May 14)

Renato Fabbri invited me to use Meemoo on "Cúpula dos Povos" at "Rio+20" conference, here in Brazil. I planned to create some modules to do photo sequencing and 3D forms synthesis based on parameters extracted from audio tags. I started creating a supershape.js module: http://meemoo.org/iframework/#/gist/2644999

Forrest pointed me to toolness' friendlycode (now the editor behind Moz Thimble) and I got some ideas for the live code editor (mainly the tips thing).

I started to plan how to use the audio APIs (Web Audio, Audio Data, MediaStream) in a compatible manner inside Meemoo. I decided to create a Web Audio API module.

We discovered RecursiveDrawing.com and some ideas exploded :-)

Jussi pointed me to his mike.js (https://github.com/jussi-kalliokoski/mike.js) to get raw data from mic while the W3C Audio spec isn't defined.

Week 4 (May 15 - May 21)

We were getting more and more ideas for apps and modules so we decided to store everything here:

Forrest was working at Hackable Clock app to Moz Summer of Code Party meeting (https://etherpad.mozilla.org/May22 ) and I forked the app with some audio thing: http://meemoo.org/iframework/#gist/2765356 ;-)

I also started to study the Web Audio API (https://gist.github.com/2728285)

Forrest started to design the Native Nodes for iframework and it make my ideas to change a bit. I decided to make the live code editor and the audio nodes as native nodes.

Coding period

Week 5 (May 22 - May 28)

Talked with Joe, the creator of Audiolet, about Web Audio API integration and he gave me some good ideas. But I still need to understand how to use the native Web Audio nodes instead of JavaScriptAudioNode.

Forrest presented Meemoo as a demo day at his University and meet Sebastian, the creator of an awesome WebPd fork!

For this week I originally proposed for GSoC: 

   May 22 - May 28 (1 week): fixing the remaining bugs on iframework and meemoo.js, 
   getting them ready to work on the live code editor.

But Forrest was working on a refactoring, so I delayed the work on iframework and focused on the live code editor and audio nodes. I thought a lot about them and I decided:

  • Having the live code editor as a friendlycode fork (livecode branch) and as a native node invoked by "view source" and "create new module" events inside iframework
  • Having the audio nodes using Web Audio API or Audiolet (selected by browser type, for now) and as native nodes. As we got more resolution from W3C Audio spec we can remove the specific code

Week 6 (May 29 - Jun 4)

For the next two weeks I proposed: 

   May 29 - June 11 (2 weeks): extending iframework to create the code editor. 
   I already have a simple editor implemented using Ace [1] and  another using 
   CodeMirror [2]. I will integrate one of those editors inside iframework.

I implemented some changes on live code editor (now a fork of friendlycode, the core of Moz Thimble).

I created some use cases (see http://etherpad.mozilla.org/meemoo2012, beginning of pad) and tested those with Web Audio API and everything worked well but just in a Chrome canary build. We need to have a node inside Audiolet to get raw data from audio and video tags sources. I have to discuss more with Joe about that.

Google launched Blockly and I planned to use that inside the live code editor, as an optional "mode" (like Emacs modes).

Forrest created a new branch for native nodes and I fetched. Things started to get a form!

Week 7 (Jun 5 - Jun 11)

I worked in the audio nodes and the live code editor. They are not native nodes of Meemoo yet, but standalone pages. I'm planning to integrate them with iframework in 2 or 3 days.

Week 8: (Jun 12 - Jun 18)

Originally proposed: 

   June 12 - June 18 (1 week): integrating the editor with share.js or etherpad-lite, 
   to make possible collaborative coding.

I have to discuss with Forrest because we planned to make the whole iframework collaborative. But for now, maybe we could have just the live code editor collaborative, based on Meteor.

I'm planning to have the editor and audio nodes as native nodes this week too. UPDATE: the audio nodes is dependant of native nodes and we have lots of work before touching the media nodes

I discussed with Forrest and we are concentrating on native nodes for now. I have started a Meteor app to use as an external live code editor while we don't have native nodes. I'm testing how to update the whole iframe every time we change the Meteor editor. When we got native nodes, we are planning to use Meteor to make the whole iframework collaborative, but it is not possible now. 

   Forrest: I think we should experiment with just modules for now, because I'm 
   guessing it will take a bigger rewrite to make iframework work with it. 
   I like the idea of different apps on different systems feeding 
   images/sounds/data into the same meteor module.
   
   Vilson: OK! So a version of the editor in meteor...

Forrest integrated Grunt, so now we can start doing tests and the deploying is much easier.

We also started to think more seriously about an integration with Google Blockly, like a module to do logic operations. I'm planning a Blockly mode for the live code editor to edit JavaScript.

Week 9: (Jun 19 - Jun 25)

Originally proposed: 

   June 19 - June 25 (1 week): creating a Web server based on Django or node.js 
   to store modules created by everyone. Every module is refered by its URL in 
   Meemoo, which makes the server design simple.

Forrest worked on local storage and we are using Gist to store public apps. So, for now we don't need a server. Maybe this will come with the Meteor "backend". We had a big push to iframework repos with all those features. Meemoo is rocking! :-D

In this way I got this week to continue the work on the editor and created a module for mr.doob's Harmony: http://meemoo.org/iframework/#gist/2959806 and we are having fun :-)

So1ik.gif 601267_323131427772915_595524936_n.jpg 550432_323079607778097_1216176047_n.jpg 552783_323134351105956_963782996_n.jpg

Some ideas for the live editor UI from http://codepen.io: 

   Vilson: Lots of good ideas from codepen. I think we can use the live editor 
   for lots of other projects too. Thinking on a editor to use "creative coding" 
   libs like three.js, paper.js or audiolet.

We are using the Facebook page of Meemoo to post some updates, together with Twitter: http://facebook.com/meemooapp

Week 10: (Jun 26 - Jul 2)

Originally proposed: 

   June 26 - July 2 (1 week): beginning creating new modules inside the editor: 
   a processing.js edit-and-run module, an event sequencer and an audio sampler 
  (based on Audiolet's BufferPlayer, using Mozilla's Audio Data API).

The whole media nodes, including audio, are dependant of native nodes. So we are delaying this. I'm creating another modules like meemoo-supershape.js as iframe nodes.

Forrest created math evaluation nodes! And now we can have cool animations like these:

And we continue having lots of fun :-D

548607_326966247389433_427146130_n.jpg jYDuh.gif B67r2.gif sDIz0.gif TcO6N.png Ch1PD.png JR5Zl.png hWdVl.png

I changed my priorities for:

  1. live code editor as meteor app
  2. native nodes
  3. audio nodes using web audio api
  4. app view as "show only this node"
  5. fullscreen view as "show only this node in fs"

Discussed with Forrest about app view and for now we are thinking on a button to hide the whole app but the current module.

Taking some inspiration from https://github.com/sullerandras/collaborative-html-editor-with-meteor to the live code editor refresh iframe thing.

Forrest wrote about his progress on June: http://meemoo.org/blog/2012-06-27-nearing-beta/

nLFKo.gif y2GKs.gif

I'll post mine soon.

Week 11: (Jul 3 - Jul 9)

Originally proposed: 

   July 3 - July 9 (1 week): testing to ensure that the editor perfectly works 
   before the mid-term evaluations, asking for feedback from my mentor and the 
   community.

I have the editor almost done! I'll deploy on meteor.com soon and start testing this weekend!

Thinking together with Forrest about ideas for more Mozilla Webmakers projects like https://wiki.mozilla.org/Webmakers/Projects/MeemooClock-DIY

  • How to create your own GIF?
  • How to create crazy 3D shapes?
  • How to create your meme and publish on Facebook?

Week 12: (Jul 10 - Jul 16)

Originally proposed: 

   July  9 - July 13 (4 days): submitting the project for mid-term evaluation,
   talking with my mentor about the current results and discussing what 
   should be changed.

I wrote the mid-term evaluation and passed!

This week I started getting in trouble with Meteor. URL routing wasn't implemented yet and I had to do it by hand using Backbone Routing. It is working now.

I deployed the app at http://liveeditor.meteor.com and the source at http://github.com/automata/meemoo-liveeditor

We decided to have two views for each module: "edit" and "module". User could then change between the two views. I planned to use http://liveeditor.meteor.com/foo/edit and http://liveeditor.meteor.com/foo/module to have both modes for the "foo" module.

I continued working on three fronts: the editor, the audio nodes and hacking new modules.

We noticed Ward Cunningham "Federated Wiki" was similar with Meemoo approach but focusing on "distributed wikis" instead of "hackable apps". Maybe we can work together soon!

Week 13: (Jul 17 - Jul 23)

Originally proposed: 

   July  14 - July 23 (more than a week): continue creating new modules: an
   audio  synthesizer (like [3]), a video mixer (inspired by Forrest thesis
   on Meemoo), a tone matrix clone [4] and a simple din clone [5]

My choice for Meteor was based on the easy to develop a WebSocket based collaborative editor that could update the generated HTML inside a module automatically. Unfortunately Meteor reveals not a good option to "non-standard" Webby things like I'm doing: an Etherpad-like editor. I decided to go with another approach: implementing the whole WebSocket thing by hand. The frontend is almost done, so it is not a big problem but I'll need to study things like patching and WebSockets.

In the modules front we got in touch with Japanese community of audio synth g200kg. The guys created amazing synths and I created two new modules to have them inside Meemoo:

Demos apps are here:

We even have Beethoven's Moonlight Sonata in Meemoo :-D

Both modules communicates using MIDI-like messages. Soon I'll implement a better MIDI type for Meemoo, respecting the draft spec from Jussi and Chris from W3C Audio WG. For now we are using MIDI messages similar of g200kg's WebMIDILink: http://www.g200kg.com/en/docs/webmidilink/spec.html.

I was not totally OK with the original proposal. I have to write some other modules but I'm in touch with Forrest to hack priority ones first.

Week 14: (Jul 24 - Jul 30)

Originally Proposed: 

   July  24 - August 6 (two weeks): working on iframework to improve its UX, 
   creating a theme system and zoom/panning functionalities based on ThreeNode.js.

This will have to wait. I passed this week studying Meteor, share.js, Node.js and WebSockets and comparing them. I still tried to implement the editor in Meteor but failed.

I contacted Carols from Google for support to go to FISL but unfortunately I could not go to Porto Alegre :-(

Week 15: (Jul 31 - Aug 6)

Starting a countdown to pencils down date!

This week we are working on a Meteor module to share state between multiple connected clients. A first version is online at http://meemoo.org/iframework/#gist/3209721. Just open that in various tabs and change the slider! The module: http://meemoo-tunnel.meteor.com and its source: http://github.com/automata/meemoo-tunnel. The point is simple but opens lots of possibilities: someone using the meemoo-tunnel can send data for other remote iframeworks using meemoo-tunnel as well. We can create, for example, a 20-channel audio synth (one channel per computer) controlled by all the participants, or a visual projection controlled remotely by someone's mobile device!

In the editor front I'm implementing the editor in Node using ShareJS instead of Meteor. ShareJS is more interesting for my approach because I only need to share text changes done in a CodeMirror editor. I don't want the entire application state being shared. Maybe we can go for Etherpad Lite in a near future, but I'm happy with ShareJS for now.

About the audio nodes, we decided to use Web Audio API in Chrome and Audiolet in Firefox. I talked with Robert O'Callahan and he created a watching bug making possible to follow Moz progress in implementing Web Audio API in Firefox. Hoping to help the media team as much as possible in the following months. Matt Diamond created an awesome thread about a possible standard to interchange Web Audio API based apps. A nice find to us because we are thinking about ways to standardize those apps inside Meemoo and MIDI support too.

And thanks to the incredible work of Forrest the native nodes works! YEY! I'm studying them and learning a lot!

At the end of week Pomax and Atul (THANKS!) fixed #485 in webpagemaker (Moz Thimble). Now we can use pages created in Thimble inside iframes and then inside Meemoo. I started talking with Atul about the possibility of JS editing inside Thimble.

I also created another module, the meemoo-intro. The app is here: http://meemoo.org/iframework/#gist/3282702 (UPDATE: now Forrest made it the default Meemoo app, yey!). It is an interactive guide to Meemoo newcomers. We have to work a lot on that to explains Meemoo as much as possible. Oh, yes, I used popcorn.js for that, it is awesome!

Week 16: (Aug 7 - Aug 13)

Originally proposed: 

   August  7 - August 13 (1 week): refactoring and fixing the code of
   iframework, meemoo.js, meemoo-modules and Meemoo modules server,
   finishing and styling the documentation. Creating example apps using the
   new modules and commiting the last updates into Meemoo repositories.
   Final review with my mentor and the  community before the "pencils-down"
   date.

I have lots of works yet! I'm pushing code reviewing for the next week. I don't see GSoC as a '12-weeks project' but as a start to work more inside Meemoo and Mozilla projects like Thimble/WebMaker. In this way my priorities are in hacking as much as possible the modules I created, the code editor and iframework, and them report my entire progress and do code review in the next week.

I participated in the WebMakers call and Forrest presented the meemoo-intro app! Thanks Fo! http://meemoo.org/iframework/#gist/3282702

This week some guys (@janesconference and @thealphanerd -> he is also a GSoCer! Small world!) mailed me about osc-web and we talked about a way to make it a standard and use it inside Meemoo. For now, we can use OSC inside Meemoo by running a local nodejs server as a 'bridge' between OSC and HTTP. Forrest suggested to use a Java applet (like midibridge does) and I'm thinking about that.

About the editor (after some fixes in the CodeMirror integration with ShareJS) it is online at http://li7e.org. I sent the fix to ShareJS too. The source of "LI7E" ("LIVE" with a rotated V ;-)) is on http://github.com/automata/li7e. Planning to test some things on LI7E and them suggest them to Atul to have that in Thimble!

So, now we can create a new Meemoo module inside our own browsers! The steps are something like:

I started the audio nodes based in Chris Wilson's Web Audio API playground. Created some simple nodes in my branch 'audio': https://github.com/automata/iframework/blob/audio/src/nodes/audio.js.

Created the apps gallery in meemoo.org: http://meemoo.org/hack-our-apps/

Week 17: (Aug 14 - Aug 20)

Originally proposed: 

   August  14 - August 19 (some days): last days to do minor fixes and
   stay in contact as much as possible with my mentor to polish the project.

Attended to WebMakers call today and was presented to TowTruck from Moz Labs: http://vimeo.com/36754286. It is awesome! It is so close of the editor (LI7E) we are creating for collaborative coding! I'll try to talk more with Atul about that.

For this week: Web Audio API native nodes, better meemoo-intro, documentation and code review of all the repos I created/pushed.

UPDATE: I made better meemoo-intro modules, worked on edge's events for (dis)connection (this is useful for Seriously nodes and audio nodes because we need to connect and disconnect them in their own 'DSP' graphs) and implemented detach mode on iframe node boxes. Taking notes about contributions I did this summer (winter ;-)). Talking with my mentor about the next steps. Meemoo hacking must continue! ;-)

Summary of contributions

Listing here almost all the contributions I did this summer. Repositories touched:

  • Meemoo iframework
    • Meemoo main repos (gh-pages officially online at http://meemoo.org/iframework). Created initial MIDI support, detach mode on nodes, started audio native nodes based on Web Audio API, events history for undo actions and so on.
  • LI7E
    • Collaborative module creator. Main goal of the original proposal. An editor based on ShareJS and CodeMirror to create new iframe modules and use inside Meemoo. Online at http://li7e.org. Meemoo apps using modules created in LI7E are available here and here
  • ShareJS
    • LI7E uses this JS lib to state sharing and OT. Sent a patch to integration of CodeMirror with ShareJS.
  • meemoo-modules
    • A fork of Forrest's one. I did some modules (and, audio-oscillator, audio-output, lived, console, evaluator, if-bang, sliders)
  • meemoo-tunnel
    • A Meteor app to share state between concurrent iframeworks. A demo here (tip: open two or more apps and change color sliders).
  • meemoo-liveeditor
    • I started LI7E as an Meteor app, but I changed my plans to use ShareJS along the way (and now, after I knew Moz TowTruck, maybe I'll change for Etherpad Lite in a near future). However, it is an interesting example on how to code a distributed app using Meteor.
  • friendlycode and webpagemaker
    • Moz Friendlycode and webpagemaker were fundamental for LI7E. I'm in touch with Atul Varma (@toolness) and I hope to continue collaborating with him.

I wrote a daily and weekly report (this wiki page) as part of documentation and created other demo apps. Those apps were posted on our Meemoo FB page. There are some issues and wiki pages on github of iframework and LI7E as well.

The end of GSoC is not the end of my participation on this projects. I'm going to collaborate in Meemoo dev (together with Forrest) and looking forward to help Thimble (together with @toolness) and continue the dev on LI7E and other Meemoo modules as well.

Let's keep hacking!

More information

The official Meemoo page is: http://meemoo.org

The official project details is in: http://www.google-melange.com/gsoc/project/google/gsoc2012/automata/14001

And the accepted proposal is publicly available at: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/automata/1

About the authors

Forrest Oliphant (mentor): http://forresto.com ___ @forresto

Vilson Vieira (student): http://automata.cc ___ @aut0mata

Retrieved from "https://wiki.mozilla.org/index.php?title=SummerOfCode/2012/Meemoo&oldid=462436"