MDN/Development/Emscripten: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(17 intermediate revisions by the same user not shown)
Line 6: Line 6:
* Run code originated in languages like Python on the web as well, for example by compiling CPython from C to JavaScript
* Run code originated in languages like Python on the web as well, for example by compiling CPython from C to JavaScript


This is a really cool and very useful technology, and it is essential for the success of some of Mozilla's current initiatives, such as [https://developer.mozilla.org/en-US/docs/Games web gaming]. But the uses of Emscripten go beyond just building amazing games, therefore I have decided that Emscripten should have its own landing page on MDN, covering Emscripten in a more generic fashion.
This is a really cool and very useful technology, and it is essential for the success of some of Mozilla's current initiatives, such as [https://developer.mozilla.org/en-US/docs/Games web gaming]. But the uses of Emscripten go beyond just building amazing games, therefore I have decided that Emscripten should have its own landing page on MDN, covering Emscripten in a more complete fashion.


[https://github.com/kripken/emscripten/wiki Emscripten currently has a wiki on github] that contains some great content, but this content could use some updating and reorganization, and we have lots of other ideas for content too, to tell the full story of how useful Emscripten is to interested parties such as C++ developers.
[https://github.com/kripken/emscripten/wiki Emscripten currently has a wiki on github] that contains some great content, but this content could use some updating and reorganization, and we have lots of other ideas for content too, to tell the full story of how useful Emscripten is to interested parties such as C++ developers.
Line 14: Line 14:
==Where will this project live?==
==Where will this project live?==


The obvious place would be a landing page at https://developer.mozilla.org/en-US/docs/Emscripten.
The main host for this docs project will be an MDN zone, with its landing page at https://developer.mozilla.org/en-US/docs/Emscripten.
 
We will also have a professional looking external site, say at emscripten.org, with nice links to demos etc., maybe a dozen pages all in all.  This will be the "landing page", but will link back to MDN for the meat of the content.


==What content should it have?==
==What content should it have?==
Line 20: Line 22:
Although Emscripten can be used to compile a number of different languages into JavaScript, it would make sense to start with C++, as that is the best supported and most useful target laguage for our current projects. The following quote from Jukka provides some good context for the purpose of these docs:
Although Emscripten can be used to compile a number of different languages into JavaScript, it would make sense to start with C++, as that is the best supported and most useful target laguage for our current projects. The following quote from Jukka provides some good context for the purpose of these docs:


<blockquote>My experience and perspective is mainly from the "Game developer who has written a native game in C/C++ for Android/iOS/PC and wants to port it to FxOS/browser" viewpoint. There, the process comes with the following questions, roughly in the following chronological learning order:</blockquote>
<blockquote>My experience and perspective is mainly from the "Game developer who has written a native game in C/C++ for Android/iOS/PC and wants to port it to FxOS/browser" viewpoint.</blockquote>


(feasibility/disbelief) 1. Web doesn't do 3D well and I know web is slow. Is this port even feasible? What kind of sacrifices will I have to do in my existing game in terms of performance? How does web compare to native Android/iOS/PC?
Proposed structure


(feasibility/disbelief) 2. Which browsers can I target with emscripten? ff/chrome/ie/opera/safari? desktop and mobile?
https://developer.mozilla.org/en-US/docs/Emscripten
*- Introducing Emscripten
*- Web abilities and limitations
* Product manual (covers latest version, currently 1.7.8)
**- Download and install
**- Workflow and tools
**- Techniques
**- Reference
*- Demos
*- Older product manual versions
**- 1.7.1
***- Download and install
***- Workflow and tools
***- Techniques
***- Reference
**- 1.5.1
***- Download and install
***- Workflow and tools
***- Techniques
***- Reference
         
===Introducing Emscripten (always cover the latest version of Emscripten)===


(feasibility research) 3. My game uses C/C++ libraries/technologies X, Y and Z. Are these supported? What do I need to rewrite?
An introduction to Emscripten, including how it works, how to get it set up and start working with it, and some beginner's tutorials. Throughout this section we can reference more detailed guides in later sections.


(feasibility research) 4. I am looking an alternative to library X, which I know isn't supported. Which C/C++ technologies can I use instead? Can C/C++ do everything, or do I need to write JS to implement X/Y/Z?
* What is Emscripten and how does it work?
* The nature of Emscripten compiled JS - brief summary
** non-threading
** async main loop
** async file loading
** memory model
** networking
** performance
* Your first steps with Emscripten
** Download and install (https://github.com/kripken/emscripten/wiki/Emscripten-SDK, https://github.com/kripken/emscripten/wiki/Getting-started-on-Mac-OS-X, https://github.com/kripken/emscripten/wiki/Getting-Started-on-Ubuntu-12.10, https://github.com/kripken/emscripten/wiki/Using-Emscripten-on-Windows)
** Hello world (https://github.com/kripken/emscripten/wiki/Tutorial, present A minimal GL rendering sample that shows how to get stuff up and running
*** Generating the code
*** The asm.js module
*** The heap
*** linking
*** Emscripten boilerplate
*** Quick guide to Emscripten compiling, including some essential compiler flags
* Quickstart guide to putting Emscripten in your toolchain
* Next steps
** Emscripten APIs
** Working out what C++ stuff is supported well in Emscripten, and what isn't
** What problems should you solve on the C++ side?
** What should be written in JS and glued on. How is this done?
** https://github.com/kripken/emscripten/wiki/Library-Support ?
** https://github.com/kripken/emscripten/wiki/CodeGuidelinesAndLimitations
** Solving specific problems (mini case studies - these don't need to be long, just present minimal examples and link to further resources in the techniques section?)
*** 2d graphics
*** 3d graphics
*** sound
*** file-system access
*** input
*** networking


(adoption,familiarization) 5. How do I get hands-on experience with the toolchain? Where is that minimal GL rendering sample that shows how to get stuff up and running?
===Web abilities and limitations (probably doesn't need to be version-specific)===


(r&d port attempt) 6. How do I work my existing build system X to target emscripten? How do I co-maintain both native and emscripten targets simultaneously?
This could be a single article but may well be multiple articles under a single section as the document resource grows. The main purpose here is to provide native developers trying to research the feasibility of running their project on the web with some answers.


(r&d port attempt) 7. How do I deploy and run my app on a mobile device?
* How does the web differ to running native code? (https://github.com/kripken/emscripten/wiki/Emscripten-browser-environment)
** non-threading
** async main loop
** async file loading
** memory model
** https://github.com/kripken/emscripten/wiki/Browser-limitations
**networking
** performance
* Performance on the web
** How does it compare to native?
** What are the bottlenecks?
** 3D on the web is slow; how come?
** What kind of sacrifices need to be made when porting to the web, in terms of performance?
* Browser support for Emscripten compiled code
** What does Emscripten rely on?
** What browsers currently support this combination?


(r&d port attempt) 8. How do I do debugging?
===Download and install (could be versioned)===


(r&d port attempt) 9. Where is the compiler and toolchain documentation? Where is the graphics/WebGL/GLES2 info I need to know?
* Win
* Mac
* Linux
* Troubleshooting
* Other useful tool links


(disbelief/solving blockers) 10. I have learned about these hard problems (non-threading, async main loop, async file loading, memory model, networking, performance, ...) that I am now able to relate to my own codebase having attempted a port. What can I do? How do I architect around these problems?
===Workflow and tools (could be versioned)===


(port development) 11. Where is the reference documentation for C/C++ library/api X in Emscripten?
As a native developer, how can I get up and running with Emscripten, including it as an additional development target that fits into my existing toolchain?


(port development) 12. Where is the reference documentation for JS library/api/spec X?
* Emscripten in your development toolchain
** An ideal toolchain
** Other alternatives
* Emscripten in your build system (https://github.com/kripken/emscripten/wiki/Building-Projects)
** https://github.com/kripken/emscripten/wiki/Code-Generation-Modes
** Reworking your existing build system to target emscripten
** Co-maintaining both native and emscripten targets simultaneously
** Deploying and running your app on a mobile device
** Installable app packaging
** Mobile runtime quirks
* Debugging Emscripten code output (https://github.com/kripken/emscripten/wiki/Debugging)


(port development) 13. How do I develop a JS library to solve X and glue it to my C/C++ game?
===Techniques for working with Emscripten (could be versioned)===


(port development) 14. How do I manage FxOS app packaging and mobile runtime?
More detailed tutorials than you got with the hello world section in the introductory material. Generally these should help developers solve specific problems with their code, and porting it across. It might be an idea to, at each point, say what C++ component is being used, and what JavaScript API/construct it is being compiled into, making it easier to understand what is going on whether you are a C++ person or a JavaScript person.


(finalization) 15. How do I do benchmarking, profiling and optimizations? How to minimize data sizes and optimize web transfers?
Generally the C++ devs won't be initially familiar with the JS specs, and if they are presented e.g. with the Gamepad API or Web Audio API, they'll reject those docs since they are foremost looking for C/C++ solutions to their problems. This is partly because they have just learned about this Emscripten C/C++ compiler that they are now harnessing, and they are right now getting their initial problems solved in the C/C++ side without a single line of JS. Another reason is that devs might not know the JS syntax at all, or if they do, there's a considerable amount of new C/C++ <-> JS interop machinery that one needs to learn, and how to integrate JS into the native build arch, and so on, so being able to develop against JS side APIs will generally come much later during the porting cycle, when the landscape has been charted sufficiently enough to know that one can't (yet) solve all the problems with just native C/C++ code alone.


(finalization) 16. How do I interact/glue the C/C++ game experience with the browser environment? Web site development around the game shell page, etc?
* C++ libraries supported by Emscripten - for each one, cover writing code that works best with Emscripten compilation
** SDL
** OpenAL
** EGL (see https://github.com/kripken/emscripten/wiki/EGL-Support-in-Emscripten for a good example)
** OPENGL (https://github.com/kripken/emscripten/wiki/OpenGL-support)
** GLES2
** https://github.com/kripken/emscripten/wiki/GC
** https://github.com/kripken/emscripten/wiki/Linking
** others...
* Architecting around common problems - provide a case study for each? (Currently we do have some examples of "how do I do X in Emscripten" as part of the unit test suite, but naturally that's not a good format for reading as a documentation/learning material. Some of those could be hoisted or used as a basis for a better set of example apps.)
** Common C++ libraries/technologies that Emscripten doesn't handle so well, provide an article for each?
** non-threading
** async main loop (async compilation is already covered at https://developer.mozilla.org/en-US/docs/Games/Techniques/Async_scripts. Perhaps include some extra explanation about what's going on inside the JS engine?)
** async file loading (https://github.com/kripken/emscripten/wiki/Filesystem-Guide, https://github.com/kripken/emscripten/wiki/Filesystem-API)
** memory model
** networking
** performance
** https://github.com/kripken/emscripten/wiki/Asm-pointer-casts
** https://github.com/kripken/emscripten/wiki/Synchronous-Virtual-XHR-Backed-File-System-Usage
* Emscripten APIs (see https://github.com/kripken/emscripten/blob/master/system/include/emscripten/emscripten.h)
** Embind (https://github.com/kripken/emscripten/wiki/embind)
** y
** z
* Combining handspun JavaScript with Emscripten code (https://github.com/kripken/emscripten/wiki/Interacting-with-code)
** Developing a JS library to solve problem X
** Gluing it to your Emscripten application
* Benchmarking, profiling and optimization
** e.g. minimizing data sizes (https://github.com/kripken/emscripten/wiki/Compressing-Downloads)
** optimize web transfers?
** optimizing generated code (https://github.com/kripken/emscripten/wiki/Optimizing-Code, https://github.com/kripken/emscripten/wiki/Optimizing-the-source-code)
* Creating the launch experience
** Creating a start page
** Launching the application into full screen or windowed mode.


Generally the devs are initially not familiar with the JS specs, and if they are presented e.g. with the Gamepad API or Web Audio API, they'll reject those docs since they are foremost looking for C/C++ solutions to their problems. This is partly because they have just learned about this Emscripten C/C++ compiler that they are now harnessing, and they are right now getting their initial problems solved in the C/C++ side without a single line of JS. Another reason is that devs might not know the JS syntax at all, or if they do, there's a considerable amount of new C/C++ <-> JS interop machinery that one needs to learn, and how to integrate JS into the native build arch, and so on, so being able to develop against JS side APIs will generally come much later during the porting cycle, when the landscape has been charted sufficiently enough to know that one can't (yet) solve all the problems with just native C/C++ code alone.
===Reference documentation (could be versioned)===


I think an important angle to cover, that we currently have a big glaring hole at in the current Emscripten docs at https://github.com/kripken/emscripten/wiki#technical-documentation is the answers to questions #3, #4 and #11. We should have strong documentation pages on each individual API/library we support directly in Emscripten, since a huge part of the Emscripten work (and one that stands us out most compared to nacl!) goes into making sure that existing C/C++ technologies like SDL, OpenAL, EGL, GLES2, etc. are directly supported. For each of these apis that are there, we should have a page "Support for X in Emscripten", similar to we now have for EGL https://github.com/kripken/emscripten/wiki/EGL-Support-in-Emscripten , since that will help people at the initial feasibility study phase and to deciding positively on whether they will take on the effort to try a port.
* Compiler reference
** Compiler flags list (The most important compiler flags are mentioned in https://github.com/kripken/emscripten/wiki/Tutorial and https://github.com/kripken/emscripten/wiki/Optimizing-Code)
* Links to relevant C/C++ library documentation, with annotations relevant to Emscripten usage
* Links to relevant JS/API documentation, with annotations relevant to Emscripten usage
* Links to other relevant docs - Graphics, GLES2?
* Links to relevant tool documentation
* Emscripten FAQ (https://github.com/kripken/emscripten/wiki/FAQ)


A second set of docs we should have more flesh in, is the set of emscripten-specific APIs that we have been developing. We don't have much of those, but generally this header file is the only doc available https://github.com/kripken/emscripten/blob/master/system/include/emscripten/emscripten.h .
===Demos (does this need to be versioned?)===


Third, the set of big problems that everyone porting a native game will eventually run into (item #10 in the list above). Until we have good fixes on those, we should have good prose-like technical documentation that describes the problems with examples of what will not work, and how to refactor the code so that it is JS-friendly, etc.
* a
* b
* c
* etc.


Currently we do have some examples of "how do I do X in Emscripten" as part of the unit test suite, but naturally that's not a good format for reading as a documentation/learning material. Some of those could be hoisted or used as a basis for a better set of example apps.
==What should we talk to?==
 
==Raw ideas==


==What should we talk to?==
Alon, Jukka, Vlad, Martin Best, Dave Herman, Luke Wagner, the rest of the games team!

Latest revision as of 10:13, 29 November 2013

Project statement

Emscripten is an LLVM to JavaScript compiler. It takes LLVM bytecode (which can be generated from C/C++ using Clang, or any other language that can be converted into LLVM bytecode) and compile that into JavaScript, which can be run on the web (or anywhere else JavaScript can run). Using Emscripten, you can

  • Compile C and C++ code into JavaScript and run that on the web
  • Run code originated in languages like Python on the web as well, for example by compiling CPython from C to JavaScript

This is a really cool and very useful technology, and it is essential for the success of some of Mozilla's current initiatives, such as web gaming. But the uses of Emscripten go beyond just building amazing games, therefore I have decided that Emscripten should have its own landing page on MDN, covering Emscripten in a more complete fashion.

Emscripten currently has a wiki on github that contains some great content, but this content could use some updating and reorganization, and we have lots of other ideas for content too, to tell the full story of how useful Emscripten is to interested parties such as C++ developers.

This document outlines the plans for an MDN resource covering Emcripten.

Where will this project live?

The main host for this docs project will be an MDN zone, with its landing page at https://developer.mozilla.org/en-US/docs/Emscripten.

We will also have a professional looking external site, say at emscripten.org, with nice links to demos etc., maybe a dozen pages all in all. This will be the "landing page", but will link back to MDN for the meat of the content.

What content should it have?

Although Emscripten can be used to compile a number of different languages into JavaScript, it would make sense to start with C++, as that is the best supported and most useful target laguage for our current projects. The following quote from Jukka provides some good context for the purpose of these docs:

My experience and perspective is mainly from the "Game developer who has written a native game in C/C++ for Android/iOS/PC and wants to port it to FxOS/browser" viewpoint.

Proposed structure

https://developer.mozilla.org/en-US/docs/Emscripten

  • - Introducing Emscripten
  • - Web abilities and limitations
  • Product manual (covers latest version, currently 1.7.8)
    • - Download and install
    • - Workflow and tools
    • - Techniques
    • - Reference
  • - Demos
  • - Older product manual versions
    • - 1.7.1
      • - Download and install
      • - Workflow and tools
      • - Techniques
      • - Reference
    • - 1.5.1
      • - Download and install
      • - Workflow and tools
      • - Techniques
      • - Reference

Introducing Emscripten (always cover the latest version of Emscripten)

An introduction to Emscripten, including how it works, how to get it set up and start working with it, and some beginner's tutorials. Throughout this section we can reference more detailed guides in later sections.

Web abilities and limitations (probably doesn't need to be version-specific)

This could be a single article but may well be multiple articles under a single section as the document resource grows. The main purpose here is to provide native developers trying to research the feasibility of running their project on the web with some answers.

Download and install (could be versioned)

  • Win
  • Mac
  • Linux
  • Troubleshooting
  • Other useful tool links

Workflow and tools (could be versioned)

As a native developer, how can I get up and running with Emscripten, including it as an additional development target that fits into my existing toolchain?

Techniques for working with Emscripten (could be versioned)

More detailed tutorials than you got with the hello world section in the introductory material. Generally these should help developers solve specific problems with their code, and porting it across. It might be an idea to, at each point, say what C++ component is being used, and what JavaScript API/construct it is being compiled into, making it easier to understand what is going on whether you are a C++ person or a JavaScript person.

Generally the C++ devs won't be initially familiar with the JS specs, and if they are presented e.g. with the Gamepad API or Web Audio API, they'll reject those docs since they are foremost looking for C/C++ solutions to their problems. This is partly because they have just learned about this Emscripten C/C++ compiler that they are now harnessing, and they are right now getting their initial problems solved in the C/C++ side without a single line of JS. Another reason is that devs might not know the JS syntax at all, or if they do, there's a considerable amount of new C/C++ <-> JS interop machinery that one needs to learn, and how to integrate JS into the native build arch, and so on, so being able to develop against JS side APIs will generally come much later during the porting cycle, when the landscape has been charted sufficiently enough to know that one can't (yet) solve all the problems with just native C/C++ code alone.

Reference documentation (could be versioned)

Demos (does this need to be versioned?)

  • a
  • b
  • c
  • etc.

What should we talk to?

Alon, Jukka, Vlad, Martin Best, Dave Herman, Luke Wagner, the rest of the games team!

Retrieved from "https://wiki.mozilla.org/index.php?title=MDN/Development/Emscripten&oldid=777116"