This is the design concept wiki page for Wide Angle Camera Panorama feature.

Background Knowledge

Camera panorama is an common feature in current smart phone OSs. The stock camera APP of Andorid and iOS all provide this feature. The wide angle panorama module in Android camera app is an open source one with Apache license(Sphere Panorama is an closed source in Android stock camera app). So we can leverage the source codes of wide angle panorama module in Android for FX OS.

iOS

Wide Angle Panorama:
Behaviour:
In panorama mode, the app will show a full screen preview and a small rectangle process indicator area. In this indicator area, you will see a smaller preview. Once you press the button, the app will ask you to turn the phone and keep the arrow of indicator area in middle line. You can stop the process at any time. I don’t hear any snap sound. But according to the pixel size, I think the app will take full size picture several times.

Picture size:
original:
3264*2448(8MP)

panorama:
7856*2332(18.3MP)

Nokia Panorama Camera (Windows Phone 8)

Wide Angle Panorama:
Behaviour:
Take at least 2 pictures. When you take first picture, the camera app will show a circle indicator to ask you hold your phone still for a while. Then the app take a snap. The app will show the next anchor point in preview. You need to move your phone to the point and hold still for the next shot. If you move too far, the app will ask you to align the preview with previous image. You will see a lighter previous image over current preview.

Picture size:
original:
3552*2000(8MP)

panorama:
3 pictures:
7926*1943(16MP)

Android

Wide Angle Panorama:

Behaviour:
When you turn the mode to wide angle panorama, the preview screen will resize to ½ size. The app will show a indicator below the preview screen. This indicator will display the progress. If you turn the phone to wrong angle, the app will show a warning. When you turn the phone to the end of progress. It will stop the progress indicator.
Picture size:
original:
2448*3264(8MP)

panorama:
2908*736

Photo Sphere Panorama:

Behaviour:
When you turn on the app, I will show small preview with a circle in the center. The background looks like you are in a room. You can see the tile floor to remain your phone position. In the middle of preview screen, you need to align the circle with the anchor point for starting. Once you align the point, you need to keep you phone still for a while.Then the phone will take a picture with snap sound. Then you will see 4 anchor points around the preview screen. You can move to the next anchor point for next picture. Then you press the stop button to end this shot.

Picture size:
original:
2448*3264(8MP)

panorama:
4 pictures: 2502*3187

Analysis

The proposed plan is trying to leverage the source codes of wide angle panorama module in Android. So we need to analysis how Android works and then to analysis related Gecko changes in FX OS. I will focus on how the input/output interface works in wide angle panorama module of Android's camera app.

How Android work

Please see below diagram. The blue block is C++(OpenGL) source code. The yellow blocks are JNI layer. The Green blocks are Java parts.

Mosaic C++ library

Mosaic Renderer

An OpenGL part for rendering stitched picture to preview buffer(Surface Texture).

Mosaic

Below description is the comments in Mosaic.h. Basically it use CPU to computes the whole process. "The class Mosaic provides a simple interface to the panoramic mosaicing algorithm. The class allows passing in individual image frames to be stitched together, computes the alignment transformation between them, and then stitches and blends them together into a single panoramic output which can then be accessed as a single image."

JNI layer

We might rewrite or replace this layer since we use C++ in Gecko.

Java layer

WideAnglePanoramaModule

The primary component to control all things. Take mCameraTexture(A SurfaceTexture) as the input from camera preview and bind this graphic buffer to OpenGL texture. Then the graphic buffer will be copied to CPU memory for Mosaic. The render target is a SurfaceTexture in mUI(WideAnglePanoramaUI). It can be bound to Android view system.

MosaicFrameProcessor

A singleton to handle the processing of each frame by Mosaicer. This class do the jobs including processing the last filled image frame through the mosaicer, updating the UI to show progress and generating final mosaic.

SurfaceTextureRender

A bridge to bind mosaic_render in C++ with Java EGL context. It creates EGLSurface from SurfaceTexture in WideAnglePanoramaUI.

The Impact to Current Gecko Camera Design

Current Gecko Camera from mikeh

In current Gecko camera design, we directly create a GonkNativeWindow for camera preview. When the preview frame is updated, the Gecko gets current buffer in GonkNativeWindow and update the GrallocImage in ImageContainer in |nsGonkCameraControl::OnNewPreviewFrame|. We need to insert the Mosaic Render into this work flow.

So we might need to extend the abilities of GonkNativeWindow including binding textures and update textures. Also we need to create EGLImage from EGLSurface and set this data to image container.

Design

Web API

Panorama Module Selection => Set to Panorama Mode.
Start Capture => Start Panorama capture.
Progress update => A Callback to notify Gaia camera app current progress.
Stop Capture => Stop Panorama capture.
Generated Finalized Preview Blob => A callback for passing generated finalized preview blob.
onframeavailable in MediaStreamTrack => Update every frame to Panorama module.

Gecko API for 3rd party library

Preview Input => Take a GonkNativeWindow to bind texture.
Preview Output => Take another GonkNativeWindow as render output. And take this GonkNativeWindow to create EGLSurface.
Update Progress Callback => Update current progress while starting capture.
Generated Picture => Generate finalized mosaic picture.
Error Handling => Some error handling functions.

Software Building Blocks

Sample Codes

Camera Panorama:

var dipManager = navigator.mozDIPManager;
var panoramaModule =  dipManager.panoramaModule;
navigator.getUserMedia({video:true, audio:false}, function(localMediaStream){
	var dipManager = navigator.mozDIPManager;
	panoramaModule =  dipManager.panoramaModule;
	var video = document.querySelector(‘video’);
	video.src = widnow.URL.createObjectURL(panoramaModule.outputStream);
	localMediaStream.onframeavailable = function(image) {
		panoramaModule.qequeImage(image);
	};
	panoramaModule.onprogressstatuschanged = function(progress) {
		console.log(“Current Progress: ” + progress);
	};
	panoramaModule.ongeneratedimage = function(blob) {
		// Save Image Blob to JPEG.
	};
	panoramaModule.onerror = function(error) {
		// Error Handling.
	};
	}, null);

function start() {
	panoramaModule.start();
}

function stop() {
	panoramaModule.stop();
}

PanoramaCameraDesign

Contents