Firefox OS/Remote Control: Difference between revisions
Luke-chang (talk | contribs) |
Luke-chang (talk | contribs) |
||
| Line 32: | Line 32: | ||
[[File:RemoteControl SecureConnectionInitialization.png]] | [[File:RemoteControl SecureConnectionInitialization.png]] | ||
# | # The user opens the URL in the client (e.g. mobile browser). | ||
# Client requests page without valid UUID. | # Client requests the page without a valid UUID. | ||
# Server | # Server responses the content of the "secure.html" as the requested page for key exchange. | ||
# Client | # Client requests the RSA public key from "secure.sjs" via AJAX. ([[Firefox_OS/Remote_Control#RSA_public_key_exchange|Detail]]) | ||
# Secure.sjs | # "Secure.sjs" responses the public key (RSA-OEAP, exported by SPKI) in base64. ([https://github.com/MDTsai/gecko-dev/blob/Bug_1235013_new_httpserver/b2g/remotecontrol/secure.sjs#L80 source]) | ||
# Client imports RSA public key, | # Client imports the RSA public key ([https://github.com/luke-chang/gaia/blob/1228262_tv_remote_control_secure/tv_apps/remote-control-client/js/secure.js#L95-L113 source]), generates a symmetric key (AES-GCM) ([https://github.com/luke-chang/gaia/blob/1228262_tv_remote_control_secure/tv_apps/remote-control-client/js/secure.js#L115-L146 source]) and wraps it by the public key (RSA-OEAP) ([https://github.com/luke-chang/gaia/blob/1228262_tv_remote_control_secure/tv_apps/remote-control-client/js/secure.js#L180-L208 source]). | ||
# Client sends the wrapped symmetric key to secure.sjs in base64. [[Firefox_OS/Remote_Control#Send_symmetric_key|Detail]] | # Client sends the wrapped symmetric key to "secure.sjs" in base64 via AJAX. ([[Firefox_OS/Remote_Control#Send_symmetric_key|Detail]]) | ||
# Secure.sjs | # "Secure.sjs" generates a ticket number (a random number for polling only) and responses it to the client immediately. | ||
# Secure.sjs unwraps the symmetric key with RSA-OEAP | # "Secure.sjs" asynchronously unwraps the symmetric key with the private key (RSA-OEAP). ([https://github.com/MDTsai/gecko-dev/blob/Bug_1235013_new_httpserver/b2g/remotecontrol/secure.sjs#L30 source]) | ||
# Secure.sjs saves the symmetric key. | # "Secure.sjs" saves the symmetric key along with a corresponding UUID generated by Remote Control Service. | ||
# Remote Control Service | # Remote Control Service encrypts the UUID with its symmetric key, binds it with the ticket number and waits for polling. (The encrypted UUID makes the client be able to confirm if the server stores the symmetric key correctly.) ([https://github.com/MDTsai/gecko-dev/blob/Bug_1235013_new_httpserver/b2g/remotecontrol/secure.sjs#L30 source]) | ||
# Client | # Client keeps polling the encrypted UUID from "secure.sjs" with the ticket number. ([[Firefox_OS/Remote_Control#Poll_UUID|Detail]]) | ||
# | # Once receiving the encrypted UUID, the client decrypts it and stores it in the cookie. | ||
# Client reloads the URL with the valid UUID. | # Client reloads the URL with the valid UUID in the cookie. | ||
=== PIN code pairing === | === PIN code pairing === | ||
Revision as of 08:36, 28 January 2016
Introduction
Remote control is a new feature for Firefox OS on TV. We hope user can control TV at any device without installing any APP. Consider compatibility, remote control uses HTTP and AJAX as communication protocol between TV and user. User can easily connect to TV via open his/her browser, enter URL on TV and use the virtual touchpad to control TV just in seconds.
Meta Bug: bug 1205939
Prototype Demo Video: http://bit.ly/1Fa2ST1
UX & Visual Spec
UX Spec: http://bit.ly/1jrWqgn
Visual Spec: http://bit.ly/1jrWQU3
Architecture Designs
- Remote Control Service: Entry point of remote control feature. Control service related logic and data in runtime.
- HTTP Server: Serves connections from user. Two types files can be served:
- Static file: Stored in Remote Control Client App, packaged in app://remote-control-client.gaiamobile.org
- Service script: Handle AJAX request and response for secure connection initialization, PIN code pairing and control event processing. Packaged in resource://gre/res/remotecontrol.
- Remote Control App: Provides service related user interface on TV. Communicate with Remote Control Service via MozSettings.
User interaction of remote control is divided into three parts:
- Establish secure connection
- PIN code pairing (optional, by user settings)
- Control event processing
Establish secure connection
Remote control is designed to protect data transmitted between client and server, for example: user input string may contains privacy data. Consider user scenario for remote control is in local network, TV is not able to get a unique name to provide HTTPS; remote control uses proprietary protocol to establish secure connection. Client is forced to exchange symmetric key (AES-GCM) with server in the first time. After key exchange, server uses UUID in cookie to identify client and key to decrypt messages. Following describes how keys are exchanged:
- The user opens the URL in the client (e.g. mobile browser).
- Client requests the page without a valid UUID.
- Server responses the content of the "secure.html" as the requested page for key exchange.
- Client requests the RSA public key from "secure.sjs" via AJAX. (Detail)
- "Secure.sjs" responses the public key (RSA-OEAP, exported by SPKI) in base64. (source)
- Client imports the RSA public key (source), generates a symmetric key (AES-GCM) (source) and wraps it by the public key (RSA-OEAP) (source).
- Client sends the wrapped symmetric key to "secure.sjs" in base64 via AJAX. (Detail)
- "Secure.sjs" generates a ticket number (a random number for polling only) and responses it to the client immediately.
- "Secure.sjs" asynchronously unwraps the symmetric key with the private key (RSA-OEAP). (source)
- "Secure.sjs" saves the symmetric key along with a corresponding UUID generated by Remote Control Service.
- Remote Control Service encrypts the UUID with its symmetric key, binds it with the ticket number and waits for polling. (The encrypted UUID makes the client be able to confirm if the server stores the symmetric key correctly.) (source)
- Client keeps polling the encrypted UUID from "secure.sjs" with the ticket number. (Detail)
- Once receiving the encrypted UUID, the client decrypts it and stores it in the cookie.
- Client reloads the URL with the valid UUID in the cookie.
PIN code pairing
PIN code is a mechanism to ensure the device connects to TV is physical near TV. While connects to TV, user needs to enter 4 digits PIN code on TV screen. After pass pairing, user is redirected to control page and not need to enter PIN code until: 1) 90 days, pairing expired or 2) user revokes all pairing in remote control app. Pairing can be turned off in remote control app. Following describes how PIN code pairing is done:
- User opens the URL.
- Client requests page with UUID requires pairing.
- Server get the UUID and generate a PIN code.
- Remote control service sends PIN code to Gaia system app to trigger pairing notification on screen. (send chrome event)
- Server returns pairing.html as requested page for PIN code pairing.
- User enter PIN code on screen.
- Client encrypts PIN code.
- Client sends PIN code to pairing.sjs. Detail
- Pairing.sjs replies a ticket number.
- Pairing.sjs decrypts PIN code using symmetric key received in establish secure connection. source
- Pairing.sjs confirms PIN code correct from remote control service and delete pending PIN code.
- Pairing.sjs requests Gaia system app to dismiss pairing notification on screen. (send chrome event)
- Client queries PIN code pairing result from pairing.sjs by ticket number. Detail
- Client reloads the URL with UUID.
After PIN code pairing, client reloads the URL with a valid UUID and get control page.
Control event processing
Control page provides: 1) a virtual touchpad, 2) a virtual vertical scrollbar, 3) 3 function keys. Every touch, swipe or click will be generated to an event in JSON format, encrypted and send to TV. Following describes how control event is sent and processed:
- User opens the URL.
- Client requests page with UUID can access control page.
- Server returns client.html as requested page for remote control.
- User operates control page.
- Client generates an event and encrypts the event using symmetric key. source
- Client sends encrypted event.
- Client.sjs responses with latest event result.
- Client.sjs decrypts event with symmetric key received in establish secure connection. source
- Client.sjs parses event and dispatch to Gecko or Gaia system app.
Data encryption/decryption
There are three kinds of data are encrypted while tranmission:
- Client polls UUID in establish secure connection stage
- Client sends PIN code in pin code pairing
- Client sends event in control event processing
As data are encrypted by symmetric key, AES-GCM, there are two things need noticed:
- AES-GCM need initialization vector(IV) to encrypt/decrypt. Currently, IV is random value for each data, length is 12 bytes. Encrypted data is appended after IV. Receiver needs to slice first 12 bytes as IV to decrypt.
- Each message type is string, use TextEncoder.encode() to encode as an UInt8Array, then encrypt to ArrayBuffer. Vice versa.
- Data encryption/decryption is asynchronous but HTTP request need response immediately. For UUID and PIN code case, remote control uses a ticket number to get status after decryption. But for event, we reduce polling result but use latest event's result instead.
Ajax Protocol
RSA public key exchange
Request
{
action: 'require-public-key'
}
Response
{
publicKey: <RSA-OEAP SPKI in base64>
}
{
error: <reason>
}
Send symmetric key
Request
{
action: 'send-symmetric-key',
wrappedSymmetricKey: <AES-GCM wrapped by RSA-OEAP in base64>
}
Response
{
ticket: <ticket>
}
{
error: <reason>
}
Poll UUID
Request
{
action: 'poll-uuid'
ticket: <ticket>
}
Response
{
done: true,
encryptedUUID: <IV + UUID encrypted by AES-GCM, in base64>,
}
{
done: true,
error: <reason>
}
{
done: false // Symmetic key unwrapping or encrypting UUID
}
Pair PIN code
Request
{
action: 'pair-pincode'
encryptedPIN: <IV + PIN encrypted by AES-GCM in base64>
}
Response
{
ticket: <ticket>
}
Poll pair result
Request
{
action: 'poll-pair-result'
ticket: <ticket>
}
Response
{
done: true
verified: <boolean>
reason: <reason>
}
{
done: false
}
Encrypted control events
Client sends encrypted event in query string as:
<IP_Address>:<Port>?message=<IV + encrypted event by AES-GCM in base64>
Response
{
verified: <boolean>
}
After decrypted, following are control events:
Touch Events
{
type: 'touchstart',
detail: {
width: <touch panel width, integer, in pixels>,
height: <touch panel height, integer, in pixels>,
identifier: <A simple unique identifier for server to distinguish touch events between different clients, string>
}
}
{
type: 'touchmove',
detail: {
dx: <dx between current point and starting point, integer, in pixels>,
dy: <dy between current point and starting point, integer, in pixels>,
identifier: <should be identical with "touchstart">,
duration: <duration from "touchstart", integer, in milliseconds>
}
}
{
type: 'touchend',
detail: {
dx: <same as "touchmove">,
dy: <same as "touchmove">,
identifier: <should be identical with "touchstart">,
duration: <same as "touchmove">,
swipe: <the gesture within this touch, string, should be "left", "right", "up" and "down">
}
}
Scroll Events
There are "scrollstart", "scrollmove" and "scrollend" events which details are all identical with corresponding touch events above.
Key Events
{
type: 'keypress',
detail: <KeyEvent constant, string, sush as "DOM_VK_RETURN">
}
Input Events
{
type: 'input',
detail: {
clear: <whether to clear the entire string in the current focused input field, boolean>,
string: <new string to append, string>,
keycode: <a specified key to be pressed after the string inputted, integer>
}
}
Custom Events
{
type: 'custom',
action: <custom action name, string>,
...
}
Pairing (Full version, without secure connection)
Meta Bug: bug 1207996
Flowchart
First Time Connection (deprecated in secure connection)
Enter PIN Code (deprecated in secure connection)
Resume Connection (deprecated in secure connection)
Dismiss Pairing
Protocol
from client to server via AJAX (deprecated in secure connection)
{
pincode: <pincode>
}
response when success (deprecated in secure connection)
{
verified: true,
uuid: <uuid>
}
reponse when error
{
verified: false,
reason: 'expired' / 'invalid'
}
internal events at server side
{
type: 'mozChromeRemoteControlEvent',
detail: {
action: 'pin-created',
pincode: <pincode>
}
}
{
type: 'mozChromeRemoteControlEvent',
detail: {
action: 'pin-destroyed'
}
}
{
type: 'mozContentEvent',
detail: {
type: 'remote-control-pin-dismissed',
detail: {
reason: 'timeout' / 'manually'
}
}
}
Secure Connection (deprecated)
To protect private data between user and TV, we provide a secure connection which refers to SSL and jCryption. Here is the concept:
- Client requests RSA public key from TV.
- Client sends symmetric key and PIN code (if any) to TV, encrypted with public key.
- TV decrypts symmetric key with private key
- TV generates an UUID, encrypts with symmetric key then sends 2 UUIDs to client, one is encrypted the other is not.
- Client decrypts UUID with symmetric key, confirm UUID
- Begin remote control with symmetric key.
Goal of Remote Control is to create an easy to use way for every device, we hide public key exchange at the background. User doesn't need to input a long URL with public key nor scan QRCode. Everything is done automatically without interrupting user experience.
Bug Status
Top blockers of bug 1205939 - [TV][2.5][meta][TV side] Remote Control
19 Total; 0 Open (0%); 19 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1207971 - [TV 2.5][meta] Provide a TV remote control app
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1203853 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | Provide a QR code on the remote control app on TV | RESOLVED | --- | 2.5+ |
| 1208003 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the functionalities of the TV remote control app | RESOLVED | --- | 2.5+ |
| 1212402 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Apply visual spec to the TV remote control app | RESOLVED | --- | --- |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1208006 - [TV 2.5][meta] Implement a client-side page for TV remote control
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1212403 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Apply visual spec to the client-side page of TV remote control | RESOLVED | --- | --- |
| 1212407 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the functionalities of the client-side page of TV remote control | RESOLVED | --- | --- |
| 1215075 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement a pairing page on the client-side | RESOLVED | --- | --- |
| 1218682 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Improve the input function in remote control client by one-way sync | RESOLVED | --- | --- |
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1212385 - [TV 2.5][meta] Remote Control features in Smart System
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1203045 | P2 | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Dispatch input messages via mozInputMethod API in server side | RESOLVED | --- | 2.5+ |
| 1203048 | P1 | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Draw a mock mouse cursor in the smart-system app | RESOLVED | --- | --- |
| 1207995 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Provide remote control service current control mode | RESOLVED | FxOS-S10 (30Oct) | 2.5+ |
| 1214993 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] "sendKeyEvent" doesn't work when sending "DOM_VK_HOME" to trigger homescreen | RESOLVED | --- | --- |
| 1215076 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the remote control pairing notification in TV system | RESOLVED | --- | --- |
5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1215457 - [TV 2.5] Secure connection between remote control server and client
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1207996 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV] Implement JPAKE authentication over TLS socket server | RESOLVED | --- | --- |
| 1228262 | P1 | Gaia::TV | C.M.Chang[:chunmin] | [TV] Implement secure connection on remote control client side | RESOLVED | --- | --- |
| 1235013 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV 2.5] Implement secure connection on remote control server side | RESOLVED | --- | --- |
| 1235019 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV 2.5] Implement clickjacking protection in remote control server side | RESOLVED | --- | --- |
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);