Firefox OS/Remote Control: Difference between revisions
Luke-chang (talk | contribs) |
(→Architecture Designs: add disconnect and reconnection description) |
||
| (57 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
Remote control is a feature for Firefox OS on TV. After [https://addons.mozilla.org/en-US/android/addon/send-tab-to-tv/ Send tab to TV], user can easily extend browsing experience on TV via Firefox on Android, using virtual touchpad to control TV. | |||
Meta Bug: {{Bug|1205939}} | Meta Bug: {{Bug|1205939}} | ||
Prototype Demo Video: http://bit.ly/1Fa2ST1 | |||
== UX & Visual Spec == | |||
UX Spec: http://bit.ly/1jrWqgn | |||
Visual Spec: http://bit.ly/1jrWQU3 | |||
== Architecture Designs == | == Architecture Designs == | ||
=== | [[File:RemoteControl BlockDiagram.png]] | ||
* [[Firefox_OS/Remote_Control_Service|Remote Control Service]]: Entry point of remote control feature. Control service related logic and data in runtime. | |||
* TLS server: Serves connections from user. | |||
User interaction of remote control is divided into two parts: | |||
# Peer authentication | |||
# Control event processing | |||
Any error occurs will close connection between server and client, reconnection is required. Server side error will sends message back to client to let user know what happen on Fennec. | |||
=== Peer authentication === | |||
Remote control is designed to protect data transmitted between client and server, for example: user input string may contains privacy data. The connection is based on self-signed TLS server. To prevent man in the middle(MITM) attack, we adopt [https://en.wikipedia.org/wiki/Password_Authenticated_Key_Exchange_by_Juggling J-PAKE] to exchange symmetric key for peer authentication. Following describes how it works: | |||
[[File:RemoteControl_FirstTimeHandshake.png]] | |||
# The user sends the tab to TV on fennec. | |||
# Client sends request to handshake without ID. | |||
# Server replies with the handshake type is 1st time handshake. | |||
# Client computes J-PAKE round 1 and sends the result to server. | |||
# Server generates PIN code and show on screen, computes J-PAKE round 1 and sends the result to client. | |||
# User enter PIN code, client computs J-PAKE round 2 with PIN code attached first 12 characters of server's TLS certificate fingerprint as weak secret. | |||
# Client sends round 2 result to server. | |||
# Server computes J-PAKE round 2 with PIN code attached first 12 characters of server's TLS certificate fingerprint as weak secret. | |||
# Server dismisses PIN code notification on screen, sends round 2 result to client. | |||
# Both client and server compute J-PAKE final round, gets AES and HMAC key. | |||
# Server use two keys to get a signature of H(H(AES key)), then send the signature to client. | |||
# Client verifies the signature matches it has. | |||
# Client use two keys to get a signature of H(AES key), then send the signature to server. | |||
# Server verifies the signature matches it has. | |||
# Server replies with the handshake finish with client's ID for connection in the future. | |||
==== Re-authentication ==== | |||
User is only required to input PIN code when first time connect to TV. In the second time, client and server leverage previous AES key value and derive new AES and HMAC key. This can reduce number of user input and make the procedure from send tab to TV to remote control more smoothly. | |||
[[File:RemoteControl SecondTimeHandshake.png]] | |||
# The user sends the tab to TV on fennec. | |||
# Client sends request to handshake with ID. | |||
# Server looks up ID/AES key mapping and replies with the handshake type is 2nd time handshake. | |||
# Client computes J-PAKE round 1 and sends the result to server. | |||
# Server computes J-PAKE round 1 and sends the result to client. | |||
# Client and server computes J-PAKE round 2 with first 4 characters of previous AES key value attached first 12 characters of server's TLS certificate fingerprint as weak secret. | |||
# Client sends round 2 result to server. | |||
# Server sends round 2 result to client. | |||
# Both client and server compute J-PAKE final round, gets new AES and HMAC key. | |||
# Server use two keys to get a signature of H(H(AES key)), then send the signature to client. | |||
# Client verifies the signature matches it has. | |||
# Client use two keys to get a signature of H(AES key), then send the signature to server. | |||
# Server verifies the signature matches it has. | |||
# Server replies with the handshake finish. | |||
After authentication, client can sends control event to server. | |||
=== Control event processing === | |||
Control page provides: 1) a virtual touchpad, 2) a virtual vertical scrollbar, 3) 3 function keys. Every touch, swipe or click will be generated to an event in JSON format then send to TV. Following describes how control event is sent and processed: | |||
[[File:RemoteControl ControlEventProcessing.png]] | |||
# User operates control page. | |||
# Client generates and sends the event in JSON format. | |||
# Server parses the event to JSON object, dispatch to client.sjs. | |||
# Client.sjs parses event and dispatch to Gecko or Gaia system app. | |||
=== Data used in JPAKE authentication === | |||
==== Singer ID for JPAKE round 1 & 2: ==== | |||
* TV: server | |||
* Fennec addon: client | |||
==== Weak secret: ==== | |||
* TV: concatenate PIN and first 12 characters of TLS server cert SHA 256 fingerprint | |||
* Fennec addon: concatenate user input PIN and first 12 characters connected TLS server cert SHA 256 fingerprint | |||
==== HMAC Input for JAPKE final: ==== | |||
We use "AES_256_CBC-HMAC256", as aHkdfInfo, includes the full crypto spec, should be the same in both TV and fennec addon | |||
==== Key confirmation: ==== | |||
Double hash of AES key: | |||
## TV converts AES key to array buffer | |||
## Sign AES key array buffer, get signature 1 (array buffer) | |||
## Sign signature 1, get signature 2 | |||
## Convert signature 2 to base 64, send to Fennec addon | |||
## Fennec addon do the same as TV from step 1 to 4 | |||
## Compare received base 64 string of self and TV's signature 2 | |||
Single hash of AES key: | |||
## Fennec addon converts AES key to array buffer | |||
## Sign AES key array buffer, get signature 1 (array buffer) | |||
## Convert signature 1 to base 64, send to TV | |||
## TV use HMAC key, received signature, AES key array buffer to verify if the signature is valid. | |||
=== Authentication and Event Protocol === | |||
==== Request handshake ==== | |||
Request | |||
{ | |||
type: 'auth' | |||
action: 'request_handshake' | |||
detail: { | |||
id: <id assigned by server, optional> | |||
} | |||
} | |||
Response | |||
{ | |||
type: 'auth' | |||
action: 'response_handshake' | |||
detail: 1 or 2, 1 for 1st handshake, 2 for 2nd handshake | |||
} | |||
==== J-PAKE key exchange ==== | |||
Client send round 1 | |||
{ | |||
type: 'auth' | |||
action: 'jpake_client_1', | |||
detail: { | |||
gx1: gx1.value, | |||
gx2: gx2.value, | |||
zkp_x1: { gr: gv1.value, b: r1.value, id: 'client' }, | |||
zkp_x2: { gr: gv2.value, b: r2.value, id: 'client' } | |||
} | |||
} | |||
Server reply round 1 | |||
{ | |||
type: 'auth' | |||
action: 'jpake_server_1', | |||
detail: { | |||
gx1: gx1.value, | |||
gx2: gx2.value, | |||
zkp_x1: { gr: gv1.value, b: r1.value, id: 'server' }, | |||
zkp_x2: { gr: gv2.value, b: r2.value, id: 'server' } | |||
} | |||
} | |||
Client send round 2 | |||
{ | |||
type: 'auth' | |||
action: 'jpake_client_2', | |||
detail: { | |||
A: A.value, | |||
zkp_A: { gr: gvA.value, b: rA.value, id: 'client' } | |||
} | |||
} | |||
Server reply round 2 | |||
{ | |||
type: 'auth' | |||
action: 'jpake_server_2', | |||
detail: { | |||
A: A.value, | |||
zkp_A: { gr: gvA.value, b: rA.value, id: 'server' } | |||
} | |||
} | |||
==== Key confirmation ==== | |||
Server key confirmation | |||
{ | |||
type: 'auth' | |||
action: 'server_key_confirm' | |||
detail: { | |||
signature: <double signature of AES key by HMAC key, in base64> | |||
} | |||
} | |||
Client key confirmation | |||
{ | |||
type: 'auth' | |||
action: 'client_key_confirmation' | |||
detail: { | |||
signature: <signature of AES key by HMAC key, in base64> | |||
} | |||
} | |||
Server finish handshake | |||
{ | |||
type: 'auth' | |||
action: 'finish_handshake' | |||
detail: { | |||
id: <id assigned by server, optional> | |||
} | |||
} | |||
==== Touch Events ==== | ==== Touch Events ==== | ||
{ | { | ||
type: 'touchstart', | type: 'command' | ||
action: 'touchstart', | |||
detail: { | detail: { | ||
width: <touch panel width, integer, in pixels>, | width: <touch panel width, integer, in pixels>, | ||
height: <touch panel height, integer, in pixels> | height: <touch panel height, integer, in pixels>, | ||
identifier: <A simple unique identifier for server to distinguish touch events between different clients, string> | |||
} | } | ||
} | } | ||
{ | { | ||
type: 'touchmove', | type: 'command' | ||
action: 'touchmove', | |||
detail: { | detail: { | ||
dx: <dx between current point and starting point, integer, in pixels>, | dx: <dx between current point and starting point, integer, in pixels>, | ||
dy: <dy between current point and starting point, integer, in pixels>, | dy: <dy between current point and starting point, integer, in pixels>, | ||
identifier: <should be identical with "touchstart">, | |||
duration: <duration from "touchstart", integer, in milliseconds> | duration: <duration from "touchstart", integer, in milliseconds> | ||
} | } | ||
| Line 26: | Line 215: | ||
{ | { | ||
type: 'touchend', | type: 'command' | ||
action: 'touchend', | |||
detail: { | detail: { | ||
dx: <same as "touchmove">, | dx: <same as "touchmove">, | ||
dy: <same as "touchmove">, | dy: <same as "touchmove">, | ||
identifier: <should be identical with "touchstart">, | |||
duration: <same as "touchmove">, | duration: <same as "touchmove">, | ||
swipe: <the gesture within this touch, string, should be "left", "right", "up" and "down"> | swipe: <the gesture within this touch, string, should be "left", "right", "up" and "down"> | ||
| Line 42: | Line 233: | ||
{ | { | ||
type: 'keypress', | type: 'command' | ||
action: 'keypress', | |||
detail: <KeyEvent constant, string, sush as "DOM_VK_RETURN"> | detail: <KeyEvent constant, string, sush as "DOM_VK_RETURN"> | ||
} | } | ||
| Line 49: | Line 241: | ||
{ | { | ||
type: ' | type: 'command' | ||
action: 'textinput', | |||
detail: { | detail: { | ||
clear: <whether to clear the entire string in the current focused input field, boolean>, | clear: <whether to clear the entire string in the current focused input field, boolean>, | ||
string: <new string to append, string> | string: <new string to append, string>, | ||
keycode: <a specified key to be pressed after the string inputted, integer> | |||
} | } | ||
} | } | ||
=== | ==== Server reply error ==== | ||
{ | |||
type: common, or the event type sent from client | |||
error: <error message of exception or root cause> | |||
} | |||
== Bug Status == | == Bug Status == | ||
| Line 109: | Line 289: | ||
{ | { | ||
"blocks":["1212385"], | "blocks":["1212385"], | ||
"status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"], | |||
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone, cf_feature_b2g" | |||
} | |||
</bugzilla> | |||
Top blockers of {{Bug|1215457}} - [TV 2.5] Secure connection between remote control server and client | |||
<bugzilla> | |||
{ | |||
"blocks":["1215457"], | |||
"status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"], | "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"], | ||
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone, cf_feature_b2g" | "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone, cf_feature_b2g" | ||
} | } | ||
</bugzilla> | </bugzilla> | ||
Latest revision as of 03:39, 24 May 2016
Introduction
Remote control is a feature for Firefox OS on TV. After Send tab to TV, user can easily extend browsing experience on TV via Firefox on Android, using virtual touchpad to control TV.
Meta Bug: bug 1205939
Prototype Demo Video: http://bit.ly/1Fa2ST1
UX & Visual Spec
UX Spec: http://bit.ly/1jrWqgn
Visual Spec: http://bit.ly/1jrWQU3
Architecture Designs
- Remote Control Service: Entry point of remote control feature. Control service related logic and data in runtime.
- TLS server: Serves connections from user.
User interaction of remote control is divided into two parts:
- Peer authentication
- Control event processing
Any error occurs will close connection between server and client, reconnection is required. Server side error will sends message back to client to let user know what happen on Fennec.
Peer authentication
Remote control is designed to protect data transmitted between client and server, for example: user input string may contains privacy data. The connection is based on self-signed TLS server. To prevent man in the middle(MITM) attack, we adopt J-PAKE to exchange symmetric key for peer authentication. Following describes how it works:
- The user sends the tab to TV on fennec.
- Client sends request to handshake without ID.
- Server replies with the handshake type is 1st time handshake.
- Client computes J-PAKE round 1 and sends the result to server.
- Server generates PIN code and show on screen, computes J-PAKE round 1 and sends the result to client.
- User enter PIN code, client computs J-PAKE round 2 with PIN code attached first 12 characters of server's TLS certificate fingerprint as weak secret.
- Client sends round 2 result to server.
- Server computes J-PAKE round 2 with PIN code attached first 12 characters of server's TLS certificate fingerprint as weak secret.
- Server dismisses PIN code notification on screen, sends round 2 result to client.
- Both client and server compute J-PAKE final round, gets AES and HMAC key.
- Server use two keys to get a signature of H(H(AES key)), then send the signature to client.
- Client verifies the signature matches it has.
- Client use two keys to get a signature of H(AES key), then send the signature to server.
- Server verifies the signature matches it has.
- Server replies with the handshake finish with client's ID for connection in the future.
Re-authentication
User is only required to input PIN code when first time connect to TV. In the second time, client and server leverage previous AES key value and derive new AES and HMAC key. This can reduce number of user input and make the procedure from send tab to TV to remote control more smoothly.
- The user sends the tab to TV on fennec.
- Client sends request to handshake with ID.
- Server looks up ID/AES key mapping and replies with the handshake type is 2nd time handshake.
- Client computes J-PAKE round 1 and sends the result to server.
- Server computes J-PAKE round 1 and sends the result to client.
- Client and server computes J-PAKE round 2 with first 4 characters of previous AES key value attached first 12 characters of server's TLS certificate fingerprint as weak secret.
- Client sends round 2 result to server.
- Server sends round 2 result to client.
- Both client and server compute J-PAKE final round, gets new AES and HMAC key.
- Server use two keys to get a signature of H(H(AES key)), then send the signature to client.
- Client verifies the signature matches it has.
- Client use two keys to get a signature of H(AES key), then send the signature to server.
- Server verifies the signature matches it has.
- Server replies with the handshake finish.
After authentication, client can sends control event to server.
Control event processing
Control page provides: 1) a virtual touchpad, 2) a virtual vertical scrollbar, 3) 3 function keys. Every touch, swipe or click will be generated to an event in JSON format then send to TV. Following describes how control event is sent and processed:
- User operates control page.
- Client generates and sends the event in JSON format.
- Server parses the event to JSON object, dispatch to client.sjs.
- Client.sjs parses event and dispatch to Gecko or Gaia system app.
Data used in JPAKE authentication
Singer ID for JPAKE round 1 & 2:
- TV: server
- Fennec addon: client
Weak secret:
- TV: concatenate PIN and first 12 characters of TLS server cert SHA 256 fingerprint
- Fennec addon: concatenate user input PIN and first 12 characters connected TLS server cert SHA 256 fingerprint
HMAC Input for JAPKE final:
We use "AES_256_CBC-HMAC256", as aHkdfInfo, includes the full crypto spec, should be the same in both TV and fennec addon
Key confirmation:
Double hash of AES key:
- TV converts AES key to array buffer
- Sign AES key array buffer, get signature 1 (array buffer)
- Sign signature 1, get signature 2
- Convert signature 2 to base 64, send to Fennec addon
- Fennec addon do the same as TV from step 1 to 4
- Compare received base 64 string of self and TV's signature 2
Single hash of AES key:
- Fennec addon converts AES key to array buffer
- Sign AES key array buffer, get signature 1 (array buffer)
- Convert signature 1 to base 64, send to TV
- TV use HMAC key, received signature, AES key array buffer to verify if the signature is valid.
Authentication and Event Protocol
Request handshake
Request
{
type: 'auth'
action: 'request_handshake'
detail: {
id: <id assigned by server, optional>
}
}
Response
{
type: 'auth'
action: 'response_handshake'
detail: 1 or 2, 1 for 1st handshake, 2 for 2nd handshake
}
J-PAKE key exchange
Client send round 1
{
type: 'auth'
action: 'jpake_client_1',
detail: {
gx1: gx1.value,
gx2: gx2.value,
zkp_x1: { gr: gv1.value, b: r1.value, id: 'client' },
zkp_x2: { gr: gv2.value, b: r2.value, id: 'client' }
}
}
Server reply round 1
{
type: 'auth'
action: 'jpake_server_1',
detail: {
gx1: gx1.value,
gx2: gx2.value,
zkp_x1: { gr: gv1.value, b: r1.value, id: 'server' },
zkp_x2: { gr: gv2.value, b: r2.value, id: 'server' }
}
}
Client send round 2
{
type: 'auth'
action: 'jpake_client_2',
detail: {
A: A.value,
zkp_A: { gr: gvA.value, b: rA.value, id: 'client' }
}
}
Server reply round 2
{
type: 'auth'
action: 'jpake_server_2',
detail: {
A: A.value,
zkp_A: { gr: gvA.value, b: rA.value, id: 'server' }
}
}
Key confirmation
Server key confirmation
{
type: 'auth'
action: 'server_key_confirm'
detail: {
signature: <double signature of AES key by HMAC key, in base64>
}
}
Client key confirmation
{
type: 'auth'
action: 'client_key_confirmation'
detail: {
signature: <signature of AES key by HMAC key, in base64>
}
}
Server finish handshake
{
type: 'auth'
action: 'finish_handshake'
detail: {
id: <id assigned by server, optional>
}
}
Touch Events
{
type: 'command'
action: 'touchstart',
detail: {
width: <touch panel width, integer, in pixels>,
height: <touch panel height, integer, in pixels>,
identifier: <A simple unique identifier for server to distinguish touch events between different clients, string>
}
}
{
type: 'command'
action: 'touchmove',
detail: {
dx: <dx between current point and starting point, integer, in pixels>,
dy: <dy between current point and starting point, integer, in pixels>,
identifier: <should be identical with "touchstart">,
duration: <duration from "touchstart", integer, in milliseconds>
}
}
{
type: 'command'
action: 'touchend',
detail: {
dx: <same as "touchmove">,
dy: <same as "touchmove">,
identifier: <should be identical with "touchstart">,
duration: <same as "touchmove">,
swipe: <the gesture within this touch, string, should be "left", "right", "up" and "down">
}
}
Scroll Events
There are "scrollstart", "scrollmove" and "scrollend" events which details are all identical with corresponding touch events above.
Key Events
{
type: 'command'
action: 'keypress',
detail: <KeyEvent constant, string, sush as "DOM_VK_RETURN">
}
Input Events
{
type: 'command'
action: 'textinput',
detail: {
clear: <whether to clear the entire string in the current focused input field, boolean>,
string: <new string to append, string>,
keycode: <a specified key to be pressed after the string inputted, integer>
}
}
Server reply error
{
type: common, or the event type sent from client
error: <error message of exception or root cause>
}
Bug Status
Top blockers of bug 1205939 - [TV][2.5][meta][TV side] Remote Control
19 Total; 0 Open (0%); 19 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1207971 - [TV 2.5][meta] Provide a TV remote control app
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1203853 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | Provide a QR code on the remote control app on TV | RESOLVED | --- | 2.5+ |
| 1208003 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the functionalities of the TV remote control app | RESOLVED | --- | 2.5+ |
| 1212402 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Apply visual spec to the TV remote control app | RESOLVED | --- | --- |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1208006 - [TV 2.5][meta] Implement a client-side page for TV remote control
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1212403 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Apply visual spec to the client-side page of TV remote control | RESOLVED | --- | --- |
| 1212407 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the functionalities of the client-side page of TV remote control | RESOLVED | --- | --- |
| 1215075 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement a pairing page on the client-side | RESOLVED | --- | --- |
| 1218682 | -- | Gaia::TV | Luke Chang [:lchang] (inactive) | [TV 2.5] Improve the input function in remote control client by one-way sync | RESOLVED | --- | --- |
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1212385 - [TV 2.5][meta] Remote Control features in Smart System
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1203045 | P2 | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Dispatch input messages via mozInputMethod API in server side | RESOLVED | --- | 2.5+ |
| 1203048 | P1 | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Draw a mock mouse cursor in the smart-system app | RESOLVED | --- | --- |
| 1207995 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Provide remote control service current control mode | RESOLVED | FxOS-S10 (30Oct) | 2.5+ |
| 1214993 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] "sendKeyEvent" doesn't work when sending "DOM_VK_HOME" to trigger homescreen | RESOLVED | --- | --- |
| 1215076 | -- | Gaia::TV::System | Luke Chang [:lchang] (inactive) | [TV 2.5] Implement the remote control pairing notification in TV system | RESOLVED | --- | --- |
5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);
Top blockers of bug 1215457 - [TV 2.5] Secure connection between remote control server and client
| ID | Priority | Component | Assigned to | Summary | Status | Target milestone | Feature b2g |
|---|---|---|---|---|---|---|---|
| 1207996 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV] Implement JPAKE authentication over TLS socket server | RESOLVED | --- | --- |
| 1228262 | P1 | Gaia::TV | C.M.Chang[:chunmin] | [TV] Implement secure connection on remote control client side | RESOLVED | --- | --- |
| 1235013 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV 2.5] Implement secure connection on remote control server side | RESOLVED | --- | --- |
| 1235019 | -- | Gaia::TV::System | Eric Tsai (no more review request accepted) | [TV 2.5] Implement clickjacking protection in remote control server side | RESOLVED | --- | --- |
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);