Loop/Data Collection: Difference between revisions

Jump to navigation Jump to search

Loop/Data Collection (view source)

Revision as of 20:03, 14 August 2014

775 bytes removed ,  14 August 2014
→‎User Issue Reports
Line 162: Line 162:
When the user indicates an issue, the Loop client will create a ZIP file containing information that potentially pertains to the failure. For the initial set of reports, it will include the following files:
When the user indicates an issue, the Loop client will create a ZIP file containing information that potentially pertains to the failure. For the initial set of reports, it will include the following files:


# '''index.txt''': Simple text file containing a JSON object. This JSON object contains indexable meta-information relating to the failure report. The format is described in [[#Index_Format]], below.
# '''index.json''': Simple text file containing a JSON object. This JSON object contains indexable meta-information relating to the failure report. The format is described in [[#Index_Format]], below.
# '''sdk_log.txt''': All entries in the Browser Console emitted by chrome://browser/content/loop/*
# '''sdk_log.txt''': All entries in the Browser Console emitted by chrome://browser/content/loop/*
# '''stats.txt''': JSON object containing JSON serialization of stats object
# '''stats.txt''': JSON object containing JSON serialization of stats object
Line 172: Line 172:


=== Index Format ===
=== Index Format ===
The "index.txt" file MUST be the first file in the ZIP file, and MUST use compression method of "stored" (uncompressed). This can be easily achieved by passing a compression level of 0 to [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIZipWriter nsIZipWriter] for this file. These properties are critical to allow for rapid indexing of the reports.
The report IDs will be formed by appending a UUID to the string "le-".
The report IDs will be formed by appending a UUID to the string "le-".


Line 249: Line 247:
To that end, we will be using Microsoft Azure blob storage for the report upload and storage. The Loop client will perform PUT requests directly against into a blob container. Azure includes an access control mechanism that allows servers to hand out time-limited signed URLs that can then be used to access the indicated resource.
To that end, we will be using Microsoft Azure blob storage for the report upload and storage. The Loop client will perform PUT requests directly against into a blob container. Azure includes an access control mechanism that allows servers to hand out time-limited signed URLs that can then be used to access the indicated resource.


When a user indicates an issue, the Loop client selects a unique issue ID, and contacts the Loop server asking for a new URL to store the issue report in (including the date on which the report was generated):
When a user indicates an issue, the Loop client collects the data from the index.json file (including a unique issue ID selected by the client), and contacts the Loop server asking for a new URL to store the issue report in:


  POST /issue-report HTTP/1.1
  POST /issue-report HTTP/1.1
Line 257: Line 255:
   
   
  {
  {
   "id": "le-13b09e3f-0839-495e-a9b0-1e917d983766",
   "id": "le-4b42e9ff-5406-4839-90f5-3ccb121ec1a7",
   "timestamp": "1407958471"
   "timestamp": "1407784618",
  "phase": "midcall",
  "type": "incoming",
  "client": "builtin",
  "channel": "aurora",
  "version": "33.0a1",
  "callId": "35e7c3a511f424d3b1d6fba442b3a9a5",
  "apiKey": "44669102",
  "sessionId": "1_MX40NDY2OTEwMn5-V2VkIEp1bCAxNiAwNjo",
  "sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaW",
  "simplePushURL": "https://push.services.mozilla.com/update/MGlYke2SrEmYE8ceyu",
  "loopServer": "https://loop.services.mozilla.com",
  "callerId": "adam@example.com",
  "callurl": "http://hello.firefox.com/nxD4V4FflQ",
  "dndMode": "available",
  "reason": "quality",
  "comment": "The video is showing up about one second after the audio",
  "okToContact": "true",
  "contactEmail": "adam@example.org",
  }
  }


''Note: the ID is client-provided rather than server-generated to allow for the "retry-after" throttling behavior described below''
''Note: the ID is client-provided rather than server-generated to allow for the "retry-after" throttling behavior described below.''
 
The Loop server forms a blob storage URL to upload the information to. The fields are constructed as follows:
 
* The host is the Azure instance assigned to Mozilla's account
* The container is "loop-" followed by a four-digit year and two-digit month (e.g., if the report date sent by the client in its POST request falls in August of 2014, UTC, then the container would be named "loop-201408").
* The filename is {id}.zip, using the id field provided by the client.
* The "signedversion" field (sv) is the Azure API version we're currently using
* The "signedexpiry" field (se) is the current time plus five minutes (this simply needs to be long enough to upload the report)
* The "signedresource" field (sr) is "b" (blob storage)
* The "signedpermissions" field (sp) is "w" (write only)
* The "signature" field (sig) is computed with our Azure shared key, [http://msdn.microsoft.com/en-us/library/azure/dn140255.aspx as described by the Azure SAS documentation]


This URL is then returned to the user:
Upon receiving such a request, the Loop server performs the following steps:
 
# Checks that the client does not need to be throttled,
HTTP/1.1 200 OK
# Stores the received fields in an Azure table, and
Access-Control-Allow-Methods: GET,POST
# Forms a blob storage URL to upload the information to
Access-Control-Allow-Origin: https://localhost:3000
Content-Type: application/json; charset=utf-8
{
  "issueURL": "https://mozilla.blob.core.windows.net/loop-201408/le-13b09e3f-0839-495e-a9b0-1e917d983766.zip?sv=2012-02-12&se=2014-08-13T08%3a49Z&sr=b&sp=w&sig=Rcp6gQRfV7WDlURdVTqCa%2bqEArnfJxDgE%2bKH3TCChIs%3d"
}


==== Report Throttling ====
To mitigate potential abuse, the Loop server needs to throttle handing out issue URLs on a per-IP basis. If a Loop client attempts to send a request more frequently than the throttle allows, then the Loop server will send an HTTP 429 response indicating how long the client must wait before submitting the report. The client will then re-attempt sending the report once that period has passed.
To mitigate potential abuse, the Loop server needs to throttle handing out issue URLs on a per-IP basis. If a Loop client attempts to send a request more frequently than the throttle allows, then the Loop server will send an HTTP 429 response indicating how long the client must wait before submitting the report. The client will then re-attempt sending the report once that period has passed.


Line 300: Line 300:
  }
  }


This means that, upon startup, the Loop client code needs to check for outstanding (not-yet-upoaded) reports, and attempt to send them.  If a report is over 30 days old and has not been successfully uploaded, clients will delete the report. The Loop server will similarly check that the issueDate field is no older than 30 days, and will reject the request for an upload URL.
This means that, upon startup, the Loop client code needs to check for outstanding (not-yet-uploaded) reports, and attempt to send them.  If a report is over 30 days old and has not been successfully uploaded, clients will delete the report. The Loop server will similarly check that the issueDate field is no older than 30 days, and will reject the request for an upload URL.
 
==== Data Index Storage ====


Once it aqcuires an issue upload URL, the loop client then performs a PUT to the supplied URL to upload the report zipfile. The [http://msdn.microsoft.com/en-us/library/azure/dd135733.aspx Azure REST API documentation] contains more detailed information about this operation.
(TBD: basically, the Loop server stores the JSON index object in an Azure table so that developers can search for reports by various criteria. If the client-provided report ID already exists and a file exists in blob storage, the client gets an error)


=== Data Indexing ===
HTTP/1.1 409 Conflict
Access-Control-Allow-Methods: GET,POST
Access-Control-Allow-Origin: https://localhost:3000
Content-Type: application/json; charset=utf-8
{
  "code": "409",
  "errno": "115", ''// or whatever is allocated for this use''
  "error": "Conflict",
  "message": "Report ID already in use"
}


Periodically (proposal: every hour), a job will traverse any new files that have appeared in the current month's container as well as the previous month's container and extract the information for placing in an index. Data extraction is performed using two ranged GET requests.
If the client receives such an error, it should assume that the report was previously uploaded successfully without being removed for its own local storage, and deletes the local file.


To help understand the procedure that follows, the ZIP file will begin with the following fields (see [http://www.pkware.com/documents/casestudies/APPNOTE.TXT the PKWARE documentation for full details]):
==== Upload URL Generation ====
{| class="wikitable"
The blob URL fields are constructed as follows:
|-
! !! +0 !! +1
|-
! Byte 0
| rowspan = 2 colspan = 2 | ''Local file header signature (0x04034b50)''
|-
! Byte 2
|-
! Byte 4
| colspan = 2 | Version needed to extract
|-
! Byte 6
| colspan = 2 | General purpose bit flag
|-
! Byte 8
| colspan = 2 | ''Compression method''
|-
! Byte 10
| colspan = 2 | Last modfied time
|-
! Byte 12
| colspan = 2 | Last modified date
|-
! Byte 14
| rowspan = 2 colspan = 2 | CRC-32
|-
! Byte 16
|-
! Byte 18
| rowspan = 2 colspan = 2 | ''Compressed size''
|-
! Byte 20
|-
! Byte 22
| rowspan = 2 colspan = 2 | ''Uncompressed size''
|-
! Byte 24
|-
! Byte 26
| colspan = 2 | ''Filename Length''
|-
! Byte 28
| colspan = 2 | ''Extra field length''
|-
! Byte 30 ...
| colspan = 2 | <br>Filename (variable length)<br>&nbsp;
|-
! Byte 30 +<br>filename length
| colspan = 2 | <br>Extra Field (variable length)<br>&nbsp;
|-
! Byte 30 +<br>filename length +<br>extra field length
| colspan = 2 | <br>''File Contents''<br>&nbsp;
|-
|}


* The host is the Azure instance assigned to Mozilla's account
* The container is "loop-" followed by a four-digit year and two-digit month (e.g., if the report date sent by the client in its POST request falls in August of 2014, UTC, then the container would be named "loop-201408").
* The filename is {id}.zip, using the id field provided by the client.
* The "signedversion" field (sv) is the Azure API version we're currently using
* The "signedexpiry" field (se) is the current time plus five minutes (this simply needs to be long enough to upload the report)
* The "signedresource" field (sr) is "b" (blob storage)
* The "signedpermissions" field (sp) is "w" (write only)
* The "signature" field (sig) is computed with our Azure shared key, [http://msdn.microsoft.com/en-us/library/azure/dn140255.aspx as described by the Azure SAS documentation]


First, the indexing job retrieves the first 30 bytes of the file, and performs the following steps:
This URL is then returned to the user:
* Verify signature == 0x04034b50
* Verify compression method == 0
* Verify compressed size == uncompressed size
* Verify compressed size < 8 kB (these should typically be ~1 kB in size)
* Set index_file_start = 30 + file_name_length + extra_field_length
* Read range of bytes from start to start + compressed_size (again, using a ranged GET request)
* Perform a JSON parse of the resulting body


If any of the verification or parsing steps fail, then the indexing job deletes the blob from the container.
HTTP/1.1 200 OK
Access-Control-Allow-Methods: GET,POST
Access-Control-Allow-Origin: https://localhost:3000
Content-Type: application/json; charset=utf-8
{
  "issueURL": "https://mozilla.blob.core.windows.net/loop-201408/le-13b09e3f-0839-495e-a9b0-1e917d983766.zip?sv=2012-02-12&se=2014-08-13T08%3a49Z&sr=b&sp=w&sig=Rcp6gQRfV7WDlURdVTqCa%2bqEArnfJxDgE%2bKH3TCChIs%3d"
}


Once the parsing is complete, the indexing job then adds an entry to the index table (location TBD -- should we put this in S3?), to allow developers to search for reports by specific criteria.
Once it acquires an issue upload URL, the loop client then performs a PUT to the supplied URL to upload the report zipfile. The [http://msdn.microsoft.com/en-us/library/azure/dd135733.aspx Azure REST API documentation] contains more detailed information about this operation.


=== Data Retrieval ===
=== Data Retrieval ===
Line 388: Line 354:


=== Data Purging ===
=== Data Purging ===
At the end of each month, a data retention job will locate containers with a name indicating that they exceed the data retention policy for these reports (proposal: 6 months), and remove the containers (including all contained reports). This job also removes corresponding data from the index table.
At the end of each month, a data retention job will locate containers with a name indicating that they exceed the data retention policy for these reports (proposal: 6 months), and remove the containers (including all contained reports). This job also removes corresponding data from the index table. (tbd: table removal can probably be optimized through the use of partitions -- I need to look into Azure table storage a bit more).
AdamRoach
Confirmed users
632

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1005879"

Navigation menu