B2G/QA/Ping Testing

From MozillaWiki
< B2G‎ | QA
Jump to: navigation, search

Metrics Testing for FxOS 2.2

Test Server Setup

  1. Sign up for an account at https://www.runscope.com
  2. After registration, you will be redirect to the dashboard. Click the "URL Helper" at the left-hand side menu.
    Click "URL Helper
  3. Enter the FTU_PING_URL in gaia's Makefile, for example : https://fxos.telemetry.mozilla.org/submit/telemetry, and press Enter
    Insert the FTU_PING_URL
  4. Copy the generated Runscope URL, similar to this: https://fxos-telemetry-mozilla-org-xxxxxxxxxxxx.runscope.net/submit/telemetry
Copy the generated URL
Notice: By using the URL, the Runscope server acts as a proxy to record the traffic. The ping will eventually reach the actual fxos telemetry server. If you don't want the ping to reach the production server, use the http://xxxxxxxxxxxx.runscope.net URL instead.

FTU Ping

  1. Clone the gaia repository: git clone https://github.com/mozilla-b2g/gaia.git
  2. Modify the following files in gaia:
    Makefile
    - FTU_PING_URL?=https://fxos.telemetry.mozilla.org/submit/telemetry
    + FTU_PING_URL?=https://fxos-telemetry-mozilla-org-xxxxxxxxxxxx.runscope.net/submit/telemetry
    (Tip: do not leave tailing spaces here. Otherwise the ping will fail)
    apps/system/js/ftu_ping.js
    - var DEBUG = false;
    + var DEBUG = true;//enable debug message in adb logcat

    - const DEFAULT_TRY_INTERVAL = 60 * 60 * 1000;
    + const DEFAULT_TRY_INTERVAL = 15 * 1000; //15 seconds
  3. (option) If you want to test FTU ping on foxfooding devices, then you have to modify the file additionally. The Ping ID for foxfooder is 15 digit ID and non-foxfooder is 4-byte hex ID.
    Makefile
    - DOGFOOD?=0
    + DOGFOOD?=1
  4. Flash the gaia and start adb log right away MOZILLA_OFFICAL=1 make production; adb logcat |grep -i -E 'ftuping|appusage'

App Usage Ping

  1. Clone the gaia repository: git clone https://github.com/mozilla-b2g/gaia.git
  2. Modify the following files in gaia:
    Makefile
    - FTU_PING_URL?=https://fxos.telemetry.mozilla.org/submit/telemetry
    + FTU_PING_URL?=https://fxos-telemetry-mozilla-org-xxxxxxxxxxxx.runscope.net/submit/telemetry
    (Tip: do not leave tailing spaces here. Otherwise the ping will fail)
    apps/system/js/app_usage_metrics.js
    [TV] tv_apps/smart-system/js/app_usage_metrics.js
    - AUM.DEBUG = false;
    + AUM.DEBUG = true; //enable debug message in adb logcat

    - AUM.REPORT_URL = 'https://fxos.telemetry.mozilla.org/submit/telemetry';
    + AUM.REPORT_URL = 'https://fxos-telemetry-mozilla-org-xxxxxxxxxxxx.runscope.net/submit/telemetry';

    - AUM.REPORT_INTERVAL = 14 * 24 * 60 * 60 * 1000; // 2 weeks
    + AUM.REPORT_INTERVAL = 60 * 1000; // 60 seconds

    - AUM.RETRY_INTERVAL = 60 * 60 * 1000; // 1 hour
    + AUM.RETRY_INTERVAL = 15 * 1000; // 15 second
    [TV] gaia/build/config/tv/settings.json
    + “debug.performance_data.shared”: true
    Debug Log
    Before this pull request landed on 2.2, you need to git apply this patch to see the correct adb log.
  3. (option) If you want to test App Usage on foxfooding devices, then you have to modify these setting additionally.
    Makefile
    - DOGFOOD?=0
    + DOGFOOD?=1
    apps/system/js/app_usage_metrics.js
    - AUM.REPORT_INTERVAL = result['metrics.appusage.reportInterval'];
    + AUM.REPORT_INTERVAL = 60 * 1000;
  4. Flash the gaia and start adb log right away MOZILLA_OFFICAL=1 make production; adb logcat |grep -i -E 'ftuping|appusage'

Verify the results

Verify the results
  • Use adb logcat |grep -i -E 'ftuping|appusage' to see the debug log while testing.
  • Log in to https://www.runscope.com, click the "Captures" at the left-hand side menu.
    • Click a POST request.
    • Verify its URL path complies with the spec.
    • Click the "Request" tab to see the request payload.

Metrics Testing for FxOS 2.1 and before

FTU Ping

Server Setup

  1. Clone the git repository here
  2. Setup an app engine project here
  3. Rename "mockmetricsserver" name in cloned code files to the name of your app engine project
  4. Install the Google App Engine SDK here
  5. Launch the Google App Engine launcher and add the existing project that points to your cloned code files
  6. Deploy the project you just added to app engine
  7. You should then see the server deployed at yourservername.appspot.com

Gaia Build Setup

  1. Flash the latest build for the version you are aiming to test
  2. Pull Gaia locally with git by running "git clone https://github.com/mozilla-b2g/gaia.git"
  3. Modify the FTU_PING_URL in the Makefile in the root directory to point to your deployed mock metrics server
  4. In the file apps/system/js/ftu_ping.js, make the following modifications based on your testing need:
    1. If you desire debug logging, then change the line of code that says "var DEBUG = false;" to "var DEBUG = true;"
    2. If you desire to change the time limit for how often a ping attempt happens, then change the line of code that says "const DEFAULT_TRY_INTERVAL = 60 * 60 * 1000;" to "const DEFAULT_TRY_INTERVAL = <your time limit in ms>;"
    3. If you desire to change how many attempts a ping will be made, then change the line of code that says "const DEFAULT_MAX_NETWORK_FAILS = 24; to "const DEFAULT_MAX_NETWORK_FAILS = <your # of attempts>;"
    4. If you desire to change the ping's timeout to connect send data to the metrics server, then change the line of code that says "DEFAULT_PING_TIMEOUT = 60 * 1000;" to "DEFAULT_PING_TIMEOUT = <your time limit in ms>;"
  5. Flash Gaia by running "MOZILLA_OFFICIAL=1 make production"

Running Test Cases

Once the FTU starts up, ping attempts will happen at each try interval (default = 1 hour) up to certain number of attempts (default = 24 attempts). When a ping attempt occurs, it will need the MCC/MNC off of each SIM card of the phone and the associated mobile network of each SIM card. If that information is known, then the ping attempt will be made towards a target telemetry server using a network connection. If the ping attempt is successful, then the ping will no longer be sent going forward. If the network connection either fails or the required SIM information wasn't found, then another attempt will be made at the next try interval. If we've reached the last attempt and SIM information wasn't found, then a ping will be attempted without the SIM information known.

Finding a Ping Test Result

  1. Go to admin console here
  2. Open the application for your mock metrics server
  3. Go to datastore viewer
  4. Select an ID of a request
  5. The body of the request you will see here contains the ping that was sent to the mock metrics server

Mock Server Test Cases

  • Test that I can send a ping immediately on the first attempt with mobile data known
  • Test that I can send a ping on the second attempt with mobile data known in which the first attempt failed due to no SIM card
  • Test that I can send a ping on the second attempt with mobile data known in which the first attempt failed due to no network
  • Test that I can send a ping on the final attempt with mobile data known
  • Test that I can send a ping on the final attempt with mobile data not known
  • Test that I can send pings for two distinct Firefox OS versions
  • Test that I can send pings for two distinct Firefox OS resolutions
  • Test that I can send pings for two distinct default locale builds
  • Test that I can send a ping that makes use of a Roaming SIM card
  • Test that I can send a ping that makes use of an inactive SIM card
  • Test that I can send a ping that makes use of a Pin Locked SIM card
  • Test that the same UUID is used for two pings on the same device within a 24 hour period
  • Test that I can send pings for two distinct build IDs
  • Test that I can send pings for two distinct update channels
  • Test that no additional pings are sent after a successful ping attempt is made
  • Test that I can send a ping on a DSDS-based device with two SIMs included references mobile data from the primary SIM
  • Test that I can send a ping on a DSDS-based device with a single SIM in the 2nd slot of a DSDS device

Resources

App Usage Ping

Server Setup

The steps are the same as FTU ping. But you need to use this modified mock server. (Please use the `app_usage` branch instead of the `master` branch.)

Gaia Build Setup

  1. Flash the latest build for the version you are aiming to test
  2. Pull Gaia locally with git by running "git clone https://github.com/mozilla-b2g/gaia.git"
  3. Modify the FTU_PING_URL in the Makefile in the root directory to point to your deployed mock metrics server
  4. In the file apps/system/js/app_usage_metrics.js, make the following modifications based on your testing need:
    1. If you desire debug logging, please change "AUM.DEBUG = true"
    2. Change the "AUM.BASE_URL" to the URL of metrics server.
    3. If you desire to change how often device try to send reports to server, please change "AUM.REPORT_INTERVAL = 60 * 60 * 1000;" to "AUM.REPORT_INTERVAL = <your time limit in ms>;"
    4. If you desire to change the timeout for sending report to the metrics server, please change the line of code that says "AUM.REPORT_TIMEOUT = 60 * 1000;" to "AUM.REPORT_TIMEOUT = <your time limit in ms>;"
    5. If you desire to change how long device wants to wait while device fails to send report at previous attemp, please change AUM.RETRY_INTERVAL to <your time limit in ms>.
  5. Flash Gaia by running "MOZILLA_OFFICIAL=1 make production"

Verifying the Test Result

  1. The spec can be found here
  2. The modified server has some problem when saving to the datastore. So please use the server log in Google App Engine to see the uploaded metrics.

Resources

Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Ping_Testing&oldid=1117414"