TestEngineering/Performance/NimbleDroid

From MozillaWiki
Jump to: navigation, search

At a glance

NimbleDroid provides functional performance testing for Android and iOS via a service. We currently use NimbleDroid exclusively for testing the performance of our Android applications on real devices.

Application Upload #Cold startup #Page load
Fennec None  ?  ?
GeckoView Example #Manual
Reference Browser #Automated (push to master)  ?
Fenix #Automated (push to master)
Chrome #Manual

Test scenarios

Cold startup

The cold startup scenario is profiled for every app. See the NimbleDroid documentation for details on what and how they measure for cold startup.

Page load

Custom scenarios are used to test page load for the following sites:

  1. https://abcnews.go.com
  2. https://amazon.com
  3. https://amazon.com/s/ref=nb_sb_noss_2/139-6317191-5622045?url=search-alias%3Daps&field-keywords=mobile+phone
  4. https://blogger.com/about
  5. https://booking.com
  6. https://booking.com/searchresults.en-gb.html?city=20088325
  7. https://buzzfeed.com/daves4/the-new-york-times-makes-the-nerdiest-correction-e?utm_term=.tlL2KE08L6#.hwrBglaV2A
  8. https://cnn.com
  9. https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other
  10. https://edition.cnn.com/ampstories/us/still-missing-cassini-heres-what-else-is-out-there
  11. https://en.m.wikipedia.org/wiki/Main_Page
  12. https://en.m.wikipedia.org/wiki/Mozilla_Corporation
  13. https://eurosport.de
  14. https://eurosport.de/fussball/fifa-wm/2018/wm-2018-spielplan-russland-alle-dfb-spiele-wm-2018-gruppen-wm-2018-termine_sto6691356/story.shtml
  15. https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470&regionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2
  16. https://fashionbeans.com
  17. https://fashionbeans.com/article/coolest-menswear-stores-in-the-world
  18. https://glassdoor.com/index.htm
  19. https://glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=true&clickSource=searchBtn&typedKeyword=Mozilla&sc.keyword=Mozilla&locT=C&locId=1147431&jobType=
  20. https://google.com
  21. https://google.com/maps?force=pwa
  22. https://google.com/search?q=restaurants+near+me
  23. https://guardian.co.uk
  24. https://healthyway.com
  25. https://instagram.com
  26. https://m.360.cn
  27. https://m.ebay-kleinanzeigen.de
  28. https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331
  29. https://m.facebook.com
  30. https://m.facebook.com/Cristiano
  31. https://m.ranker.com
  32. https://m.ranker.com/list/hunger-games-book-vs-movie-comparisons/lisa-waugh?ref=browse_list_5&l=2&pos=1
  33. https://m.vk.com
  34. https://m.weibo.cn
  35. https://m.yelp.com
  36. https://m.yelp.de
  37. https://m.yelp.de/biz/burgeramt-berlin-2
  38. https://marvel.wikia.com/wiki/Black_Panther
  39. https://medium.com
  40. https://medium.com/s/coincidences-are-a-lie/could-america-have-also-been-the-birthplace-of-impressionism-cb3d31a2e22d
  41. https://people.com/amp-stories/royal-a-to-z
  42. https://rumble.com/v3c44t-foxes-jumping-on-my-trampoline.html
  43. https://sbnation.com/2018/2/12/16719476/college-football-playoff-rankings-explained
  44. https://sima.com.cn
  45. https://tripadvisor.com
  46. https://tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html
  47. https://urbandictionary.com
  48. https://urbandictionary.com/define.php?term=awesome%20sauce
  49. https://washingtonpost.com
  50. https://washingtonpost.com/news/the-switch/wp/2017/06/06/amazon-kickstarter-reddit-and-mozilla-are-staging-a-net-neutrality-online-protest/?noredirect=on&utm_term=.c2cc1fa4862a
  51. https://wired.com
  52. https://wired.com/story/vehicle-future-bike
  53. https://youtube.com
  54. https://zalando.de
  55. https://zalando.de/herrenschuhe/?exclusive=true

Uploading apps

Automated

This is how we upload apks to Nimbledroid for our nightly runs.

#!/bin/bash

APK="target.apk"

cd /tmp
rm -f "${APK}"

curl -L -O "https://index.taskcluster.net/v1/task/project.mobile.fenix.v2.nightly.latest/artifacts/public/build/arm/${APK}" || exit 1

curl -F apk="@${APK}" -F device_config="android5" -v -u SECRETHEREXXX: https://nimble.app/api/v2/apks || exit 1

For Reference Browser and Geckoview, see here

For Fenix, see here

Manual

Documentation

Results

The test results can be seen in Firefox Health and NimbleDroid's dashboard.

Alerting

[TODO]