TestEngineering/Performance/NimbleDroid

At a glance

NimbleDroid provides functional performance testing for Android and iOS via a service. We currently use NimbleDroid exclusively for testing the performance of our Android applications on real devices.

Application	Upload	#Cold startup	#Page load
Fennec	Not Done	✅	[TODO]
GeckoView Example	Manual	✅	✅
Reference Browser	Every Push to Master	✅	[TODO]
Fenix	Every Push to Master	✅	✅
Chrome	Manual	✅	✅

Test scenarios

Cold startup

The cold startup scenario is profiled for every app. See the NimbleDroid documentation for details on what and how they measure for cold startup.

Page load

Custom scenarios are used to test page load for the following sites:

1. https://abcnews.go.com
2. https://amazon.com
3. https://amazon.com/s/ref=nb_sb_noss_2/139-6317191-5622045?url=search-alias%3Daps&field-keywords=mobile+phone
4. https://blogger.com/about
5. https://booking.com
6. https://booking.com/searchresults.en-gb.html?city=20088325
7. https://buzzfeed.com/daves4/the-new-york-times-makes-the-nerdiest-correction-e?utm_term=.tlL2KE08L6#.hwrBglaV2A
8. https://cnn.com
9. https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other
10. https://edition.cnn.com/ampstories/us/still-missing-cassini-heres-what-else-is-out-there
11. https://en.m.wikipedia.org/wiki/Main_Page
12. https://en.m.wikipedia.org/wiki/Mozilla_Corporation
13. https://eurosport.de
14. https://eurosport.de/fussball/fifa-wm/2018/wm-2018-spielplan-russland-alle-dfb-spiele-wm-2018-gruppen-wm-2018-termine_sto6691356/story.shtml
15. https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470&regionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2
16. https://fashionbeans.com
17. https://fashionbeans.com/article/coolest-menswear-stores-in-the-world
18. https://glassdoor.com/index.htm
19. https://glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=true&clickSource=searchBtn&typedKeyword=Mozilla&sc.keyword=Mozilla&locT=C&locId=1147431&jobType=
20. https://google.com
21. https://google.com/maps?force=pwa
22. https://google.com/search?q=restaurants+near+me
23. https://guardian.co.uk
24. https://healthyway.com
25. https://instagram.com
26. https://m.360.cn
27. https://m.ebay-kleinanzeigen.de
28. https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331
29. https://m.facebook.com
30. https://m.facebook.com/Cristiano
31. https://m.ranker.com
32. https://m.ranker.com/list/hunger-games-book-vs-movie-comparisons/lisa-waugh?ref=browse_list_5&l=2&pos=1
33. https://m.vk.com
34. https://m.weibo.cn
35. https://m.yelp.com
36. https://m.yelp.de
37. https://m.yelp.de/biz/burgeramt-berlin-2
38. https://marvel.wikia.com/wiki/Black_Panther
39. https://medium.com
40. https://medium.com/s/coincidences-are-a-lie/could-america-have-also-been-the-birthplace-of-impressionism-cb3d31a2e22d
41. https://people.com/amp-stories/royal-a-to-z
42. https://rumble.com/v3c44t-foxes-jumping-on-my-trampoline.html
43. https://sbnation.com/2018/2/12/16719476/college-football-playoff-rankings-explained
44. https://sima.com.cn
45. https://tripadvisor.com
46. https://tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html
47. https://urbandictionary.com
48. https://urbandictionary.com/define.php?term=awesome%20sauce
49. https://washingtonpost.com
50. https://washingtonpost.com/news/the-switch/wp/2017/06/06/amazon-kickstarter-reddit-and-mozilla-are-staging-a-net-neutrality-online-protest/?noredirect=on&utm_term=.c2cc1fa4862a
51. https://wired.com
52. https://wired.com/story/vehicle-future-bike
53. https://youtube.com
54. https://zalando.de
55. https://zalando.de/herrenschuhe/?exclusive=true

Uploading apps

Automated

[TODO]

Manual

[TODO]

Results

The test results can be seen in Firefox Health and NimbleDroid's dashboard.

Alerting

[TODO]