Hasal

About Hasal

Hasal /ha'sɑlu/

This word is came from Indigenous Taiwanese "Bunun", and it means hail. Why we choose this word is because we expect our testing could like the hail, fast and weighty. And through our testing could bring more improvement on our performance or quality.

This is a performance measurement project for web apps.

Contacts

Name	Email address
Google Group - Hasal Project	[hasal-project@mozilla.com Hasal Project]	Send email if you need any support
Shako Ho	sho@mozilla.com	You can contact him to get more project detail.
Askeing Yen	askeing@mozilla.com
Mike Lien	mlien@mozilla.com
Walter Chen	wachen@mozilla.com
Bobby Chien	bchien@mozilla.com	Program Management
Ken Chang	kchang@mozilla.com	Point of Contact for Engineering

State of this project

We are finish the testing for google doc in July. Now, we are shifting the focus to Facebook and Gmail.

Execution Summary

• Google doc: Total we have 39 cases tested, 34 test cases indicate Firefox slower than Chrome. 6 cases how 30% gap between Firefox and Chrome.
• Facebook: Pilot test result will come out before end of July.
• Gmail: Pilot test result will come out before end of July.

Testing Result

• Google doc testing result:

 You can find all testing result here Testing Result
 And the analysis diagram for module consume the most running time in each test cases could generate automatically like below:
 

Or you can download the XMind file here.

About this project

We are trying to measure the difference between Firefox and Chrome on high visibility web apps (Google Doc, Google Sheet, Google Slide, etc.)

We use the video recording to record the user actions and extract the frames to get the running time from user perspective. Based on same user actions, we measure the difference between Firefox and Chrome.

Project Plan

• • Resource Plan
  • Test Plan
  • Meeting minutes

Google Doc

Phase 1 (Done)

• Build up a workable framework that can run some manual created test cases
  • Trigger actions on browser
  • Record video on the background
  • Enable profiler on firefox automatically
  • Convert the video to image automatically
  • Compare images with sample images
• Measure the difference between Firefox and Chrome for sample user actions (40% found between Firefox and Chrome)

Phase 2

• Create test cases based on test matrices
• Create test scripts for high priority cases and execute them
• Generate profiler data for each testing result
• Go through the whole process from analyzing to provide information
• Framework improvement
  • Replace Selenium with Sikuli
  • Get the medium value and other statistics
• Support windows and mac

Phase 3

• Create scripts for medium and low priority cases and execute them.
• Generate profiler data for each testing result.

Working Measurements

Measurements

App Name	Meta Bugs	Reports	Status
Google Docs	1260981	Pilot, Full	User would aware performance issues on Firefox
Outlook Web Access (Office 365)	1275000	Pilot, Full	No big issue per pilot test
Outlook Web Access (Exchange)		Pilot, Full	Suspended, awaiting measurable builds
Facebook	1288602	Pilot, Full	Found issues, working on full test
Gmail	1288604	Pilot, Full	No big issue per pilot test
BBC		Pilot, Full
Youtube		Pilot, Full

Google Docs

Full Query

ID	Component	Summary	Status	Assigned to	Target milestone	Depends on
1278554	Graphics: Canvas2D	CanvasRenderingContext2D::DrawImage takes lots of time when scrolling Google Spreadsheet	NEW		---	1237058
1307024	General	[Perf] [google docs] Firefox will get 3~9% faster when running gdoc test cases with chrome user agent	NEW		---	1321623

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Outlook Web Access (Exchange)

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Facebook

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

GMail

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Testing Tool

Testing Framework

Following the steps here to setup the framework. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.

Template VM

You can use vagrant to download the template VM with Hasal's automation framework. Please follow the instruction below:

• vagrant init shako/hasal
• vagrant up --provider virtualbox

Testing Data Summarize Tool

Following the steps here to setup the tool. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.

Add-on

Performance mark on profiling results at pageMod start and end

You can install this add-on from Here

Sending Performance Mark by Hotkeys

You can install this add-on from Here