MozillaWiki - User contributions [en]

Firefox/Projects/Startup Time Improvements

2009-12-10T14:25:01Z

Joel Reymont: /* Measuring Ts */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** [http://blog.mozilla.com/dmandelin/2009/02/26/tracevis-performance-visualization-for-tracemonkey/ dmandelin's blog post]
** traceviz ({{bug|497999}})
* [[Firefox/Projects/Startup_Time_Improvements/Static_Build|static build]]

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<pre>
#!/bin/sh

FF=/Volumes/Fujitsu80Gb

diskutil unmount force $FF
diskutil mount /dev/disk1s2

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground -P clean file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</pre>

This assumes that your removable media is at /dev/disk1s2.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

Firefox/Projects/Startup Time Improvements/Static Build

2009-12-04T21:49:43Z

Joel Reymont:

= Overview =
'''Description'''

A static build nets us a ~15% startup performance win on the Mac. A static build breaks XPCOM components, though, so a solution needs to be provided.

'''Bugs'''
* {{bug|525013}} - investigate a more static build configuration of Firefox
** {{bug|532531}} - make binary components work on the Mac
** {{bug|532769}} - get a Linux static build going
** {{bug|532763}} - make binary components work on Linux
** {{bug|532771}} - get a Windows static build going
** {{bug|532765}} - make binary components work on Windows

'''Results'''

There are no issues on Mac or Linux but Windows seems to build a dependency on firefox.exe into the components (see {{bug|532765}}).

We definitely want to use _-mdynamic-no-pic_ on the Mac. At one point in time this resulted in a 50% startup boost when running from an external USB HD. The next day we were down to a more reasonable 15%.

'''Areas of work'''

Results definitely need to be verified on all platforms. Need to fix our cold startup harness on the Mac to fully clear the OS caches ({{bug|532965}}).

Firefox/Projects/Startup Time Improvements/Static Build

2009-12-04T21:40:16Z

Joel Reymont: Created page with '= Overview = '''Description:''' A static build nets us a ~15% startup performance win on the Mac. A static build breaks XPCOM components, though, so a solution needs to be prov…'

= Overview =
'''Description:'''

A static build nets us a ~15% startup performance win on the Mac. A static build breaks XPCOM components, though, so a solution needs to be provided.

'''Bugs:'''
* {{bug|525013}} - investigate a more static build configuration of Firefox
** {{bug|532531}} - make binary components work on the Mac
** {{bug|532769}} - get a Linux static build going
** {{bug|532763}} - make binary components work on Linux
** {{bug|532771}} - get a Windows static build going
** {{bug|532765}} - make binary components work on Windows

'''Results'''

There are no issues on Mac or Linux but Windows seems to build a dependency on firefox.exe into the components (see {{bug|532765}}).

We definitely want to use _-mdynamic-no-pic_ on the Mac. At one point in time this resulted in a 50% startup boost when running from an external USB HD. The next day we were down to a more reasonable 15%.

'''Areas of work'''

Results definitely need to be verified on all platforms. Need to fix our cold startup harness on the Mac to fully clear the OS caches ({{bug|532965}}).

Firefox/Projects/Startup Time Improvements

2009-12-04T21:39:37Z

Joel Reymont: /* Projects */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** traceviz ({{bug|497999}})
* [[Firefox/Projects/Startup_Time_Improvements/Static_Build|static build]]

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<pre>
#!/bin/sh

FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</pre>

This assumes that your removable media is at /dev/disk1s2.

First, mount you external USB HD normally and use <code>df</code> to see what device is being used. Then eject the drive using <code>diskutil umount /Volumes/YourDrive</code> and you are set.

Do not eject the device using the Finder as that will not make it available at /dev/disk1s2 or wherever it was.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

Firefox/Projects/Startup Time Improvements

2009-12-04T21:13:42Z

Joel Reymont: /* Projects */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** traceviz ({{bug|497999}})
* [[Firefox/Projects/Startup_Time_Improvements/StaticBuild|static build]]

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<pre>
#!/bin/sh

FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</pre>

This assumes that your removable media is at /dev/disk1s2.

First, mount you external USB HD normally and use <code>df</code> to see what device is being used. Then eject the drive using <code>diskutil umount /Volumes/YourDrive</code> and you are set.

Do not eject the device using the Finder as that will not make it available at /dev/disk1s2 or wherever it was.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

Firefox/Projects/Startup Time Improvements

2009-12-01T19:38:54Z

Joel Reymont: /* Measuring Ts */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** traceviz ({{bug|497999}})

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<pre>
#!/bin/sh

FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</pre>

This assumes that your removable media is at /dev/disk1s2.

First, mount you external USB HD normally and use <code>df</code> to see what device is being used. Then eject the drive using <code>diskutil umount /Volumes/YourDrive</code> and you are set.

Do not eject the device using the Finder as that will not make it available at /dev/disk1s2 or wherever it was.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

Firefox/Projects/Startup Time Improvements

2009-12-01T18:58:41Z

Joel Reymont: /* Measuring Ts */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** traceviz ({{bug|497999}})

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<pre><code>
#!/bin/sh

FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code></pre>

This assumes that your removable media is at /dev/disk1s2.

First, mount you external USB HD normally and use <code>df</code> to see what device is being used. Then eject the drive using <code>diskutil umount /Volumes/YourDrive</code> and you are set.

Do not eject the device using the Finder as that will not make it available at /dev/disk1s2 or wherever it was.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

Firefox/Projects/Startup Time Improvements

2009-12-01T18:56:57Z

Joel Reymont: /* Measuring Ts */

= Overview =
'''Project lead:''' Dietrich Ayala 
'''Project members:''' Ryan Flint, Drew Willcoxon, David Dahl, Vlad Vukićević, Taras Glek, Joel Reymont, Ben Hsieh

'''Description:''' Investigate areas for improving startup time of the Mozilla platform, find & fix issues that help Fennec and WinCE.

'''Bugs:'''
* [http://tinyurl.com/ygbnzmd Open] (identified by "[ts]" in the whiteboard field)
* [http://tinyurl.com/yjp8d6w Fixed]

'''Results'''

See results in the [http://people.mozilla.org/~dietrich/snapshot/ Performance Snapshot].

= Projects =

'''Areas of Work:'''
* minification
** {{bug|524858}} - minify js
** {{bug|513132}} - minify css
** {{bug|523479}} - minify extension's contents
* file combining
** {{bug|507101}} - combining js components
** {{bug|509755}} - put modules into a jar
* {{bug|FILEME}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=279839#c29 string pool for jsxpcom]
* taking advantage of tracing
** figure out where we're not tracing in the startup log
** traceviz ({{bug|497999}})

'''Measurement & Testing'''
* cold startup testing
** <strike>{{bug|510587}} - mac and linux</strike>
** {{bug|522807}} - windows
* dirty profile testing
** <strike>{{bug|414660}} - infrastructure & places</strike>
** {{bug|515540}} - dirty+cold startup
** {{bug|527596}} - dirty Tp (teehee)
** {{bug|FILEME}} - test an all-dirty profile
** {{bug|FILEME}} - other dirty profiles
* micro-benchmarking
** {{bug|523146}} - test framework
** {{bug|479528}} - for location bar
** {{bug|451607}} - for places UI
** {{bug|479528}} - for places SQL
** more, eg: test for opening the 80th tab
* built-in startup measurement ({{bug|522375}})
** user-runnable Ts script, paste into js console
** restarts browser, passing last known time
** loads data URI that reads startup time and prints it out
** record startup timestamp as a pref?
* timeline of the startup path (David, Vlad)
** Status: incomplete (held up on hangs, nspr changes)
** [https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements/StartupPatchUsage#Quick_Start docs]
** {{bug|480735}} instrumentation
** {{bug|503605}} visualization ([http://mozilla.ddahl.com/startup/timeline-similie.html Simile timeline], [http://people.mozilla.org/~dietrich/ddahl_sample_output.txt raw log])

'''Docs:'''
* write best practices recommendation

'''Inactive Projects'''

* Mine the Ts DTrace viz/logs (Drew)
** Status: complete, moved on to filesystem operations below
** Results
*** cold startup is heavily I/O-bound. 90% of cold startup time is spent in page faults and other I/O syscalls, especially read, getattrlist, and stat.
*** majority of time spent after Firefox actually gets up and running is in XBL and CSS.
* Log and mine filesystem operations during startup (Ryan, Drew)
** Status: complete, bugs filed for egregious offenders
** Results
*** checking out Xperf and others for Windows work
*** investigating dtrace timing/FS cache issues on OS X
**** Instruments appears to have the same issues
*** Simple JS file exec script has yielded [https://bugzilla.mozilla.org/buglist.cgi?bug_id=504513 some results]
** References
*** [http://boomswaggerboom.wordpress.com/2009/06/24/firefox-for-mac-os-x-and-filesystem-related-system-calls/ Josh Aas's post on filesystem calls]
*** [http://blog.mozilla.com/tglek/2009/07/01/python-gdb-logging-file-io/ Taras' post] on using GDB for logging
*** [[Firefox/Projects/Startup_Time_Improvements/IO_Scratchpad]]
*** [[Firefox/Projects/Startup_Time_Improvements/adw_notes|Drew's notes]]
*** [[Firefox/Projects/Startup_Time_Improvements/joelr_notes|Joel's notes]]
* Find dead code with JSHydra (David Dahl)
** Status: incomplete, needs tuning and file bugs on dead code found
** {{bug|506128}} - run jshydra to find js functions that have no callers
* JSHydra continuous testing
** possible test scenarios
*** ending semicolon bug (XXX)
*** dead code finder

=Status Updates=

* [http://autonome.wordpress.com/2009/10/02/firefox-startup-performance-weekly-summary-4/ 10/2/09]
* [http://autonome.wordpress.com/2009/09/26/firefox-startup-performance-weekly-summary-3/ 9/26/09]
* [http://autonome.wordpress.com/2009/09/18/firefox-startup-performance-weekly-summary-2/ 9/18/09]
* [http://autonome.wordpress.com/2009/09/06/firefox-startup-performance-weekly-summary/ 9/6/09]
* [http://autonome.wordpress.com/2009/08/29/firefox-startup-performance-weekly-roundup/ 8/29/09]
* [http://autonome.wordpress.com/2009/08/22/weekly-firefox-startup-performance-roundup-2/ 8/22/09]
* [http://autonome.wordpress.com/2009/08/18/firefox-performance-the-dont-touch-the-damn-disk-edition/ 8/18/09]
* [http://autonome.wordpress.com/2009/08/08/firefox-startup-performance/ 8/8/09]

= References =

;Meta bugs
* {{bug|459117}} Fennec performance meta bug
* {{bug|447581}} Startup meta bug
* {{bug|479078}} Startup research meta bug
* {{bug|7251}} very old startup meta bug

* Vlad's blog posts
** [http://blog.vlad1.com/2009/06/17/firefox-startup-wrangling/ Vlad's June posting on startup work]
** Vlad's July posts:
*** [http://blog.vlad1.com/2009/07/27/startup-part-1/ 1]
*** [http://blog.vlad1.com/2009/07/27/startup-again/ 2 - fs cache]
*** [http://blog.vlad1.com/2009/07/28/startup-again-2/ 3 - event processing]
*** [http://blog.vlad1.com/2009/07/28/measuring-startup/ 4 - measuring startup]
*** [http://blog.vlad1.com/2009/07/29/startup-again-3/ 5 iosnoop]

* zpao's JS treemap visualizations
** [http://playground.zpao.com/dtrace_treemaps2/ newer]
** [http://playground.zpao.com/dtrace_treemaps/ older]

* Taras' Fennec Startup Work
** {{bug|459117}} "make fennec faster", mostly startup bugs
** [http://people.mozilla.com/~tglek/fennec/startup.txt startup log]
** {{bug|470116}} various timing scripts

* [[Performance:Home_Page]]

==Tips, Tools==

* Cold startup
** [http://aegisknight.org/2009/04/flushing-disk-cache/ Flushing Windows' disk cache]
** [http://tuvix.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man8/purge.8.html Flushing Mac OS X's disk cache]
** [[Firefox/Projects/Startup_Time_Improvements_Notes]]: Vlad's emails about cold startup, including Linux (maybe move this info to this page)

* Windows Prefetch/Superfetch
** [http://en.wikipedia.org/wiki/Prefetcher XP Prefetch wiki]
** [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Vista/Win7 SuperFetch wiki]
** [http://members.rushmore.com/~jsky/id37.html useful Pre/SuperFetch overview]
** [http://jessekornblum.com/presentations/dodcc08-2.html Presentation slides on the guts of Pref/SuperFetch by military forensic analyst]
*** Prefetch
**** Profile for each application: List of DLLs used, Run count, Time stamps
**** Prefetch files stored in %SystemRoot%\Prefetch: .pf extension, Up to 128 are kept
**** Can tell if same app run from multiple locations
*** SuperFetch
**** turn off: net stop sysmain
**** AG*.db files, contain "scenarios" for preloading
** [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Technet article on Vista mem mgmnt]
** [http://www.howtogeek.com/howto/windows-vista/change-superfetch-to-only-cache-system-boot-files-in-vista/ registry keys for toggling SuperFetch]

* DTrace
** [https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace nuts and bolts how-to for getting dtrace running with mozilla]
** [http://www.opensolaris.org/os/project/mozilla-dtrace/ project overview of integrating dtrace with mozilla]
** [http://www.opensolaris.org/os/community/dtrace/ dtrace community page, lots of links]
** [http://www.sun.com/bigadmin/content/dtrace/ dtrace homepage? not as useful as previous link]
** [http://wikis.sun.com/display/DTrace/Introduction intro from dtrace guidebook, really useful but not mozilla/js-specific]
** [http://blogs.sun.com/jmr/entry/dtrace_mozilla_rfe_javascript_tracing brief descriptions of the javascript* probes]
** [http://blog.mozilla.com/dmandelin/2008/02/14/dtrace-c-mysteries-solved/ David Mandelin's notes on tracing C++ code]

* comparing dynamic linking time between firefox and safari on mac
** DYLD_PRINT_STATISTICS=YES /Applications/Firefox.app/Contents/MacOS/firefox-bin
** DYLD_PRINT_STATISTICS=YES open -a Safari

==Fastload==
* why does js_Execute take so long for fastloaded components?
* [http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/42c0e9df84b935fe what is fastload?]
* [http://kb.mozillazine.org/Nglayout.debug.disable_xul_fastload fastload pref docs]

==Measuring Ts==

We've been using a method Vlad blogged about [http://blog.vlad1.com/2009/07/28/measuring-startup/ here]. The steps are below.

# save [http://people.mozilla.com/~vladimir/misc/startup.html this html file] somewhere locally
# create a new profile, run Firefox with it at least once, so all files are created, etc
# execute this command, with your profile name substituted, and invoking firefox in a way appropriate for your platform:

<code>
./firefox -P YourProfileName -no-remote file:///..../startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

The following is specific to Mac OSX and is required to evict Firefox code pages from memory and clear the Unified Buffer Cache (UBC). <code>purge</code> does NOT do this.

<code>
FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`
</code>

This assumes that your removable media is at /dev/disk1s2.

First, mount you external USB HD normally and use <code>df</code> to see what device is being used. Then eject the drive using <code>diskutil umount /Volumes/YourDrive</code> and you are set.

Do not eject the device using the Finder as that will not make it available at /dev/disk1s2 or wherever it was.

==Test Coverage==

This is an index of Firefox startup scenarios which need automated test coverage.

TODO:
* groups: chrome, content, extensions, labs, other?
* mark which need no-chrome testing

{| border=solid black 1px
| test || warm start || cold start || notes
|-
| empty profile ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes ||
|-
| everything dirty || || || {{bug|515540}}
|-
| min places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || informed by places-stats, {{bug|515540}}
|-
| med places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| max places ||style="background-color: lawngreen"|yes ||style="background-color: lawngreen"|yes || ditto
|-
| med session || || || informed by test pilot data
|-
| max session || || || ditto
|-
| downloads || || ||
|-
| url-classifier data || || || data is fully downloaded, should test page-load for a match and no match
|-
| form history|| || ||
|-
| passwords || || ||
|-
| plugins || || ||
|-
| content preferences || || ||
|-
| extensions || || || top 5?
|-
| extension update || || ||
|-
| application update || || ||
|-
| persona applied || || ||
|-
| weave || || ||
|-
| jetpack || || ||
|-
| ubiquity || || ||
|-
| page links are visited || || ||
|-
| page has microformats || || ||
|-
| page is ssl || || ||
|-
| page has plugin || || ||
|}

User:Joel Reymont/Android Notes

2009-10-20T19:03:33Z

Joel Reymont: /* Can Android run a completely native app? */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 12, 2009 ==

Via Chris Lord...

The [http://hg.mozilla.org/incubator/offscreen headless branch of Mozilla that we work on] renders to system memory and it is up to the embedder to get this to a texture however they like. We have our own embedding library that uses this with Clutter
[http://git.clutter-project.org/?r=clutter-mozembed clutter-mozembed] and we just upload each changed region to the texture.

A possible speed-up is to use a shared memory pixmap and texture-from-pixmap (or whatever the platform's equivalent is), which
would remove the need to copy from system to video memory.

I wrote a (now out-dated, but still relevant) [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten guide on building the Moblin browser components] on my blog. There's also a [http://chrislord.net/files/moz-headless-screenshot.c short demo application I wrote] that just uses mozilla-headless alone to take a screenshot of a web-page.

Unfortunately, there is no documentation beyond the standard Mozilla documentation and the source-code (and the above-mentioned links).

I think the easiest way to port Firefox to android would be to write an android-backend (which could well be based on our headless backend) - I didn't think that the Android native development kit was fully featured enough to allow something like this though?

...

You can't run Firefox with the headless backend as it is, it's designed to be embedded and for libxul to be used. The headless backend is also heavily dependent on GLib (for the main loop, signals and idle handlers).

If you wanted to run Firefox, you'd have to write a new backend, or modify an existing one, to provide the application framework. In the gtk2 and headless backends, this is GLib-based. The Gtk2 backend creates a GtkWindow and that is how the user sees and interacts with the application. On the headless backend, there is no visible window. It can only be used for embedding via libxul.

---

Chris has a [http://chrislord.net/files/fosdem-09-slides.odp presentation] that explains the headless branch a bit and [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten instructions on how to build it].

---

There's an [https://bugzilla.mozilla.org/show_bug.cgi?id=446591 existing bug] that deals with headless UI.

= Previous statuses =

= Android Notes =

== Can Android run a completely native app? ==

According to [http://sites.google.com/site/io/anatomy--physiology-of-an-android Anatomy and physiology] and [http://developer.android.com/guide/topics/fundamentals.html Android Fundamentals], each Android process has its own Java virtual machine (VM), so application code runs in isolation from the code of all other applications.

Android processes (Java VMs) communicate with the Home App and various system services via the [http://en.wikipedia.org/wiki/OpenBinder Binder IPC] described using [http://developer.android.com/guide/developing/tools/aidl.html Android Interface Definition Language (AIDL)].

The ''Home Application'' (Launcher) is the app that displays the Android desktop and allows the [http:///Volumes/android/mydroid/frameworks/base/core/java/android/app/LauncherActivity.java launching of other apps] by clicking on their icons.

An ''activity'' presents a visual user interface for one focused endeavor the user can undertake. For example, an activity might present a list of menu items users can choose from or it might display photographs along with their captions. A text messaging application might have one activity that shows a list of contacts to send messages to, a second activity to write the message to the chosen contact, and other activities to review old messages or change settings.

A ''task'' is a group of related activities. May include activities from other apps, running in other processes, e.g. map viewer. A task has an activity stack.

Any given activity

* Has a content view that is created using a layout resource.
* Can be floating or embedded inside another activity.
* Has a pause callback invoked when the user is leaving the activity.
* Is visible when it's at the top of the stack for the current task.
* Is paused when there's another activity on top of it but remains visible. It has lost focus but remains completely alive and attached to the window manager but can be killed in low memory conditions.
* Is stopped when it's completely obscured by another activity. The window is hidden and activity will be killed when memory is needed elsewhere.
* Has a finish callback that may be called when the activity is being evicted from memory.
* Generates a thumbnail of the activity via a callback that's called before pausing.
* Receives trackball and key events via callbacks.

When an app icon is clicked, the Home app uses the package manager to retrive app info and package it into an [http:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Intent.java intent], an asynchronous message with information necessary to start a given activity. The Home app then uses startActivity, a method of Context to launch the activity. [http:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Context.java Context is an abstract class] whose implementation is provided by the Android system.

The launch request is sent via the Binder IPC to the [http:///Volumes/android/mydroid/frameworks/base/services/java/com/android/server/am/ActivityManagerService.java Activity Manager Service] which uses the [http:///Volumes/android/mydroid/frameworks/base/core/java/android/os/Process.java Process] class to start an [http:///Volumes/android/mydroid/frameworks/base/core/java/android/app/ActivityThread.java Activity Thread] in a new instance of the Dalvik Java VM via the [http:///Volumes/android/mydroid//Volumes/android/mydroid/system/core/libcutils/zygote.c Zygote].

The mechanism above assumes Java is used to implement activities. For example, the first non-option argument to Zygote should be a class name in the system class path. Also, [http://developer.android.com/guide/developing/tools/aidl.html aidl command-line tool] only generates Java code for Binder IPCs.

While I'm confident that I could write a native application that plugs into the Android event stream as well as the activity, window, etc. managers, doing so would require adding C++ code generation to the ''aidl tool'' and implementing the portion of the Android Java SDK that deals with the activity life cycle.

Still, it will not be possible to launch a native app from the Home app due to the Java assumptions in the core launching mechanism, so a Java shim will be required. There may be a great deal of flexibility in running a completely native app since, for example, OpenGL ES is not required and the screen (surface) buffer can be written to directly.

Another alternative is to let the Java shim handle act as an invisible ''activity'' that uses the NDK to pass events to native code running in a shared library. OpenGL ES will be required in the latter case to write to a pre-configured screen context.

To summarize...

The Android activity and application launching mechanism assumes Java. Apps run in separate processes (and Java VMs) and talk to system services via the Binder IPC.

There's no C++ code generation from the AIDL that describes the IPC so we'll need to write that. We'll also need to write a whole bunch of other C++ code that covers the portion of the Android Java SDK that deals with the application life cycle to be able to suspend, resume, stop and generally be a good citizen of the Android desktop (Home application).

We won't be able to launch a native app right from the Android desktop so a Java shim will be needed. If we go the long route and talk to system services using Binder IPC, then we won't need OpenGL ES and will be able to write directly to the screen buffer (surface). If we make our Java shim a proper Android activity that feeds all kinds of events to our shared library using the NDK (JNI), then we'll be required to use OpenGL ES to write to a pre-configured context as well as potentially implement calls back to Java from our native libraries.

I volunteer to write the C++ from AIDL code generator, as well as re-implement necessary portions of the Android Java SDK in C++. Many a C++ programmer will thank us for that and announcing a Mozilla Android C++ SDK may actually force Google's hand and make them release something official.

Going the officially sanctioned way and using a proper Java activity shim with the NDK will positively be easier initially but may turn out to be a pain down the road what with OpenGL ES and all the data flying back and forth between Java and native code.

== Sample external application Makefile ==

Here's how you can build your own native apps without being shoehorned into the Android SDK build structure. I got this by massaging the output of ''make showcommands''.

<pre>
SDK := /Volumes/android/mydroid
TOOLCHAIN := $(SDK)/prebuilt/darwin-x86/toolchain/arm-eabi-4.2.1
ABILIB := $(TOOLCHAIN)/lib/gcc/arm-eabi/4.2.1/
BIN := $(TOOLCHAIN)/bin
C++ := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc

INCLUDE := \
-I$(SDK)/system/core/include \
-I$(SDK)/hardware/libhardware/include \
-I$(SDK)/hardware/libhardware_legacy/include \
-I$(SDK)/hardware/ril/include \
-I$(SDK)/dalvik/libnativehelper/include \
-I$(SDK)/frameworks/base/include \
-I$(SDK)/frameworks/base/opengl/include \
-I$(SDK)/external/skia/include \
-I$(SDK)/out/target/product/generic/obj/include \
-I$(SDK)/bionic/libc/arch-arm/include \
-I$(SDK)/bionic/libc/include \
-I$(SDK)/bionic/libstdc++/include \
-I$(SDK)/bionic/libc/kernel/common \
-I$(SDK)/bionic/libc/kernel/arch-arm \
-I$(SDK)/bionic/libm/include \
-I$(SDK)/bionic/libm/include/arch/arm \
-I$(SDK)/bionic/libthread_db/include \
-I$(SDK)/frameworks/base/cmds/demo \
-I$(SDK)/system/core/include/arch/linux-arm \
-include $(SDK)/system/core/include/arch/linux-arm/AndroidConfig.h \
$(NULL)

CFLAGS = \
-D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5TE__ -DANDROID \
-DSK_RELEASE -DNDEBUG -UDEBUG \
-mthumb -mthumb-interwork -msoft-float -march=armv5te -mtune=xscale \
-fpic -fno-exceptions -ffunction-sections -funwind-tables -fstack-protector \
-fno-short-enums -fmessage-length=0 -finline-functions -fno-inline-functions-called-once \
-fgcse-after-reload -frerun-cse-after-loop -frename-registers -fvisibility-inlines-hidden \
-fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -fno-rtti \
-W -Wall -Wno-multichar -Wno-unused -Werror=return-type -Wstrict-aliasing=2 \
-Wnon-virtual-dtor -Werror=return-type \
-MD -Os -g \
$(NULL)

LIBDIR := $(SDK)/out/target/product/generic/obj/lib

LIBS := -lui -llog -lutils -lc -lstdc++ -lm $(LIBDIR)/crtbegin_dynamic.o

LDFLAGS := \
-nostdlib -Bdynamic -Wl,--no-undefined -Wl,-T,$(SDK)/build/core/armelf.x \
-Wl,-dynamic-linker,$(SDK)/system/bin/linker -Wl,--gc-sections -Wl,-z,nocopyreloc \
-L$(LIBDIR) -Wl,-rpath-link=$(LIBDIR) \
$(NULL)

all: demo

demo: surfaceflinger.o demo.o
$(C++) $(LDFLAGS) -o demo $(LIBS) surfaceflinger.o demo.o \
$(ABILIB)/interwork/libgcc.a $(LIBDIR)/crtend_android.o

surfaceflinger.o:
$(C++) $(INCLUDE) -c $(CFLAGS) surfaceflinger.cpp

demo.o:
$(CC) $(INCLUDE) -c $(CFLAGS) demo.c

clean:
rm -f demo *.o
</pre>

== Introduction to the Android Window System ==

[http://people.debian.org.tw/%7Eolv/surfaceflinger/surfaceflinger.pdf Slides] and [http://people.debian.org.tw/%7Eolv/surfaceflinger/demo.tar.gz demo code].

== Building and running the OpenGL ES NDK example ==

Here's how to build and run the san-angeles application from NDK 1.6. This program demonstrates how to use a GLSurfaceView from Java along with native OpenGL calls to perform frame rendering.

<pre>
cd /Volumes/android/mydroid/
. build/envsetup.sh
including vendor/aosp/vendorsetup.sh
lunch sdk-eng

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=1.6
TARGET_PRODUCT=sdk
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=darwin
HOST_BUILD_TYPE=release
BUILD_ID=Donut
============================================
</pre>

First, we need to build the shared library of native code

<pre>
cd ~/work/android/android-ndk-1.6_r1/
build/host-setup.sh
make APP=san-angeles

Android NDK: Building for application 'san-angeles'
Compile thumb : sanangeles <= apps/san-angeles/project/jni/importgl.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/demo.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/app-android.c
SharedLibrary : libsanangeles.so
Install : libsanangeles.so => apps/san-angeles/project/libs/armeabi
</pre>

Next, we need to make sure we can build the Java portion of the project

<pre>
android update project --path apps/san-angeles/project

Updated local.properties
File build.xml is too old and needs to be updated.
Added file apps/san-angeles/project/build.xml
</pre>

Now, let's build the debug release of project

<pre>
cd apps/san-angeles/project/
ant debug

Buildfile: build.xml
[setup] Project Target: Android 1.6
[setup] API level: 4

dirs:
[echo] Creating output directories if needed...

resource-src:
[echo] Generating R.java / Manifest.java from the resources...

aidl:
[echo] Compiling aidl files into Java classes...

compile:
[javac] Compiling 1 source file to /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes

dex:
[echo] Converting compiled files and external libraries into bin/classes.dex...

package-resources:
[echo] Packaging resources
[aaptexec] Creating full resource package...

debug-sign:

package:
[apkbuilder] Creating .DemoActivity-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /Users/joelr/.android/debug.keystore
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes.dex => classes.dex
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/libs/armeabi/libsanangeles.so => lib/armeabi/libsanangeles.so

debug:
[echo] Running zip align on final apk...
[echo] Debug Package: bin/.DemoActivity-debug.apk

BUILD SUCCESSFUL
Total time: 1 second
</pre>

Finally, let's install the app on the device

<pre>
adb install -r bin/.DemoActivity-debug.apk

1107 KB/s (13707 bytes in 0.012s)
pkg: /data/local/tmp/.DemoActivity-debug.apk
Success
</pre>

To run the app click on the "tab" at the bottom of the Android emulator screen and then on the DemoActivity icon in the list of applications. Enjoy!

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. lunch sdk-eng && make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].
''Italic text''

Platform/2009-10-20

2009-10-20T18:23:43Z

Joel Reymont: /* Roundtable */

[[Platform/2009-10-13|« previous week]] | [[Platform|index]] | [[Platform/2009-10-27|next week »]]

=== Notices / Schedule ===
'''[[Releases/Firefox 3.0.15|Firefox 3.0.15]] / [[Releases/Firefox_3.5.4|Firefox 3.5.4]]'''
* both releases headed to beta today (second time for 3.5.4)
* please work on your blockers for 3.0.16 and 3.5.5

'''Firefox 3.6b1'''
* builds available
* in QA

= Blocker Report =
See more [[Firefox/3.6|Firefox 3.6 related blocker queries]], or learn about the [[Releases/Flags|new status and blocker flags]]
* Release Blockers (flag: blocking1.9.2 or blocking-firefox3.6)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=flag%3Ablocking1.9.2%2B,blocking-firefox3.6%2B 115 OPEN] (-3 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIXED%20-status1.9.2%3Afixed,unaf,wont%20flag%3Ablocking1.9.2%2B,blocking-firefox3.6%2B 33 FIXED but not yet fixed on mozilla-1.9.2] (+7 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=flag%3Ablocking1.9.2%3F,blocking-firefox3.6%3F 69 nominations] (+22 w/w)
** Handy charts: [http://people.mozilla.org/~mnandigama/openBlockers.html Blocker snapshots], [http://people.mozilla.org/~mnandigama/openBlockersTrend.html Blocker and Noms trends]
* Approvals
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20-status1.9.2%3Afixed,unaf,wont%20flag%3Aapproval1.9.2%3F 97 requests] (+5 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20-status1.9.2%3Afixed,unaf,wont%20flag%3Aapproval1.9.2%2B 37 approved but not yet fixed on mozilla-1.9.2] (+3 w/w)
** Charts: [http://people.mozilla.org/~mnandigama/openblockersNominated.html Nominations snapshots]

=== Browser / Front End ===
(Progress reports every weekend on [http://planet.firefox.com Planet Firefox])
* see our [[Firefox/Projects|active projects]] and get involved / '''propose others'''
* Namoroka/mozilla-1.9.2 front end development:
** [[Firefox/Projects/Per_Tab_Network_Prioritization|session restore speedup]] is the only potential change for 1.9.2 at this point, will know by end of this week

* Axel asked about strings left to land
** already at string freeze, anything else is considered an exception
** axel says we can make limited string changes until this Friday, October 23rd
** beltzner to follow up in dev-planning today

* Juan from QA expressed concern about two issues
** windows7 tab previews aren't working great
*** blockers on file, we might have to pull it out
** tegra OS builds are slow
*** being resolved with a new OS image from NVIDIA

=== GFX Update ===
* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20priority:P1%20comp:gfx,imagelib,graphics%20flag:blocking1.9.2%2B%20-status1.9.2:fixed,unaffected,wontfix 0 beta blockers]; [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20comp:gfx,imagelib,graphics%20flag:blocking1.9.2%2B%20-status1.9.2:fixed,unaffected,wontfix 4 1.9.2 blockers].
** The only one that worries me is the NVIDIA Windows CE image corruption bug, {{bug|500200}}, which we have not been able to make progress on, and for which we suspect a driver bug.
* Bas has made a lot of progress on the DirectWrite font backend for Windows 7. There are lots of screenshots in {{bug|517642}}. There's still a ways to go before this will be ready for prime time, but we're getting great results.
* Jim has been having trouble reproducing a drag-and-drop cursor flickering problem on Windows. If you have seen cursor flickering, please weigh in on {{bug|521966}}.
* The WebGL standard has been submitted to the Khronos board for intellectual property considerations. This will be a 45-day process, after which the spec will be released for public review.

=== Layout Update ===

* Blocker report
** 14 Layout
*** 2 fixed on trunk and need landing
*** Some others need review
*** A handful of background-image invalidation/repaint issues
*** Need triage, but nothing scary on the list
*** Silverlight 3 doesn't work, Microsoft unresponsive in Bugzilla
** 1 Video/audio, fixed on trunk and needs branch landing
* Frame poisoning mitigated a newly discovered bug (yay)
* Ongoing animation improvements, can animate 'opacity' now (dholbert)
* New Ogg index code based on Ogg Skeleton track (cpearce)
* Ogg decoder rewrite grinding through tests (doublec)
* Removed media element "load" event and NETWORK_LOADING state on trunk and branch
* Discuss changing CSS gradient syntax for 1.9.2
* jdaggett and jfkthame prepping for AtypI

=== Content Update ===
* 24 1.9.1 blockers left.
* Investigating how much and where to focus on websocket/HyBi protocol spec work (Jason and Jonas)
* (josh) We have an accepted spec for Linux plugin accessibility and Novell is working on a patch for 1.9.3. Works with out-of-process plugins.
** https://wiki.mozilla.org/Plugins:NativeAccessibility
* Multi-file select patch done, but needs testing before ready (Jonas)
* HTML5 parser update (hsivonen)

=== Platform-specific Support Update ===
* (josh) Still waiting on sqlite library conflict resolution ({{bug|513747}}) so we can produce 64-bit Mac OS X builds without patches and set up a tinderbox. This also blocks running mochitests on 10.6.
* (smichaud) 1.9.2/JEP update -- old bugs fixed, new bugs found:
** {{bug|513979}} (Disable Java setting not preserved when switching from FF 3.5 to 3.6):
*** Fixed and landed, in beta1.
** {{bug|521818}} (Current plugin instance not destroyed when disabling Java):
*** Fixed and landed, not in beta1.
** {{bug|522962}} (Applet needs to be reloaded after switching to/from full screen mode):
*** Needs to be fixed in JEP, probably not a blocker.
** {{bug|523129}} (Java's AppletContext.showDocument() doesn't work on 1.9.2 branch):
*** Patch available, blocker.

=== JS ===

* Continuing allocation performance improvements
* recursive function changes
* identified major source of browser/shell perf delta: wrappers / security checks. About 15% on SunSpider.

=== Startup Performance ===

Remeasuring 3.5 cold startup on WinXP is giving roller-coaster numbers, ups and downs in the >20% range between builds with changes that are seemingly innocuous. Focus this week is on figuring out how to get stable numbers there.

At all ranges in the graph, the 3.5 WinXP cold startup numbers are all showing a regression in 3.6/3.7... again. I'll re-open that bug and work on figuring out a regression range. However, this is blocked on getting reliable numbers out of 3.5, see above.

Highlights from the [http://autonome.wordpress.com/2009/10/17/firefox-startup-performance-weekly-summary-6/ weekly update post]:

* Ben Hsieh’s work to improve fastload cache invalidation landed in {{bug|511761}}, with a 3% win on warm startup of WinXP.
* Alfred Kayser landed {{bug|511754}}, which improves JAR file reading efficiency.
* Rob Strong closed {{bug|521956}} and started some cleanup of nsUpdateService, further reducing the size of it for {{bug|311965}}.
* Drew has a patch up for {{bug|506814}}, getting rid of ChangeGetPersistentDescriptor/SetPersistentDescriptor on Mac, needs review Josh.

Join us on IRC in [http://irc.mozilla.org/#startup #startup].

=== Security ===
Lightweight themes security review today at 12:30 (PDT).

Other [[Firefox3.6/Security|pending reviews]]

=== Electrolysis ===

* Plugins:
** working towards getting stuff landed in mozilla-central, preffed off. See {{bug|523904}} for dependencies.
** Windowless impl work continues, basic Windows support has been reviewed, needs to land.
** NPRuntime basically complete, some edge cases to clean up but nothing that renders it inoperable.

* Tabs
** Focusing on getting a Fennec demo working.
** Smaug has been dealing with focus and embedding issues which should make keyboard events work properly.
** bsmedberg working on removing assumptions about browsers having docshells
** Probably need to remote some basic events such as MozAfterPaint in order for the Fennec tilebrowser to work correctly

* Necko
** close to having our full set of necko xpcshell unit tests running in content process--researching failures/hangs.

=== Tree Management ===
* Cold Ts on Mac and Linux turned on last week
* Turned on 'make check' tests on optimized and debug builds on trunk and project branches.
* Turned on valgrind tests on debug builds on linux tracemonkey
* 7 new mac slaves added to build/test pool today, 2 xserves + 1 mini coming soon
* Proposing to land split-mochitests on Thursday morning.

=== Roundtable ===
* Plan for Websockets patch? [roc]
* Minor update for 1.9.2? [roc]
* Firebug status
* [https://wiki.mozilla.org/User:Joel_Reymont/Android_Notes#Can_Android_run_a_completely_native_app.3F Can we run a completely native app on Android?] Join us on IRC in [http://irc.mozilla.org/#android #android].
* interface changes between now and mozilla-1.9.2 release? [beltzner]

Platform/2009-10-20

2009-10-20T18:14:13Z

Joel Reymont: /* Roundtable */

[[Platform/2009-10-13|« previous week]] | [[Platform|index]] | [[Platform/2009-10-27|next week »]]

=== Notices / Schedule ===
'''[[Releases/Firefox 3.0.15|Firefox 3.0.15]] / [[Releases/Firefox_3.5.4|Firefox 3.5.4]]'''
* both releases headed to beta today (second time for 3.5.4)
* please work on your blockers for 3.0.16 and 3.5.5

'''Firefox 3.6b1'''
* builds available
* in QA

= Blocker Report =
See more [[Firefox/3.6|Firefox 3.6 related blocker queries]], or learn about the [[Releases/Flags|new status and blocker flags]]
* Release Blockers (flag: blocking1.9.2 or blocking-firefox3.6)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=flag%3Ablocking1.9.2%2B,blocking-firefox3.6%2B 115 OPEN] (-3 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIXED%20-status1.9.2%3Afixed,unaf,wont%20flag%3Ablocking1.9.2%2B,blocking-firefox3.6%2B 33 FIXED but not yet fixed on mozilla-1.9.2] (+7 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=flag%3Ablocking1.9.2%3F,blocking-firefox3.6%3F 69 nominations] (+22 w/w)
** Handy charts: [http://people.mozilla.org/~mnandigama/openBlockers.html Blocker snapshots], [http://people.mozilla.org/~mnandigama/openBlockersTrend.html Blocker and Noms trends]
* Approvals
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20-status1.9.2%3Afixed,unaf,wont%20flag%3Aapproval1.9.2%3F 97 requests] (+5 w/w)
** [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20-status1.9.2%3Afixed,unaf,wont%20flag%3Aapproval1.9.2%2B 37 approved but not yet fixed on mozilla-1.9.2] (+3 w/w)
** Charts: [http://people.mozilla.org/~mnandigama/openblockersNominated.html Nominations snapshots]

=== Browser / Front End ===
(Progress reports every weekend on [http://planet.firefox.com Planet Firefox])
* see our [[Firefox/Projects|active projects]] and get involved / '''propose others'''
* Namoroka/mozilla-1.9.2 front end development:

=== GFX Update ===
* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20priority:P1%20comp:gfx,imagelib,graphics%20flag:blocking1.9.2%2B%20-status1.9.2:fixed,unaffected,wontfix 0 beta blockers]; [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20comp:gfx,imagelib,graphics%20flag:blocking1.9.2%2B%20-status1.9.2:fixed,unaffected,wontfix 4 1.9.2 blockers].
** The only one that worries me is the NVIDIA Windows CE image corruption bug, {{bug|500200}}, which we have not been able to make progress on, and for which we suspect a driver bug.
* Bas has made a lot of progress on the DirectWrite font backend for Windows 7. There are lots of screenshots in {{bug|517642}}. There's still a ways to go before this will be ready for prime time, but we're getting great results.
* Jim has been having trouble reproducing a drag-and-drop cursor flickering problem on Windows. If you have seen cursor flickering, please weigh in on {{bug|521966}}.
* The WebGL standard has been submitted to the Khronos board for intellectual property considerations. This will be a 45-day process, after which the spec will be released for public review.

=== Layout Update ===

* Blocker report
** 18 Layout
*** 4 fixed on trunk and need landing
*** Some others need review
*** Need triage, but nothing scary on the list
*** Silverlight 3 doesn't work, Microsoft unresponsive in Bugzilla
** 5 Video/audio, all fixed on trunk and need branch landing
* Frame poisoning mitigated a newly discovered bug (yay)
* Ongoing animation improvements, can animate 'opacity' now (dholbert)
* New Ogg index code based on Ogg Skeleton track (cpearce)
* Ogg decoder rewrite grinding through tests (doublec)
* Removed media element "load" event and NETWORK_LOADING state on trunk and branch
* Discuss changing CSS gradient syntax for 1.9.2
* jdaggett and jfkthame prepping for AtypI

=== Content Update ===
* 24 1.9.1 blockers left.
* Investigating how much and where to focus on websocket/HiBy protocol spec work (Jason and Jonas)
* (josh) We have an accepted spec for Linux plugin accessibility and Novell is working on a patch for 1.9.3. Works with out-of-process plugins.
** https://wiki.mozilla.org/Plugins:NativeAccessibility
* Multi-file select patch done, but needs testing before ready (Jonas)
* HTML5 parser update (hsivonen)

=== Platform-specific Support Update ===
* (josh) Still waiting on sqlite library conflict resolution ({{bug|513747}}) so we can produce 64-bit Mac OS X builds without patches and set up a tinderbox. This also blocks running mochitests on 10.6.
* (smichaud) 1.9.2/JEP update -- old bugs fixed, new bugs found:
** {{bug|513979}} (Disable Java setting not preserved when switching from FF 3.5 to 3.6):
*** Fixed and landed, in beta1.
** {{bug|521818}} (Current plugin instance not destroyed when disabling Java):
*** Fixed and landed, not in beta1.
** {{bug|522962}} (Applet needs to be reloaded after switching to/from full screen mode):
*** Needs to be fixed in JEP, probably not a blocker.
** {{bug|523129}} (Java's AppletContext.showDocument() doesn't work on 1.9.2 branch):
*** Patch available.

=== JS ===

* Continuing allocation performance improvements
* recursive function changes
* identified major source of browser/shell perf delta: wrappers / security checks. About 15% on SunSpider.

=== Startup Performance ===

Remeasuring 3.5 cold startup on WinXP is giving roller-coaster numbers, ups and downs in the >20% range between builds with changes that are seemingly innocuous. Focus this week is on figuring out how to get stable numbers there.

At all ranges in the graph, the 3.5 WinXP cold startup numbers are all showing a regression in 3.6/3.7... again. I'll re-open that bug and work on figuring out a regression range. However, this is blocked on getting reliable numbers out of 3.5, see above.

Highlights from the [http://autonome.wordpress.com/2009/10/17/firefox-startup-performance-weekly-summary-6/ weekly update post]:

* Ben Hsieh’s work to improve fastload cache invalidation landed in {{bug|511761}}, with a 3% win on warm startup of WinXP.
* Alfred Kayser landed {{bug|511754}}, which improves JAR file reading efficiency.
* Rob Strong closed {{bug|521956}} and started some cleanup of nsUpdateService, further reducing the size of it for {{bug|311965}}.
* Drew has a patch up for {{bug|506814}}, getting rid of ChangeGetPersistentDescriptor/SetPersistentDescriptor on Mac, needs review Josh.

Join us on IRC in [http://irc.mozilla.org/#startup #startup].

=== Security ===

=== Electrolysis ===

* Plugins:
** working towards getting stuff landed in mozilla-central, preffed off. See {{bug|523904}} for dependencies.
** Windowless impl work continues, basic Windows support has been reviewed, needs to land.
** NPRuntime basically complete, some edge cases to clean up but nothing that renders it inoperable.

* Tabs
** Focusing on getting a Fennec demo working.
** Smaug has been dealing with focus and embedding issues which should make keyboard events work properly.
** bsmedberg working on removing assumptions about browsers having docshells
** Probably need to remote some basic events such as MozAfterPaint in order for the Fennec tilebrowser to work correctly

* Necko
** close to having our full set of necko xpcshell unit tests running in content process--researching failures/hangs.

=== Tree Management ===
* Cold Ts on Mac and Linux turned on last week
* Turned on 'make check' tests on optimized and debug builds on trunk and project branches.
* Turned on valgrind tests on debug builds on linux tracemonkey
* 8 new mac slaves added to build/test pool today, 2 xserves coming soon
* Proposing to land split-mochitests on Thursday morning.

=== Roundtable ===
* Plan for Websockets patch? [roc]
* Minor update for 1.9.2? [roc]
* Firebug status
* [https://wiki.mozilla.org/User:Joel_Reymont/Android_Notes#Can_Android_run_a_completely_native_app.3F Can we run a completely native app on Android?]

User:Joel Reymont/Android Notes

2009-10-20T17:29:37Z

Joel Reymont: /* Can Android run a completely native app? */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 12, 2009 ==

Via Chris Lord...

The [http://hg.mozilla.org/incubator/offscreen headless branch of Mozilla that we work on] renders to system memory and it is up to the embedder to get this to a texture however they like. We have our own embedding library that uses this with Clutter
[http://git.clutter-project.org/?r=clutter-mozembed clutter-mozembed] and we just upload each changed region to the texture.

A possible speed-up is to use a shared memory pixmap and texture-from-pixmap (or whatever the platform's equivalent is), which
would remove the need to copy from system to video memory.

I wrote a (now out-dated, but still relevant) [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten guide on building the Moblin browser components] on my blog. There's also a [http://chrislord.net/files/moz-headless-screenshot.c short demo application I wrote] that just uses mozilla-headless alone to take a screenshot of a web-page.

Unfortunately, there is no documentation beyond the standard Mozilla documentation and the source-code (and the above-mentioned links).

I think the easiest way to port Firefox to android would be to write an android-backend (which could well be based on our headless backend) - I didn't think that the Android native development kit was fully featured enough to allow something like this though?

...

You can't run Firefox with the headless backend as it is, it's designed to be embedded and for libxul to be used. The headless backend is also heavily dependent on GLib (for the main loop, signals and idle handlers).

If you wanted to run Firefox, you'd have to write a new backend, or modify an existing one, to provide the application framework. In the gtk2 and headless backends, this is GLib-based. The Gtk2 backend creates a GtkWindow and that is how the user sees and interacts with the application. On the headless backend, there is no visible window. It can only be used for embedding via libxul.

---

Chris has a [http://chrislord.net/files/fosdem-09-slides.odp presentation] that explains the headless branch a bit and [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten instructions on how to build it].

---

There's an [https://bugzilla.mozilla.org/show_bug.cgi?id=446591 existing bug] that deals with headless UI.

= Previous statuses =

= Android Notes =

== Can Android run a completely native app? ==

According to [http://sites.google.com/site/io/anatomy--physiology-of-an-android Anatomy and physiology] and [http://developer.android.com/guide/topics/fundamentals.html Android Fundamentals], each Android process has its own Java virtual machine (VM), so application code runs in isolation from the code of all other applications.

Android processes (Java VMs) communicate with the Home App and various system services via the [http://en.wikipedia.org/wiki/OpenBinder Binder IPC] described using [http://developer.android.com/guide/developing/tools/aidl.html Android Interface Definition Language (AIDL)].

The ''Home Application'' (Launcher) is the app that displays the Android desktop and allows the [http:///Volumes/android/mydroid/frameworks/base/core/java/android/app/LauncherActivity.java launching of other apps] by clicking on their icons.

An ''activity'' presents a visual user interface for one focused endeavor the user can undertake. For example, an activity might present a list of menu items users can choose from or it might display photographs along with their captions. A text messaging application might have one activity that shows a list of contacts to send messages to, a second activity to write the message to the chosen contact, and other activities to review old messages or change settings.

A ''task'' is a group of related activities. May include activities from other apps, running in other processes, e.g. map viewer. A task has an activity stack.

Any given activity

* Has a content view that is created using a layout resource.
* Can be floating or embedded inside another activity.
* Has a pause callback invoked when the user is leaving the activity.
* Is visible when it's at the top of the stack for the current task.
* Is paused when there's another activity on top of it but remains visible. It has lost focus but remains completely alive and attached to the window manager but can be killed in low memory conditions.
* Is stopped when it's completely obscured by another activity. The window is hidden and activity will be killed when memory is needed elsewhere.
* Has a finish callback that may be called when the activity is being evicted from memory.
* Generates a thumbnail of the activity via a callback that's called before pausing.
* Receives trackball and key events via callbacks.

When an app icon is clicked, the Home app uses the package manager to retrive app info and package it into an [http:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Intent.java intent], an asynchronous message with information necessary to start a given activity. The Home app then uses startActivity, a method of Context to launch the activity. [http:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Context.java Context is an abstract class] whose implementation is provided by the Android system.

The launch request is sent via the Binder IPC to the [http:///Volumes/android/mydroid/frameworks/base/services/java/com/android/server/am/ActivityManagerService.java Activity Manager Service] which uses the [http:///Volumes/android/mydroid/frameworks/base/core/java/android/os/Process.java Process] class to start an [http:///Volumes/android/mydroid/frameworks/base/core/java/android/app/ActivityThread.java Activity Thread] in a new instance of the Dalvik Java VM via the [http:///Volumes/android/mydroid//Volumes/android/mydroid/system/core/libcutils/zygote.c Zygote].

The mechanism above assumes Java is used to implement activities. For example, the first non-option argument to Zygote should be a class name in the system class path. Also, [http://developer.android.com/guide/developing/tools/aidl.html aidl command-line tool] only generates Java code for Binder IPCs.

While I'm confident that I could write a native application that plugs into the Android event stream as well as the activity, window, etc. managers, doing so would require adding C++ code generation to the ''aidl tool'' and implementing the portion of the Android Java SDK that deals with the activity life cycle.

Still, it will not be possible to launch a native app from the Home app due to the Java assumptions in the core launching mechanism, so a Java shim will be required. There may be a great deal of flexibility in running a completely native app since, for example, OpenGL ES is not required and the screen (surface) buffer can be written to directly.

Another alternative is to let the Java shim handle act as an invisible ''activity'' that uses the NDK to pass events to native code running in a shared library. OpenGL ES will be required in the latter case to write to a pre-configured screen context.

== Sample external application Makefile ==

Here's how you can build your own native apps without being shoehorned into the Android SDK build structure. I got this by massaging the output of ''make showcommands''.

<pre>
SDK := /Volumes/android/mydroid
TOOLCHAIN := $(SDK)/prebuilt/darwin-x86/toolchain/arm-eabi-4.2.1
ABILIB := $(TOOLCHAIN)/lib/gcc/arm-eabi/4.2.1/
BIN := $(TOOLCHAIN)/bin
C++ := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc

INCLUDE := \
-I$(SDK)/system/core/include \
-I$(SDK)/hardware/libhardware/include \
-I$(SDK)/hardware/libhardware_legacy/include \
-I$(SDK)/hardware/ril/include \
-I$(SDK)/dalvik/libnativehelper/include \
-I$(SDK)/frameworks/base/include \
-I$(SDK)/frameworks/base/opengl/include \
-I$(SDK)/external/skia/include \
-I$(SDK)/out/target/product/generic/obj/include \
-I$(SDK)/bionic/libc/arch-arm/include \
-I$(SDK)/bionic/libc/include \
-I$(SDK)/bionic/libstdc++/include \
-I$(SDK)/bionic/libc/kernel/common \
-I$(SDK)/bionic/libc/kernel/arch-arm \
-I$(SDK)/bionic/libm/include \
-I$(SDK)/bionic/libm/include/arch/arm \
-I$(SDK)/bionic/libthread_db/include \
-I$(SDK)/frameworks/base/cmds/demo \
-I$(SDK)/system/core/include/arch/linux-arm \
-include $(SDK)/system/core/include/arch/linux-arm/AndroidConfig.h \
$(NULL)

CFLAGS = \
-D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5TE__ -DANDROID \
-DSK_RELEASE -DNDEBUG -UDEBUG \
-mthumb -mthumb-interwork -msoft-float -march=armv5te -mtune=xscale \
-fpic -fno-exceptions -ffunction-sections -funwind-tables -fstack-protector \
-fno-short-enums -fmessage-length=0 -finline-functions -fno-inline-functions-called-once \
-fgcse-after-reload -frerun-cse-after-loop -frename-registers -fvisibility-inlines-hidden \
-fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -fno-rtti \
-W -Wall -Wno-multichar -Wno-unused -Werror=return-type -Wstrict-aliasing=2 \
-Wnon-virtual-dtor -Werror=return-type \
-MD -Os -g \
$(NULL)

LIBDIR := $(SDK)/out/target/product/generic/obj/lib

LIBS := -lui -llog -lutils -lc -lstdc++ -lm $(LIBDIR)/crtbegin_dynamic.o

LDFLAGS := \
-nostdlib -Bdynamic -Wl,--no-undefined -Wl,-T,$(SDK)/build/core/armelf.x \
-Wl,-dynamic-linker,$(SDK)/system/bin/linker -Wl,--gc-sections -Wl,-z,nocopyreloc \
-L$(LIBDIR) -Wl,-rpath-link=$(LIBDIR) \
$(NULL)

all: demo

demo: surfaceflinger.o demo.o
$(C++) $(LDFLAGS) -o demo $(LIBS) surfaceflinger.o demo.o \
$(ABILIB)/interwork/libgcc.a $(LIBDIR)/crtend_android.o

surfaceflinger.o:
$(C++) $(INCLUDE) -c $(CFLAGS) surfaceflinger.cpp

demo.o:
$(CC) $(INCLUDE) -c $(CFLAGS) demo.c

clean:
rm -f demo *.o
</pre>

== Introduction to the Android Window System ==

[http://people.debian.org.tw/%7Eolv/surfaceflinger/surfaceflinger.pdf Slides] and [http://people.debian.org.tw/%7Eolv/surfaceflinger/demo.tar.gz demo code].

== Building and running the OpenGL ES NDK example ==

Here's how to build and run the san-angeles application from NDK 1.6. This program demonstrates how to use a GLSurfaceView from Java along with native OpenGL calls to perform frame rendering.

<pre>
cd /Volumes/android/mydroid/
. build/envsetup.sh
including vendor/aosp/vendorsetup.sh
lunch sdk-eng

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=1.6
TARGET_PRODUCT=sdk
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=darwin
HOST_BUILD_TYPE=release
BUILD_ID=Donut
============================================
</pre>

First, we need to build the shared library of native code

<pre>
cd ~/work/android/android-ndk-1.6_r1/
build/host-setup.sh
make APP=san-angeles

Android NDK: Building for application 'san-angeles'
Compile thumb : sanangeles <= apps/san-angeles/project/jni/importgl.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/demo.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/app-android.c
SharedLibrary : libsanangeles.so
Install : libsanangeles.so => apps/san-angeles/project/libs/armeabi
</pre>

Next, we need to make sure we can build the Java portion of the project

<pre>
android update project --path apps/san-angeles/project

Updated local.properties
File build.xml is too old and needs to be updated.
Added file apps/san-angeles/project/build.xml
</pre>

Now, let's build the debug release of project

<pre>
cd apps/san-angeles/project/
ant debug

Buildfile: build.xml
[setup] Project Target: Android 1.6
[setup] API level: 4

dirs:
[echo] Creating output directories if needed...

resource-src:
[echo] Generating R.java / Manifest.java from the resources...

aidl:
[echo] Compiling aidl files into Java classes...

compile:
[javac] Compiling 1 source file to /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes

dex:
[echo] Converting compiled files and external libraries into bin/classes.dex...

package-resources:
[echo] Packaging resources
[aaptexec] Creating full resource package...

debug-sign:

package:
[apkbuilder] Creating .DemoActivity-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /Users/joelr/.android/debug.keystore
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes.dex => classes.dex
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/libs/armeabi/libsanangeles.so => lib/armeabi/libsanangeles.so

debug:
[echo] Running zip align on final apk...
[echo] Debug Package: bin/.DemoActivity-debug.apk

BUILD SUCCESSFUL
Total time: 1 second
</pre>

Finally, let's install the app on the device

<pre>
adb install -r bin/.DemoActivity-debug.apk

1107 KB/s (13707 bytes in 0.012s)
pkg: /data/local/tmp/.DemoActivity-debug.apk
Success
</pre>

To run the app click on the "tab" at the bottom of the Android emulator screen and then on the DemoActivity icon in the list of applications. Enjoy!

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. lunch sdk-eng && make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].
''Italic text''

User:Joel Reymont/Android Notes

2009-10-20T17:23:24Z

Joel Reymont: /* Android Notes */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 12, 2009 ==

Via Chris Lord...

The [http://hg.mozilla.org/incubator/offscreen headless branch of Mozilla that we work on] renders to system memory and it is up to the embedder to get this to a texture however they like. We have our own embedding library that uses this with Clutter
[http://git.clutter-project.org/?r=clutter-mozembed clutter-mozembed] and we just upload each changed region to the texture.

A possible speed-up is to use a shared memory pixmap and texture-from-pixmap (or whatever the platform's equivalent is), which
would remove the need to copy from system to video memory.

I wrote a (now out-dated, but still relevant) [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten guide on building the Moblin browser components] on my blog. There's also a [http://chrislord.net/files/moz-headless-screenshot.c short demo application I wrote] that just uses mozilla-headless alone to take a screenshot of a web-page.

Unfortunately, there is no documentation beyond the standard Mozilla documentation and the source-code (and the above-mentioned links).

I think the easiest way to port Firefox to android would be to write an android-backend (which could well be based on our headless backend) - I didn't think that the Android native development kit was fully featured enough to allow something like this though?

...

You can't run Firefox with the headless backend as it is, it's designed to be embedded and for libxul to be used. The headless backend is also heavily dependent on GLib (for the main loop, signals and idle handlers).

If you wanted to run Firefox, you'd have to write a new backend, or modify an existing one, to provide the application framework. In the gtk2 and headless backends, this is GLib-based. The Gtk2 backend creates a GtkWindow and that is how the user sees and interacts with the application. On the headless backend, there is no visible window. It can only be used for embedding via libxul.

---

Chris has a [http://chrislord.net/files/fosdem-09-slides.odp presentation] that explains the headless branch a bit and [http://chrislord.net/blog/Software/building-moblin-web-browser.enlighten instructions on how to build it].

---

There's an [https://bugzilla.mozilla.org/show_bug.cgi?id=446591 existing bug] that deals with headless UI.

= Previous statuses =

= Android Notes =

== Can Android run a completely native app? ==

According to [http://sites.google.com/site/io/anatomy--physiology-of-an-android Anatomy and physiology] and [http://developer.android.com/guide/topics/fundamentals.html Android Fundamentals], each Android process has its own Java virtual machine (VM), so application code runs in isolation from the code of all other applications.

Android processes (Java VMs) communicate with the Home App and various system services via the [http://en.wikipedia.org/wiki/OpenBinder Binder IPC] described using [http://developer.android.com/guide/developing/tools/aidl.html Android Interface Definition Language (AIDL)].

The ''Home Application'' (Launcher) is the app that displays the Android desktop and allows the [file:///Volumes/android/mydroid/frameworks/base/core/java/android/app/LauncherActivity.java launching of other apps] by clicking on their icons.

An ''activity'' presents a visual user interface for one focused endeavor the user can undertake. For example, an activity might present a list of menu items users can choose from or it might display photographs along with their captions. A text messaging application might have one activity that shows a list of contacts to send messages to, a second activity to write the message to the chosen contact, and other activities to review old messages or change settings.

A ''task'' is a group of related activities. May include activities from other apps, running in other processes, e.g. map viewer. A task has an activity stack.

Any given activity
- Has a content view that is created using a layout resource.
- Can be floating or embedded inside another activity.
- Has a pause callback invoked when the user is leaving the activity.
- Is visible when it's at the top of the stack for the current task.
- Is paused when there's another activity on top of it but remains visible. It has lost focus but remains completely alive and attached to the window manager but can be killed in low memory conditions.
- Is stopped when it's completely obscured by another activity. The window is hidden and activity will be killed when memory is needed elsewhere.
- Has a finish callback that may be called when the activity is being evicted from memory.
- Generates a thumbnail of the activity via a callback that's called before pausing.
- Receives trackball and key events via callbacks.

When an app icon is clicked, the Home app uses the package manager to retrive app info and package it into an [file:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Intent.java intent], an asynchronous message with information necessary to start a given activity. The Home app then uses startActivity, a method of Context to launch the activity. [file:///Volumes/android/mydroid/frameworks/base/core/java/android/content/Context.java Context is an abstract class] whose implementation is provided by the Android system.

The launch request is sent via the Binder IPC to the [file:///Volumes/android/mydroid/frameworks/base/services/java/com/android/server/am/ActivityManagerService.java Activity Manager Service] which uses the [file:///Volumes/android/mydroid/frameworks/base/core/java/android/os/Process.java Process] class to start an [file:///Volumes/android/mydroid/frameworks/base/core/java/android/app/ActivityThread.java Activity Thread] in a new instance of the Dalvik Java VM via the [file:///Volumes/android/mydroid//Volumes/android/mydroid/system/core/libcutils/zygote.c Zygote].

The mechanism above assumes Java is used to implement activities. For example, the first non-option argument to Zygote should be a class name in the system class path. Also, [http://developer.android.com/guide/developing/tools/aidl.html aidl command-line tool] only generates Java code for Binder IPCs.

While I'm confident that I could write a native application that plugs into the Android event stream as well as the activity, window, etc. managers, doing so would require adding C++ code generation to the ''aidl tool'' and implementing the portion of the Android Java SDK that deals with the activity life cycle.

Still, it will not be possible to launch a native app from the Home app due to the Java assumptions in the core launching mechanism, so a Java shim will be required. There may be a great deal of flexibility in running a completely native app since, for example, OpenGL ES is not required and the screen (surface) buffer can be written to directly. Another alternative is to let the Java shim handle act as an invisible ''activity'' that uses the NDK to pass events to native code running in a shared library. OpenGL ES will be required in the latter case to write to a pre-configured screen context.

== Sample external application Makefile ==

Here's how you can build your own native apps without being shoehorned into the Android SDK build structure. I got this by massaging the output of ''make showcommands''.

<pre>
SDK := /Volumes/android/mydroid
TOOLCHAIN := $(SDK)/prebuilt/darwin-x86/toolchain/arm-eabi-4.2.1
ABILIB := $(TOOLCHAIN)/lib/gcc/arm-eabi/4.2.1/
BIN := $(TOOLCHAIN)/bin
C++ := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc

INCLUDE := \
-I$(SDK)/system/core/include \
-I$(SDK)/hardware/libhardware/include \
-I$(SDK)/hardware/libhardware_legacy/include \
-I$(SDK)/hardware/ril/include \
-I$(SDK)/dalvik/libnativehelper/include \
-I$(SDK)/frameworks/base/include \
-I$(SDK)/frameworks/base/opengl/include \
-I$(SDK)/external/skia/include \
-I$(SDK)/out/target/product/generic/obj/include \
-I$(SDK)/bionic/libc/arch-arm/include \
-I$(SDK)/bionic/libc/include \
-I$(SDK)/bionic/libstdc++/include \
-I$(SDK)/bionic/libc/kernel/common \
-I$(SDK)/bionic/libc/kernel/arch-arm \
-I$(SDK)/bionic/libm/include \
-I$(SDK)/bionic/libm/include/arch/arm \
-I$(SDK)/bionic/libthread_db/include \
-I$(SDK)/frameworks/base/cmds/demo \
-I$(SDK)/system/core/include/arch/linux-arm \
-include $(SDK)/system/core/include/arch/linux-arm/AndroidConfig.h \
$(NULL)

CFLAGS = \
-D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5TE__ -DANDROID \
-DSK_RELEASE -DNDEBUG -UDEBUG \
-mthumb -mthumb-interwork -msoft-float -march=armv5te -mtune=xscale \
-fpic -fno-exceptions -ffunction-sections -funwind-tables -fstack-protector \
-fno-short-enums -fmessage-length=0 -finline-functions -fno-inline-functions-called-once \
-fgcse-after-reload -frerun-cse-after-loop -frename-registers -fvisibility-inlines-hidden \
-fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -fno-rtti \
-W -Wall -Wno-multichar -Wno-unused -Werror=return-type -Wstrict-aliasing=2 \
-Wnon-virtual-dtor -Werror=return-type \
-MD -Os -g \
$(NULL)

LIBDIR := $(SDK)/out/target/product/generic/obj/lib

LIBS := -lui -llog -lutils -lc -lstdc++ -lm $(LIBDIR)/crtbegin_dynamic.o

LDFLAGS := \
-nostdlib -Bdynamic -Wl,--no-undefined -Wl,-T,$(SDK)/build/core/armelf.x \
-Wl,-dynamic-linker,$(SDK)/system/bin/linker -Wl,--gc-sections -Wl,-z,nocopyreloc \
-L$(LIBDIR) -Wl,-rpath-link=$(LIBDIR) \
$(NULL)

all: demo

demo: surfaceflinger.o demo.o
$(C++) $(LDFLAGS) -o demo $(LIBS) surfaceflinger.o demo.o \
$(ABILIB)/interwork/libgcc.a $(LIBDIR)/crtend_android.o

surfaceflinger.o:
$(C++) $(INCLUDE) -c $(CFLAGS) surfaceflinger.cpp

demo.o:
$(CC) $(INCLUDE) -c $(CFLAGS) demo.c

clean:
rm -f demo *.o
</pre>

== Introduction to the Android Window System ==

[http://people.debian.org.tw/%7Eolv/surfaceflinger/surfaceflinger.pdf Slides] and [http://people.debian.org.tw/%7Eolv/surfaceflinger/demo.tar.gz demo code].

== Building and running the OpenGL ES NDK example ==

Here's how to build and run the san-angeles application from NDK 1.6. This program demonstrates how to use a GLSurfaceView from Java along with native OpenGL calls to perform frame rendering.

<pre>
cd /Volumes/android/mydroid/
. build/envsetup.sh
including vendor/aosp/vendorsetup.sh
lunch sdk-eng

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=1.6
TARGET_PRODUCT=sdk
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=darwin
HOST_BUILD_TYPE=release
BUILD_ID=Donut
============================================
</pre>

First, we need to build the shared library of native code

<pre>
cd ~/work/android/android-ndk-1.6_r1/
build/host-setup.sh
make APP=san-angeles

Android NDK: Building for application 'san-angeles'
Compile thumb : sanangeles <= apps/san-angeles/project/jni/importgl.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/demo.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/app-android.c
SharedLibrary : libsanangeles.so
Install : libsanangeles.so => apps/san-angeles/project/libs/armeabi
</pre>

Next, we need to make sure we can build the Java portion of the project

<pre>
android update project --path apps/san-angeles/project

Updated local.properties
File build.xml is too old and needs to be updated.
Added file apps/san-angeles/project/build.xml
</pre>

Now, let's build the debug release of project

<pre>
cd apps/san-angeles/project/
ant debug

Buildfile: build.xml
[setup] Project Target: Android 1.6
[setup] API level: 4

dirs:
[echo] Creating output directories if needed...

resource-src:
[echo] Generating R.java / Manifest.java from the resources...

aidl:
[echo] Compiling aidl files into Java classes...

compile:
[javac] Compiling 1 source file to /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes

dex:
[echo] Converting compiled files and external libraries into bin/classes.dex...

package-resources:
[echo] Packaging resources
[aaptexec] Creating full resource package...

debug-sign:

package:
[apkbuilder] Creating .DemoActivity-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /Users/joelr/.android/debug.keystore
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes.dex => classes.dex
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/libs/armeabi/libsanangeles.so => lib/armeabi/libsanangeles.so

debug:
[echo] Running zip align on final apk...
[echo] Debug Package: bin/.DemoActivity-debug.apk

BUILD SUCCESSFUL
Total time: 1 second
</pre>

Finally, let's install the app on the device

<pre>
adb install -r bin/.DemoActivity-debug.apk

1107 KB/s (13707 bytes in 0.012s)
pkg: /data/local/tmp/.DemoActivity-debug.apk
Success
</pre>

To run the app click on the "tab" at the bottom of the Android emulator screen and then on the DemoActivity icon in the list of applications. Enjoy!

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. lunch sdk-eng && make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].
''Italic text''

User:Joel Reymont/Android Notes

2009-10-16T15:37:24Z

Joel Reymont: /* Android Notes */

User:Joel Reymont/Android Notes

2009-10-14T15:30:07Z

Joel Reymont: /* Android Notes */

User:Joel Reymont/Android Notes

2009-10-13T21:48:43Z

Joel Reymont: /* Building and running the OpenGL ES NDK example */

User:Joel Reymont/Android Notes

2009-10-12T16:43:29Z

Joel Reymont: /* October 12, 2009 */

User:Joel Reymont/Android Notes

2009-10-12T15:40:02Z

Joel Reymont: /* October 12, 2009 */

User:Joel Reymont/Android Notes

2009-10-12T12:41:59Z

Joel Reymont: /* October 12, 2009 */

User:Joel Reymont/Android Notes

2009-10-12T10:11:36Z

Joel Reymont: /* Current status */

User:Joel Reymont/Android Notes

2009-10-09T14:55:55Z

Joel Reymont: /* Android Notes */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building and running the OpenGL ES NDK example ==

Here's how to build and run the san-angeles application from NDK 1.6. This program demonstrates how to use a GLSurfaceView from Java along with native OpenGL calls to perform frame rendering.

<pre>
cd /Volumes/android/mydroid/
. build/envsetup.sh
including vendor/aosp/vendorsetup.sh
lunch sdk-eng

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=1.6
TARGET_PRODUCT=sdk
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=darwin
HOST_BUILD_TYPE=release
BUILD_ID=Donut
============================================
</pre>

First, we need to build the shared library of native code

<pre>
cd ~/work/android/android-ndk-1.6_r1/
build/host-setup.sh
make APP=san-angeles

Android NDK: Building for application 'san-angeles'
Compile thumb : sanangeles <= apps/san-angeles/project/jni/importgl.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/demo.c
Compile thumb : sanangeles <= apps/san-angeles/project/jni/app-android.c
SharedLibrary : libsanangeles.so
Install : libsanangeles.so => apps/san-angeles/project/libs/armeabi
</pre>

Next, we need to make sure we can build the Java portion of the project

<pre>
android update project --path apps/san-angeles/project

Updated local.properties
File build.xml is too old and needs to be updated.
Added file apps/san-angeles/project/build.xml
</pre>

Now, lets build the debug release of project

<pre>
cd apps/san-angeles/project/
ant debug

Buildfile: build.xml
[setup] Project Target: Android 1.6
[setup] API level: 4

dirs:
[echo] Creating output directories if needed...

resource-src:
[echo] Generating R.java / Manifest.java from the resources...

aidl:
[echo] Compiling aidl files into Java classes...

compile:
[javac] Compiling 1 source file to /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes

dex:
[echo] Converting compiled files and external libraries into bin/classes.dex...

package-resources:
[echo] Packaging resources
[aaptexec] Creating full resource package...

debug-sign:

package:
[apkbuilder] Creating .DemoActivity-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /Users/joelr/.android/debug.keystore
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/bin/classes.dex => classes.dex
[apkbuilder] /Users/joelr/Work/android/android-ndk-1.6_r1/apps/san-angeles/project/libs/armeabi/libsanangeles.so => lib/armeabi/libsanangeles.so

debug:
[echo] Running zip align on final apk...
[echo] Debug Package: bin/.DemoActivity-debug.apk

BUILD SUCCESSFUL
Total time: 1 second
</pre>

Finally, let's install the app on the device

<pre>
adb install -r bin/.DemoActivity-debug.apk

1107 KB/s (13707 bytes in 0.012s)
pkg: /data/local/tmp/.DemoActivity-debug.apk
Success
</pre>

To run the app click on the "tab" at the bottom of the Android emulator screen and then on the DemoActivity icon in the list of applications. Enjoy!

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. lunch sdk-eng && make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-09T13:46:20Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. lunch sdk-eng && make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-08T16:41:14Z

Joel Reymont: /* Drawing and input from native code */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For [http://code.google.com/p/glesquake/ Quake], GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
lunch generic-eng && emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-08T16:40:09Z

Joel Reymont: /* Android Notes */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Drawing and input from native code ==

Via Jack Palevich...

Input is done by subclassing a view, and override the various input methods. Make sure your view can get focus, using setFocusableInTouchMode(true);  Otherwise you won't get input events. You could use any View class as your base class, but I suggest that you subclass GLSurfaceView because that makes it easier to use OpenGL graphics for drawing.

Probably the fastest way to get output from native code is done using OpenGL ES, which is newly available in NDK 1.6. Create a texture the size of the  screen ( it has to be a power of two on each side, so 512 x 256 if your screen is 320 x 240.) Then update the texture and render it to the screen as a screen aligned quad.

For native OpenGL ES APIs the drawing surface is implicit in the thread. For Quake, GLSurfaceView's rendering thread sets up the context and calls the
QuakeView's Renderer, which calls the native code, which makes the native GL calls.

Because the context is implicit in the thread, there's nothing that needs to be passed between Java and native code.

The downside of this approach is that you can't wrap the native GL calls as easily as you can wrap the JSR 239 Java OpenGL APIs. It's sometimes useful
to wrap the GL APIs to keep track of the matrix stack, or log debugging messages, or check for error messages.

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
lunch generic-eng && emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-07T21:19:17Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854
</pre>

and

<pre>
android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
lunch generic-eng && emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-07T21:17:24Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply these changes

<pre>
cd system/core && git pull git://android.git.kernel.org/platform/system/core refs/changes/45/11845/2 && cd ../..
cd external/qemu && git pull git://android.git.kernel.org/platform/external/qemu refs/changes/46/11846/2 && cd ../..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/14/12014/1 && cd ..
cd prebuilt && git pull git://android.git.kernel.org/platform/prebuilt refs/changes/75/12075/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/74/12074/1 && cd ..
cd build && git pull git://android.git.kernel.org/platform/build refs/changes/93/12093/1 && cd ..
</pre>

The following almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. make sdk

You should see something like this if everything goes well

<pre>
ls out/host/darwin-x86/sdk/
android-sdk_eng.joelr_mac-x86 sdk_deps.mk
android-sdk_eng.joelr_mac-x86.zip
</pre>

One more step is needed

<pre>
export PATH=$PATH:`pwd`/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/
pushd prebuilt && ln -s darwin-x86 darwin-x86_64 && popd
pushd out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86/tools/lib
ln -s x86 x86_64 && popd
</pre>

Verify that everything works

<pre>
android list targets

Available Android targets:
id: 1
Name: Android 1.6
Type: Platform
API level: 4
Revision: 1
Skins: HVGA (default), QVGA, WVGA800, WVGA854

android list avd

Available Android Virtual Devices:

</pre>

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_SDK=$AND_HOME/out/host/darwin-x86/sdk/android-sdk_eng.joelr_mac-x86
export AND_BIN=$AND_SDK/tools
export AND_IMG=$AND_SDK/platforms/android-1.6/images
export AND_KERNEL=$AND_IMG/kernel-qemu
export PATH=$PATH:$AND_BIN

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android
cd /Volumes/android/mydroid
. ./build/envsetup.sh
lunch generic-eng && emulator
</pre>

You should see home screen in a minute or less.

You can also run the emulator in verbose mode if you want to troubleshoot or just see lots of interesting output, e.g.

<pre>
emulator -verbose -logcat "*:v"

emulator: found Android build root: /Volumes/android/mydroid
emulator: found Android build out: /Volumes/android/mydroid/out/target/product/generic
emulator: locking user data image at /Volumes/android/mydroid/out/target/product/generic/userdata-qemu.img
emulator: selecting default skin name 'HVGA'
emulator: autoconfig: -skin HVGA
emulator: autoconfig: -skindir /Volumes/android/mydroid/development/emulator/skins
emulator: keyset loaded from: /Users/joelr/.android/default.keyset
emulator: trying to load skin file '/Volumes/android/mydroid/development/emulator/skins/HVGA/layout'
emulator: skin network speed: 'full'
emulator: skin network delay: 'none'
emulator: no SD Card image at '/Volumes/android/mydroid/out/target/product/generic/sdcard.img'
emulator: registered 'boot-properties' qemud service
emulator: registered 'boot-properties' qemud service
emulator: Adding boot property: 'qemu.sf.lcd_density' = '160'
...
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-07T17:59:34Z

Joel Reymont: /* Building Android on Snow Leopard */

User:Joel Reymont/Android Notes

2009-10-07T16:57:42Z

Joel Reymont: /* Building Android on Snow Leopard */

User:Joel Reymont/Android Notes

2009-10-07T16:54:53Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply [http://groups.google.com/group/android-platform/msg/efd76f9a3bd02a2b changes] making sure there spaces before ''refs''.

This almost works but resets the head each time. Both of the last two changes are needed but the Java one gets written over by the very last change. Also, changes downloaded this way get blown away on ''repo sync''.

<pre>
repo download platform/system/core 11845/2
repo download platform/external/qemu 11846/2
repo download platform/prebuilt 12014/1
repo download platform/prebuilt 12075/1
repo download platform/build 12074/1
repo download platform/build 12093/1
</pre>

3. make

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_BIN=$AND_HOME/out/host/darwin-x86/bin
export AND_PROD=$AND_HOME/out/target/product/generic
export AND_QEMU=$AND_HOME/prebuilt/android-arm/kernel/kernel-qemu
export AND_IMG=$AND_HOME/out/target/product/generic/userdata.img

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android

$AND_BIN/emulator -sysdir $AND_PROD -kernel $AND_QEMU -system $AND_IMG
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont/Android Notes

2009-10-07T15:51:04Z

Joel Reymont: /* Android Notes */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply [http://groups.google.com/group/android-platform/msg/efd76f9a3bd02a2b changes] making sure there spaces before ''refs''.

3. make

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_BIN=$AND_HOME/out/host/darwin-x86/bin
export AND_PROD=$AND_HOME/out/target/product/generic
export AND_QEMU=$AND_HOME/prebuilt/android-arm/kernel/kernel-qemu
export AND_IMG=$AND_HOME/out/target/product/generic/userdata.img

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android

$AND_BIN/emulator -sysdir $AND_PROD -kernel $AND_QEMU -system $AND_IMG
</pre>

== Vlad's Android Notes ==

Can be found [https://wiki.mozilla.org/User:VladVukicevic/Android_Notes here].

User:Joel Reymont

2009-10-07T15:48:32Z

Joel Reymont: Created page with '= [https://wiki.mozilla.org/User:Joel_Reymont/Android_Notes Android Notes] = = [https://wiki.mozilla.org/Firefox/Projects/Startup_Time_Improvements/joelr_notes Firefox startup t…'

= [https://wiki.mozilla.org/User:Joel_Reymont/Android_Notes Android Notes] =

= [https://wiki.mozilla.org/Firefox/Projects/Startup_Time_Improvements/joelr_notes Firefox startup time improvements] =

User:Joel Reymont/Android Notes

2009-10-07T15:46:42Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply [http://groups.google.com/group/android-platform/msg/efd76f9a3bd02a2b changes] making sure there spaces before ''refs''.

3. make

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_BIN=$AND_HOME/out/host/darwin-x86/bin
export AND_PROD=$AND_HOME/out/target/product/generic
export AND_QEMU=$AND_HOME/prebuilt/android-arm/kernel/kernel-qemu
export AND_IMG=$AND_HOME/out/target/product/generic/userdata.img

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android

$AND_BIN/emulator -sysdir $AND_PROD -kernel $AND_QEMU -system $AND_IMG
</pre>

User:Joel Reymont/Android Notes

2009-10-07T15:43:35Z

Joel Reymont:

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

== October 7, 2009 ==

= Previous statuses =

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply [http://groups.google.com/group/android-platform/msg/efd76f9a3bd02a2b changes] making sure there spaces before ''refs''.

3. make

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_BIN=$AND_HOME/out/host/darwin-x86/bin
export AND_PROD=$AND_HOME/out/target/product/generic
export AND_QEMU=$AND_HOME/prebuilt/android-arm/kernel/kernel-qemu
export AND_IMG=$AND_HOME/out/target/product/generic/userdata.img

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android

$AND_BIN/emulator -system $AND_PROD -kernel $AND_QEMU -data $AND_IMG
</pre>

User:Joel Reymont/Android Notes

2009-10-07T15:42:43Z

Joel Reymont: /* Building Android on Snow Leopard */

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

=== October 7, 2009 ===

= Android Notes =

== Building Android on Snow Leopard ==

1. [http://source.android.com/download Get the sources].

2. Apply [http://groups.google.com/group/android-platform/msg/efd76f9a3bd02a2b changes] making sure there spaces before ''refs''.

3. make

4. You are set and you can now run the emulator, e.g.

<pre>
export AND_HOME=/Volumes/android/mydroid
export AND_BIN=$AND_HOME/out/host/darwin-x86/bin
export AND_PROD=$AND_HOME/out/target/product/generic
export AND_QEMU=$AND_HOME/prebuilt/android-arm/kernel/kernel-qemu
export AND_IMG=$AND_HOME/out/target/product/generic/userdata.img

hdiutil attach ~/work/android/android.dmg -mountpoint /Volumes/android

$AND_BIN/emulator -system $AND_PROD -kernel $AND_QEMU -data $AND_IMG
</pre>

User:Joel Reymont/Android Notes

2009-10-07T15:36:45Z

Joel Reymont: Created page with '= Intro = This page collects my notes on porting Firefox to Android. = Current status = === October 7, 2009 === = Android Notes = == Building Android on Snow Leopard =='

= Intro =

This page collects my notes on porting Firefox to Android.

= Current status =

=== October 7, 2009 ===

= Android Notes =

== Building Android on Snow Leopard ==

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-23T23:02:04Z

Joel Reymont: /* September 23, 2009 */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 23, 2009 ===

==== x86-64 ====

I now have x86-64 Firefox up and running on Snow Leopard. Josh Aas did all the work, I just [https://bugzilla.mozilla.org/show_bug.cgi?id=518435 put on the final touches].

==== A shiny new world ====

Damon thinks I should help Josh with the 64-bit port. 64-bit Trace Monkey gives us a [http://www.bailopan.net/blog/?p=595 20% improvement in speed] which beats the 5% DWARF improvement from -fomit-frame-pointer.

I didn't know it was my birthday. I'm leaving the 32-bit world behind. 64 bits here I come!

= Previous statuses =

=== September 22, 2009 ===

I have two bottlenecks on my plate, [https://bugzilla.mozilla.org/show_bug.cgi?id=517045 Bug 517045] and [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576]. Both of them may not be possible to speed up significantly, not without a large research effort. The former requires consultation with the owners and the later requires me to become a layout expert.

Benjamin Smedberg suggested taking care of DWARF symbols in breakpad as that will let us use the -fomit-frame-pointer gcc option. Omitting the frame pointer would result in at least a 5% overall speed up because a register will be freed on x86-32.

DWARF is a [https://bugzilla.mozilla.org/showdependencygraph.cgi?id=464750 tree of dependent bugs], though, and Jim Blandy is working on them. Jim is a Linux guy, though, and also has ECMAScript 5 strict mode to take care of whereas I'm strictly focused on performance optimization.

I'll ask management to see if I can work on DWARF with Jim.

=== September 21, 2009 ===

==== Wins ====

Kudos to Masayuki for fixing [https://bugzilla.mozilla.org/show_bug.cgi?id=517549 Bug 517549]. It's hard to believe but it looks like our Mac Ts [http://graphs.mozilla.org/graph.html#tests=[{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22169%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22173%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22178%22}]&sel=1252833380,1253542320 went down 30%] and it was only 12.3% of my startup. I wonder what happens when [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576] gets fixed, that's 21% of my startup!

One other thing I'm wondering about... What if all the bottlenecks I find will get fixed by someone else? Damon wants me to write code!!!

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-23T13:07:30Z

Joel Reymont: /* A shiny new world */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 23, 2009 ===

==== A shiny new world ====

Damon thinks I should help Josh with the 64-bit port. 64-bit Trace Monkey gives us a [http://www.bailopan.net/blog/?p=595 20% improvement in speed] which beats the 5% DWARF improvement from -fomit-frame-pointer.

I didn't know it was my birthday. I'm leaving the 32-bit world behind. 64 bits here I come!

= Previous statuses =

=== September 22, 2009 ===

I have two bottlenecks on my plate, [https://bugzilla.mozilla.org/show_bug.cgi?id=517045 Bug 517045] and [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576]. Both of them may not be possible to speed up significantly, not without a large research effort. The former requires consultation with the owners and the later requires me to become a layout expert.

Benjamin Smedberg suggested taking care of DWARF symbols in breakpad as that will let us use the -fomit-frame-pointer gcc option. Omitting the frame pointer would result in at least a 5% overall speed up because a register will be freed on x86-32.

DWARF is a [https://bugzilla.mozilla.org/showdependencygraph.cgi?id=464750 tree of dependent bugs], though, and Jim Blandy is working on them. Jim is a Linux guy, though, and also has ECMAScript 5 strict mode to take care of whereas I'm strictly focused on performance optimization.

I'll ask management to see if I can work on DWARF with Jim.

=== September 21, 2009 ===

==== Wins ====

Kudos to Masayuki for fixing [https://bugzilla.mozilla.org/show_bug.cgi?id=517549 Bug 517549]. It's hard to believe but it looks like our Mac Ts [http://graphs.mozilla.org/graph.html#tests=[{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22169%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22173%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22178%22}]&sel=1252833380,1253542320 went down 30%] and it was only 12.3% of my startup. I wonder what happens when [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576] gets fixed, that's 21% of my startup!

One other thing I'm wondering about... What if all the bottlenecks I find will get fixed by someone else? Damon wants me to write code!!!

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-23T13:06:17Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 23, 2009 ===

==== A shiny new world ====

Damon thinks I should help Josh with the 64-bit port. 64-bit Trace Monkey gives us a [http://www.bailopan.net/blog/?p=595 20% improvement in speed] which beats the 5% DWARF improvement from -fomit-frame-pointer. I didn't know it was my birthday!

I'm leaving the 32-bit world behind. 64 bits here I come!

= Previous statuses =

=== September 22, 2009 ===

I have two bottlenecks on my plate, [https://bugzilla.mozilla.org/show_bug.cgi?id=517045 Bug 517045] and [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576]. Both of them may not be possible to speed up significantly, not without a large research effort. The former requires consultation with the owners and the later requires me to become a layout expert.

Benjamin Smedberg suggested taking care of DWARF symbols in breakpad as that will let us use the -fomit-frame-pointer gcc option. Omitting the frame pointer would result in at least a 5% overall speed up because a register will be freed on x86-32.

DWARF is a [https://bugzilla.mozilla.org/showdependencygraph.cgi?id=464750 tree of dependent bugs], though, and Jim Blandy is working on them. Jim is a Linux guy, though, and also has ECMAScript 5 strict mode to take care of whereas I'm strictly focused on performance optimization.

I'll ask management to see if I can work on DWARF with Jim.

=== September 21, 2009 ===

==== Wins ====

Kudos to Masayuki for fixing [https://bugzilla.mozilla.org/show_bug.cgi?id=517549 Bug 517549]. It's hard to believe but it looks like our Mac Ts [http://graphs.mozilla.org/graph.html#tests=[{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22169%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22173%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22178%22}]&sel=1252833380,1253542320 went down 30%] and it was only 12.3% of my startup. I wonder what happens when [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576] gets fixed, that's 21% of my startup!

One other thing I'm wondering about... What if all the bottlenecks I find will get fixed by someone else? Damon wants me to write code!!!

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-21T16:38:15Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 21, 2009 ===

==== Wins ====

Kudos to Masayuki for fixing [https://bugzilla.mozilla.org/show_bug.cgi?id=517549 Bug 517549]. It's hard to believe but it looks like our Mac Ts [http://graphs.mozilla.org/graph.html#tests=[{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22169%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22173%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22178%22}]&sel=1252833380,1253542320 went down 30%] and it was only 12.3% of my startup. I wonder what happens when [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576] gets fixed, that's 21% of my startup!

One other thing I'm wondering about... What if all the bottlenecks I find will get fixed by someone else? Damon wants me to write code!!!

= Previous statuses =

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-21T16:37:11Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 21, 2009 ===

==== Wins ====

Kudos to Masayuki for fixing [https://bugzilla.mozilla.org/show_bug.cgi?id=517549 Bug 517549]. It's hard to believe but it looks like our Mac Ts [http://graphs.mozilla.org/graph.html#tests=[{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22169%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22173%22},{%22test%22:%2216%22,%22branch%22:%221%22,%22machine%22:%22178%22}]&sel=1252833380,1253542320 went down 30%] and it was only 12.3% of my startup. I wonder what happens when [https://bugzilla.mozilla.org/show_bug.cgi?id=517576 Bug 517576] gets fixed, that's 21% of my startup!

One other thing I'm wondering about... What if all the bottlenecks I find will get fixed by someone else? Damon wants me to write code!!!

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-18T15:46:20Z

Joel Reymont: /* Shark tips and tricks */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== Expanding the call tree ===

Alt-Click does it.

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-18T15:43:51Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= Shark tips and tricks =

=== CHUD signposts on Snow Leopard ===

There are no CHUD header files on Snow Leopard. If you're only interest is in emitting signposts, the easiest thing for you to do is switch to the syscall method instead. That'll also mean you won't have to link against the CHUD framework. The full details are in the Shark User Guide, but in brief:

<pre>
int err = syscall(SYS_kdebug_trace, APPSDBG_CODE(DBG_MACH_CHUD, <your code number>) | <type>, arg1, arg2, arg3, arg4);
</pre>

Types are DBG_FUNC_START, DBG_FUNC_END and DBG_FUNC_NONE (for point signposts).

Don't forget to include <sys/kdebug.h> and <sys/syscall.h> for the necessary macros and function.

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-16T19:32:56Z

Joel Reymont: /* September 16, 2009 */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== That mysterious forking ====

Apparently, Firefox forks the first time after a build to rebuild caches, etc. Need to make sure to profile from the 2nd run on. This forking behavior baffled me when I started with DTrace.

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-16T16:50:59Z

Joel Reymont: /* September 16, 2009 */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== I'm an idiot ====

I spent a month becoming an expert on DTrace, only to realize that it's not very useful to get a global picture of the bottlenecks because the overhead of tracing everything is so high!

Shark is a much better tool to get a global picture, specially using "Time profile all states" which allocates time spent blocking to the calling function. I overlooked this initially since I did not zero in on the right thread, the one that has ''start'' and ''XRE_main'' or ''exit''. Without this it looks like Firefox is spending most of its time waiting on threads and Mach messages.

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-16T12:20:48Z

Joel Reymont: /* Timing startup */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 92.8% firefox-bin start
0.0% 92.8% firefox-bin main
0.0% 91.6% XUL XRE_main
0.0% 40.8% XUL nsAppStartup::CreateHiddenWindow()
0.0% 34.7% XUL ScopedXPCOMStartup::Initialize()
0.0% 6.4% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 3.4% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 2.4% XUL nsXPTCStubBase::Stub3()
0.0% 0.8% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 0.8% HIToolbox GetCurrentEventKeyModifiers
0.0% 0.6% XUL nsXREDirProvider::DoStartup()
0.0% 0.5% XUL RemoveComponentRegistries(nsIFile*, nsIFile*, int)
0.0% 0.4% XUL NS_NewToolkitProfileService(nsIToolkitProfileService**)
0.0% 0.3% XUL nsToolkitProfile::Lock(nsIProfileUnlocker**, nsIProfileLock**)
0.0% 0.1% XUL WriteVersion(nsIFile*, nsCString const&, nsCString const&, nsIFile*, nsIFile*)
0.0% 0.1% XUL nsINIParser_internal::Init(nsILocalFile*)
0.0% 0.1% XUL LaunchChildMac
0.0% 0.4% XUL XRE_GetBinaryPath
0.3% 0.4% XUL XRE_CreateAppData
0.4% 0.4% XUL NS_LogInit_P
0.0% 6.2% dyld _dyld_start
1.0% 1.0% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-16T11:59:57Z

Joel Reymont: /* = Breaking startup into smaller chunks */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 87.7% firefox-bin start
0.0% 87.7% firefox-bin main
0.0% 86.4% XUL XRE_main
0.0% 38.3% XUL ScopedXPCOMStartup::Initialize()
0.0% 30.9% XUL nsAppStartup::CreateHiddenWindow()
0.0% 11.1% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 2.5% XUL nsXPTCStubBase::Stub3()
0.0% 1.2% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 1.2% XUL nsXREDirProvider::DoStartup()
0.0% 1.2% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 1.2% XUL XRE_GetBinaryPath
0.0% 7.4% dyld _dyld_start
4.9% 4.9% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ====

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-16T11:58:49Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 16, 2009 ===

==== Timing startup ====

A clear picture courtesy of Shark

<pre>
0.0% 87.7% firefox-bin start
0.0% 87.7% firefox-bin main
0.0% 86.4% XUL XRE_main
0.0% 38.3% XUL ScopedXPCOMStartup::Initialize()
0.0% 30.9% XUL nsAppStartup::CreateHiddenWindow()
0.0% 11.1% XUL ScopedXPCOMStartup::~ScopedXPCOMStartup()
0.0% 2.5% XUL nsXPTCStubBase::Stub3()
0.0% 1.2% XUL ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport*)
0.0% 1.2% XUL nsXREDirProvider::DoStartup()
0.0% 1.2% XUL nsAppStartupNotifier::Observe(nsISupports*, char const*, unsigned short const*)
0.0% 1.2% XUL XRE_GetBinaryPath
0.0% 7.4% dyld _dyld_start
4.9% 4.9% libSystem.B.dylib _exit
</pre>

= Previous statuses =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ===

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-14T16:45:41Z

Joel Reymont: /* September 14, 2009 */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 14, 2009 ===

==== Breaking startup into smaller chunks ===

<pre>
bsmedberg:joelr: I really think that "Ts" is still much too large of a chunk to make meaningful improvements on
bsmedberg:joelr: in order to make meaninful discoveries, I think you have to break startup down into pieces and then maybe improve each piece
joelr:bsmedberg: i'm listening
bsmedberg:well, you've got various stuff that happens up until you start the event loop
bsmedberg:breaking XRE_main into phases up until nsAppStartup::Run would be interesting
bsmedberg:I still haven't seen a graph or anything of how we break down
bsmedberg:joelr: I think stepping through in a debugger might give you a better sense of what's going on (up until Run()) than profiles would
bsmedberg:even if you broke things up into "from main() to the beginning of event loop" and "after the event loop starts" that would be more interesting
bsmedberg:joelr: and then within the first chunk, timing NS_InitXPCOM, the extension manager bits, chromereg startup
bsmedberg:joelr: I think this intersects with some of the startup-timeline stuff taras was doing
joelr:bsmedberg: what are the functions for "extension manager bits" and "chromereg startup"?
bsmedberg:joelr: basically from http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2607 to #3453
joelr:bsmedberg: thanks for the suggestion, i'll do that
bsmedberg:joelr: #3382 to #3408 or so
</pre>

==== XUL reflow ====

<pre>
bz:joelr: so I'm still wondering about some of the reflow stuff
bz:joelr: I know it's hard, but that's why no one's really dived into it yet
bz:joelr: and I'd love to know whether we do in fact end up looping significantly in sprocket layout during startup, for example
joelr:bz: i'm still wondering about reflow myself but i'm not quite ready to tackle it yet
joelr:bz: don't know enough
</pre>

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

= Previous statuses =

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-14T16:19:29Z

Joel Reymont: /* September 14, 2009 */

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 14, 2009 ===

==== XPT_ArenaMalloc and zero-filling memory ====

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

<pre>
bsmedberg:joelr: do you have evidence that the calloc is hurting? It really
doesn't sound like something that would take a lot of time
bsmedberg:there's no IO, and arena pages are roughly page-sized, right?
joelr:bsmedberg: i don't have evidence that it's hurting, not at all. i just
see it in the startup path a lot.
bsmedberg:"see it" in what way?
joelr:bsmedberg: there's no IO but arena pages are not page-sized. they are 1k
blocks whereas pages are 4k
joelr:bsmedberg: i'm dtracing startup, all functions, that's how i see it
bsmedberg:it's less than a page, then... you really shouldn't be having
memory-thrashing issues with the calloc
bsmedberg:joelr: you mean it takes time, or its just a callcount?
bsmedberg:I'm a little skeptical that the arena actually helps at all, but
that's a different matter.
joelr:bsmedberg: call count, i suppose. i'm also skeptical that arena helps at
all since mac osx has it's own memory allocation optimizations
bsmedberg:And the xpt code has no tests, so unles you have a measured
performance problem I suggest not touching it.
joelr:bsmedberg: i see (tests)... well, i'll look into it a little bit more
since i want to measure a difference vs malloc (no zero-fill). it does crash
with malloc at the moment so i won't be looking into it too much.
bsmedberg:you'd have to basically zero-fill when you allocate items from the
arena
joelr:bsmedberg: because those items expect to be zero-filled?
bsmedberg:seems so, if you're crashing with malloc...
bsmedberg:joelr: ok, well, I'm saying "you have no evidence this bug is worth
fixing, please move on"
</pre>

= Previous statuses =

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''

Firefox/Projects/Startup Time Improvements/joelr notes

2009-09-14T16:16:31Z

Joel Reymont:

= Intro =

I'm trying to figure out where Firefox startup time goes, up to the return from BrowserStartup (Javascript function). I'm also manipulating DTrace into telling me where time is going, without making any assumptions.

= Current status =

=== September 14, 2009 ===

Closed [https://bugzilla.mozilla.org/show_bug.cgi?id=516241 Bug 516241]. There's no evidence that zero-filling memset is a bottleneck. DTrace cannot latch onto the return from memset on Snow Leopard, so I can't time it. All I know is that it's called 133,245 times during startup and zero-fills 14,851,995 bytes of memory.

= Previous statuses =

=== September 11, 2009 ===

==== Reflows ====

<pre>
taras: joelr: btw, you gave up on reflows?
taras: that makes me sad
taras: i was hoping we'd end up with some sort of tool to measure cost of laying out various xul components
joelr: taras: i did not give up, i just don't know how to go about it yet
taras: so people could profile their guis
taras: instead of going at it blindly
joelr: taras: i didn't hear your prayers, my apologies
joelr: i'll make a note on the idea of a tool that measures guis, though
joelr: taras: how do i measure? from where to where?
joelr: layout? reflow? something else?
taras: joelr: it's simple, time how long they take
taras: just see long functions take to run, ones that trigger all those fun reflow stacks
</pre>

==== Restarting from the ground up ====

I'm very comfortable with DTrace but still not comfortable with the Firefox code base. It will take me a while to grok reflow, for example, much less optimize it. I feel that there are startup speedups that can be achieved quicker, with a better return on investment (aka low-hanging fruit).

Yes, reflow is in the hot path during startup but it's not clear if it's a major source of slowness. I would like to know what happens during Firefox startup, in as much detail as possible and DTrace has granted me my wish in the form of a 400Mb+ log file. The file lists all the function calls up to the return from the JS function BrowserStartup, e.g.

<pre>
...
6200 . . -> ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6202 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
6203 81 1 <- ImageLoaderMachOCompressed::segmentCommandOffsets() const = 10
6205 270 4 <- ImageLoaderMachO::segIsReadOnlyImport(unsigned int) const = 116
6206 . . -> ImageLoaderMachO::segFileOffset(unsigned int) const(0x8FE46820, 0x1, 0xBFFFC1C8)
6208 . . -> ImageLoaderMachOCompressed::segmentCommandOffsets() const(0x8FE46820, 0x0, 0x0)
...
</pre>

There are obviously interesting bits here. How can we piggyback on the ''ImageLoaderMachOCompressed'' for example?

Minimizing file system access provides a good return on investment since it apparently takes a few milliseconds on mobile devices. I'm focusing on this first and foremost since we do have file system access in the midst of dynamic linker static initialization, e.g.

<pre>
20974362 861470 708476 <- xptiSortFileList(void const*, void const*, void*) = 622
20974385 . . -> xptiSortFileList(void const*, void const*, void*)(0x3050F9C, 0x3050F7C, 0xBFFFEBC0)
20974394 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE978, 0xBFFFE98C, 0x0)
20974397 113 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974400 . . -> nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int)(0xBFFFE924, 0xBFFFE938, 0x0)
20974402 112 2 <- nsACString_internal::nsACString_internal(char*, unsigned int, unsigned int) = 24
20974424 . . -> nsLocalFile::GetNativeLeafName(nsACString_internal&)(0x3083600, 0xBFFFE978, 0x40)
20974427 . . -> objc_exception_try_enter(0xBFFFE868, 0xE000002, 0x181A4)
</pre>

It may be interesting to study the startup path of Chromium and Safari.

=== September 9, 2009 ===

==== Reflow optimization ====

I've been thinking about speeding up reflows ([https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]) but did not make progress until I got Boris Zbarsky involved. There are tons and tons of classes that reflow and I'm completely unfamiliar with that code.

Boris suggested restricting invocations of PresShell::ProcessReflowCommands to those that actually do work

<pre>
0 != mDirtyRoots.Length()
</pre>

on XUL documents

<pre>
FrameManager()->GetRootFrame()->GetFirstChild(nsnull) &&
FrameManager()->GetRootFrame()->GetFirstChild(nsnull)->GetType() == nsGkAtoms::rootFrame
</pre>

Looking at the [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399254 25 reflows and timings] for the "Getting involved with Mozilla..." page, it's clear that some reflows are more expensive than others. It's also interesting that there are [https://bug514275.bugzilla.mozilla.org/attachment.cgi?id=399445 just 11 reflows] for a blank page.

<pre>
bz: 1) try to reduce number of reflows
bz: (e.g. make text controls not reflow sync in this setup if we can; reduce the number of text value sets if we can't)
bz: Or see whether the reflow events we get are really needed (e.g. if the load is not done yet, why are they happening?)
bz: But separately, it looks like reflows seem to be generally expensive
</pre>

==== Performance measurement ====

Using [http://blog.vlad1.com/2009/07/28/measuring-startup/ Vlad's benchmark], I have 5.3s cold start and 0.9s warm start. Timing using the return from [http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#903 BrowserStartup (JS)] gives me almost exactly the same results as Vlad's.

<pre>
javascript*:::function-return
/copyinstr(arg2) == "BrowserStartup"/
{
exit(1);
}
</pre>

The difference is about 50ms which copyinstr probably accounts for.

=== September 3, 2009 ===

On reflow...

<pre>
bz:so there are a few things we could talk about:
bz:1) make any given reflow pass faster (this is hard)
bz:2) reduce number of reflow passes
bz:s/this is hard/this may be hard/
bz:So one place to start would be seeing why we're doing reflow at all
bz:that is, what things cause posting of reflow events and what things trigger reflow in their post-reflow callbacks
bz:and seeing whether we can reduce or eliminate them.
joelr:post-reflow callbacks?
bz:see comment 1 in the bug
bz:firefox has about 4000 elements in its ui
bz:which means about 4000 boxes to reflow
bz:modulo optimizations, etc
bz:BoxReflow is the reflow of a single box
bz:those are the two things to look into
bz:reducing number of reflow passes
bz:and speeding up individual passes
bz:if you have info on what parts actually take up time during reflow, we should look into that
bz:I would also be interested in the number of nsIFrame::Layout calls you see during startup
bz:that's a more interesting metric to me
bz:esp. if we can chunk it up per reflow pass
</pre>

A reflow pass is ''PresShell::ProcessReflowCommands'' which can re-enter itself via post-reflow callbacks.

[https://bugzilla.mozilla.org/show_bug.cgi?id=514275 Bug 514275]

=== September 2, 2009 ===

My previous stack traces were all bogus since symbols are stripped during packaging. This is all sorted out now using this config (for Snow Leopard):

<pre>
export CC="gcc-4.2 -arch i386"
export CXX="g++-4.2 -arch i386"
ac_add_options --target=i386-apple-darwin8.0.0
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --enable-macos-target=10.5

export HOST_CC="gcc-4.2"
export HOST_CXX="g++-4.2"
export RANLIB=ranlib
export AR=ar
export AS=$CC
export LD=ld
export STRIP="strip -x -S"
export CROSS_COMPILE=1
export MOZ_DEBUG_SYMBOLS=1

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-release-profiling
mk_add_options MOZ_MAKE_FLAGS=-j2
ac_add_options --disable-tests
ac_add_options --disable-crashreporter
ac_add_options --enable-dtrace
ac_add_options --enable-libxul
ac_add_options --enable-optimize
ac_add_options --enable-debugger-info-modules
ac_add_options --disable-strip
ac_add_options --disable-install-strip
</pre>

I'm getting 3-5s from cold start to return from ''BrowserStartup'' (JavaScript), e.g.

<pre>
./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55752 -s startup.d -s stop-js.d -s sigcont.d

Total: 5000.294085ms

./d firefox-bin startup.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 55855 -s startup.d -s stop-js.d -s sigcont.d

Total: 3387.623720ms
</pre>

The hot path for those 5s looks like this:

<pre>
./d firefox-bin hotpath.d stop-js.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 55765 -s hotpath.d -s stop-js.d -s sigcont.d
kill -CONT 55765

XUL`nsFrame::BoxReflow(nsBoxLayoutState&, nsPresContext*, nsHTMLReflowMetrics&, nsIRenderingContext*, int, int, int, int, int) 1382
firefox-bin`0x0 623
XUL`nsHTMLInputElement::GetReadOnly(int*) 53
XUL`nsOverflowContinuationTracker::Insert(nsIFrame*, unsigned int&) 48
XUL`MapAttributesIntoRule(nsMappedAttributes const*, nsRuleData*) 24
XUL`nsBlockFrame::Reflow(nsPresContext*, nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&) 20
XUL`nsColumnSetFrame::ReflowChildren(nsHTMLReflowMetrics&, nsHTMLReflowState const&, unsigned int&, nsColumnSetFrame::ReflowConfig const&, int, nsCollapsingMargin*, nsColumnSetFrame::ColumnBalanceData&) 18
XUL`DocumentViewerImpl::PageHide(int) 18
XUL`nsBlockFrame::DoReflowInlineFrames(nsBlockReflowState&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, int*, LineReflowStatus*, int) 14
XUL`non-virtual thunk to nsHTMLInputElement::Release() 11
...
</pre>

=== August 30, 2009 ===

Using my "Lego blocks" [http://github.com/wagerlabs/firefox-startup/tree/master DTrace toolchain] I thought I would check the "hot paths" through Firefox code during the first 20 seconds of startup.

<pre>
./d firefox-bin hotpath.d stop-20s.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1478 -s hotpath.d -s stop-20s.d -s sigcont.d
kill -CONT 1478

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*) 2766
firefox-bin`0x0 2234
XUL`operator new(unsigned long, void*) 348
XUL`gfxImageSurface::Height() const 335
XUL`gfxPattern::Release() 60
XUL`DumpJSValue 44
XUL`gfxContextAutoSaveRestore::gfxContextAutoSaveRestore(gfxContext*) 29
XUL`gfxUserFontSet::Release() 17
XUL`gfxImageSurface::Data() const 16
XUL`gfxContextPathAutoSaveRestore::~gfxContextPathAutoSaveRestore() 8
</pre>

I'm sampling at 997Hz so that's roughly 2.7s for the top result. A small script is all it took to see where ''gfxContextAutoSaveRestore::Reset'' is being called from.

<pre>
BEGIN
{
i = 0;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i < 10 /
{
printf("arg0: 0x%x, arg1: 0x%x\n", arg0, arg1);
ustack();
i++;
}

pid$target::gfxContextAutoSaveRestore??Reset(gfxContext*):return
/ i >= 10 /
{
exit(0);
}
</pre>

The [http://gist.github.com/177980 stack traces] (first one of ten below) include ''gfxImageSurface::Height'' and ''DumpJSValue''.

<pre>
./d firefox-bin autosave.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1527 -s autosave.d -s sigcont.d
kill -CONT 1527
arg0: 0x90, arg1: 0x1a71da8

XUL`gfxContextAutoSaveRestore::Reset(gfxContext*)+0x90
XUL`gfxImageSurface::Height() const+0x2d976
XUL`gfxImageSurface::Height() const+0x135b6
XUL`gfxImageSurface::Height() const+0x107b7
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53d16
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x53f1b
XUL`NS_GetComponentRegistrar_P+0x21cd
XUL`NS_GetComponentRegistrar_P+0x2324
XUL`NS_GetComponentRegistrar_P+0x4964
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b3fd
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4b850
XUL`non-virtual thunk to nsPrintSession::QueryInterface(nsID const&, void**)+0x4a9e1
XUL`DumpJSValue+0x1dfbab
XUL`DumpJSValue+0x1e0103
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xc9e8e
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xb64b4
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0xbb918
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x151bdf
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14e234
XUL`gfxCornerSizes::gfxCornerSizes(gfxSize const&, gfxSize const&, gfxSize const&, gfxSize const&)+0x14ed99
</pre>

''gfxImageSurface::Height'' is low-hanging fruit, defined as

<pre>
PRInt32 Height() const { return mSize.height; }
</pre>

I would suggest making it inline and checking startup time again.

''DumpJSValue'' is trickier because it appears to be a [http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#2714 JavaScript API function]. Still, it's called during the static initialization portion of dynamic linking and all that printing is not helping in a release version of Firefox.

<pre>
./d firefox-bin dump-js-value.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=preinit -p 1579 -s dump-js-value.d -s sigcont.d
kill -CONT 1579

XUL`DumpJSValue+0x3917
XUL`DumpJSValue+0x3b92
XUL`0x111c146
XUL`0x111c161
dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&)+0xeb
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0xf6
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int)+0x9a
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&)+0x3d
dyld`dyld::initializeMainExecutable()+0x86
dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**)+0x105d
dyld`dyldbootstrap::start(macho_header const*, int, char const**, long)+0x30b
dyld`_dyld_start+0x27
firefox-bin`0x1000
</pre>

I suppose the invocation should only be compiled in in a debug version of Firefox but I'm still searching for that invocation!

=== August 28, 2009 ===

Eric Gouriou on my prior use of ''dtrace -c...'':
<pre>
I am afraid that the data collected using this methodology is highly suspect.

- firefox-bin gets run as root
This is not a normal environment for an application. There are plenty of high-level
frameworks which are not quite happy running in that environment, in good part because
the root user will not have a typical home directory setup and due to differences in environment
variables.

This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com

- A typical launch is not just fork/exec or posix_spawn
Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch.
dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch
(Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well
address both issues together.

I'd recommend reworking your script to watch the exec-success of firefox-bin and use
a static probe (no need for $target then) to mark the end time. Then you can use a more normal
launch method, for example open .../Minefiled.app .
</pre>

I did [http://github.com/wagerlabs/firefox-startup/blob/75f719f3a3cdb380071ef8c354cf52564ac4754d/d rework my cold start script] which can be used like this now:

<pre>
./d firefox-bin startup.d stop-main.d
dtrace -Zqw -x dynvarsize=64m -x evaltime=exec -p 35331 -s sigcont.d -s startup.d -s stop-main.d
kill -CONT 35331
Total: 2000.275985ms
</pre>

The above assumes that I launch Minefield.app after ''d'', from the dock, Finder, command line, etc.

=== August 27, 2009 ===

The dyld shared cache lives in ''/var/db/dyld/''. The two files of interest are '''dyld_shared_cache_i386.map''' (for x86-32) and '''shared_region_roots/Applications.paths'''. Both are regular text files. The former shows the contents of the shared cache for the i386 architecture and the latter is what '''update_dyld_shared_cache''' inspects.

There's no prebinding on newer versions of Mac OSX and the dyld shared cache is
automatically updated as needed. Tracing Safari disk activity during startup
reveals that basically all its dynamic libraries are pulled from the dyld
shared cache.

It's possible to add Firefox (''.../Firefox.app/Contents/MacOS/firefox-bin'') to
Applications.paths and the change will persist across reboots. Unfortunately,
only a handful of libraries that Firefox uses are pulled into the cache by
'''update_dyld_shared_cache'''. I'm speculating that this may have something to do
with @executable_path/XUL and friends (otool -L .../firefox-bin).

Safari uses absolute paths to frameworks in ''/System/Library/Frameworks'' so I
speculate that relative paths are what is preventing XUL and others from going
into the cache. We may want to investigate
''/System/Library/Frameworks/Firefox.framework'' or similar.

[https://bugzilla.mozilla.org/show_bug.cgi?id=513076 Bug 513076].

It's possible to fix relative library paths in a given library, e.g. ''fix.sh XUL'' where ''fix.sh'' looks like this

<pre>
#!/bin/bash

function dylibs () {
otool -L $1 |grep executable_path|awk '{print $1}'|cut -d"/" -f2
}

for i in `dylibs $1`
do
install_name_tool -change @executable_path/$i `pwd`/$i $1
done

install_name_tool -id `pwd`/$1 $1
</pre>

Firefox has to be recompiled with ''LDFLAGS=-header-pad_max_install_names'' in ''MOZCONFIG'' to make this happen since new library paths are greater than the space allocated in the Mach-O binary. See the man page for '''install_name_tool''' for details.

It's possible to force dynamic libraries into the cache by putting dynamic library paths into '''shared_region_roots/Applications.paths''' instead of executables. I wasn't successful in caching XUL, though, regardless of what I did.

In the end it doesn't seem to matter since there's a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs.

Here's the cold startup stats for Safari

<pre>
sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/Contents/MacOS/Safari
total time: 696.9 milliseconds (100.0%)
total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups)
total segments mapped: 5, into 30 pages with 10 pages pre-fetched
total images loading time: 204.9 milliseconds (29.4%)
total rebase fixups: 1,298
total rebase fixups time: 0.1 milliseconds (0.0%)
total binding fixups: 2,476
total binding symbol lookups: 234, average images searched per symbol: 1.6
total binding fixups time: 80.5 milliseconds (11.5%)
total bindings lazily fixed up: 3 of 901
total init time time: 411.3 milliseconds (59.0%)
total images with weak exports: 1
</pre>

and Firefox

<pre>
total time: 731.2 milliseconds (100.0%)
total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups)
total segments mapped: 49, into 5903 pages with 684 pages pre-fetched
total images loading time: 235.3 milliseconds (32.1%)
total rebase fixups: 149,011
total rebase fixups time: 3.7 milliseconds (0.5%)
total binding fixups: 24,932
total binding symbol lookups: 797, average images searched per symbol: 2.3
total binding fixups time: 149.9 milliseconds (20.5%)
total bindings lazily fixed up: 45 of 19,109
total init time time: 342.2 milliseconds (46.8%)
total images with weak exports: 3
</pre>

Notice a large and significant difference? Me neither.

The other thing that I cannot explain at the moment is where the rest of the startup time goes, e.g.

<pre>
./cold.sh startup.d
Total: 10001.723521ms
</pre>

So it took less than 1 second to dynamically link Firefox but where did the other 9 seconds of startup go?

cold.sh is rather simple

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1
</pre>

and the startup.d script doesn't do much either

<pre>
#pragma D option quiet

BEGIN
{
start = timestamp;
}

/* stop tracing here */

mozilla$target:::main-entry
{
exit(0);
}

END
{
this->total = timestamp - start;
printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000);
}
</pre>

I don't have an explanation yet but 9 seconds is a very large difference.

=== August 26, 2009 ===

Suppose we sample at about 1Khz on a dual-core CPU and run an app for 10 seconds. An app that hogs the CPU should give us 20k samples, or 10k samples if it's pegged to a single core.

I coded this up in a [http://github.com/wagerlabs/firefox-startup/blob/0daf8a950b91e8d9c24187d54b04fb51f2764490/cpu.d cpu.d] DTrace script and sampled Firefox. Apparently, Firefox only gets ~1700 samples on the CPU or 1.7s so its first 10 seconds of life are spent doing something else, e.g. disk IO.

We already know that Firefox is slow to start up but this type of sampling neatly points the finger in the right direction. Props to Brendan Gregg for teaching me how to fish!

What kind of IO does Firefox do upon startup, though? I find the output from ''iosnoop'' a bit confusing so I whipped up [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/io.d a version of my own] that prints just disk access times, disk operation deltas, block size and file name accessed.

The script samples Firefox during the first 20 seconds of its life and the [http://pastie.org/595737 output from ''./cold.sh io.d''] looks like this:

<pre>
DELTA DTIME D SIZE PATHNAME
1399 1407 R 4096 ??/dtrace/libdtrace_dyld.dylib
1141 1145 R 8192 ??/dtrace/libdtrace_dyld.dylib
1301 1305 R 4096 ??/MacOS/XUL
1701 1705 R 4096 ??/MacOS/XUL
11081 685 R 4096 ??/MacOS/libmozjs.dylib
1989 2253 R 36864 ??/MacOS/libmozjs.dylib
1768 1894 R 24576 ??/MacOS/libmozjs.dylib
27111 2135 R 200704 ??/MacOS/XUL
1411 1134 R 8192 ??/MacOS/libxpcom.dylib
1784 1968 R 4096 ??/MacOS/libplds4.dylib
2383 464 R 4096 ??/MacOS/libxpcom.dylib
2886 456 R 4096 ??/MacOS/libxpcom.dylib
1431 732 R 4096 ??/MacOS/libplds4.dylib
2640 706 R 4096 ??/MacOS/libnspr4.dylib
3092 484 R 4096 ??/MacOS/libnspr4.dylib
...
</pre>

I also have [http://github.com/wagerlabs/firefox-startup/blob/7b78d70584bf9783d203e46786cbe800be4f5ba6/iosum.d a version that rolls disk operations by file name and read/write op]. It also samples during 20 seconds and [http://pastie.org/595727 outputs something like this]:

<pre>
DELTA DTIME SIZE D PATHNAME
307 311 4096 R ??/en_US.ISO8859-1/LC_TIME
437 441 4096 R ??/pref/firefox-branding.js
439 443 4096 R ??/su3x40q6.2/extensions.cache
441 445 4096 R ??/en_US.ISO8859-1/LC_NUMERIC
640 644 4096 R ??/en_US.ISO8859-1/LC_MONETARY
643 648 4096 R ??/chrome/browser.manifest
644 650 4096 R ??/la_LN.US-ASCII/LC_COLLATE
645 650 4096 R ??/Firefox/profiles.ini
646 650 4096 R ??/chrome/en-US.manifest
...
</pre>

All timings are in microseconds and specific to my Apple 256Gb SSD.

There's something else that I discovered during my snooping around with io.d. Compare the first 20 seconds of [http://gist.github.com/175843 the life of Safari] and [http://gist.github.com/175842 the life of Firefox].

Basically, all the shared libraries used by Safari are pulled in from the dynamic library shared cache. Safari uses mostly system frameworks, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 /Applications/Safari.app/Contents/MacOS/Safari

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
</pre>

whereas Firefox uses a whole lot more custom shared libraries, e.g.

<pre>
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 ./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2

dyld: loaded: /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
dyld: loaded: /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
dyld: loaded: /System/Library/PrivateFrameworks/HelpData.framework/Versions/A/HelpData
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
dyld: loaded:
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib

dyld: loaded: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
dyld: loaded: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin
dyld: loaded: /System/Library/QuickTime/QuickTimeComponents.component/Contents/MacOS/QuickTimeComponents
dyld: loaded: /System/Library/Frameworks/AGL.framework/Versions/A/AGL
dyld: loaded: /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC

dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowserdirprovider.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/components/libbrowsercomps.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
dyld: loaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
dyld: unloaded: /Users/joelr/Work/mozilla/startup/Minefield.app/Contents/MacOS/libnssckbi.dylib
</pre>

I separated the list of libraries loaded by Firefox into three groups: same libraries as Safari, other system frameworks, custom libraries.

What needs to be done to make Firefox use the dynamic library shared cache as aggressively as Safari?

Another puzzling bit involves the difference between Safari libraries loaded by dyld (above) and the list of libraries that Safari is linked against, e.g.

<pre>
otool -L /Applications/Safari.app/Contents/MacOS/Safari

/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
/System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface
/System/Library/PrivateFrameworks/SyndicationUI.framework/Versions/A/SyndicationUI
/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
/usr/lib/libicucore.A.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
</pre>

Does Safari lazily loading some frameworks? Why do I see OpenGL initializers run when I set ''DYLD_PRINT_INITIALIZERS'' then? The OpenGL framework does not appear to be loaded when Safari starts (dyld output above).

For the sake of comparing, here's the list of libraries that Firefox is linked against

<pre>
otool -L ./Minefield.app/Contents/MacOS/firefox-bin./Minefield.app/Contents/MacOS/firefox-bin:

/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/XUL
@executable_path/libmozjs.dylib
@executable_path/libxpcom.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
</pre>

as well as the list of libraries for XUL

<pre>
otool -L ./Minefield.app/Contents/MacOS/XUL

@executable_path/XUL
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
@executable_path/libsqlite3.dylib
@executable_path/libmozjs.dylib
@executable_path/libsmime3.dylib
@executable_path/libssl3.dylib
@executable_path/libnss3.dylib
@executable_path/libnssutil3.dylib
@executable_path/libsoftokn3.dylib
/usr/lib/libcups.2.dylib
@executable_path/libplds4.dylib
@executable_path/libplc4.dylib
@executable_path/libnspr4.dylib
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/usr/lib/libcrypto.0.9.7.dylib
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/usr/lib/libobjc.A.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
</pre>

It turns out that ''DYLD_PRINT_LIBRARIES=1'' is the way to show libraries loaded by dyld before a program's entry point (main) whereas ''DYLD_PRINT_LIBRARIES_POST_LAUNCH=1'' will show libraries loaded afterwards, e.g. using ''dlopen'' or ''NSBundle''.

=== August 25, 2009 ===

Created a [http://github.com/wagerlabs/firefox-startup/blob/c0b7691c60eb0455d7475aa17777edcf3ed3d605/main-entry-probe.patch static probe that fires first thing in XRE_main] and updated my [http://github.com/wagerlabs/firefox-startup/tree/master DTrace scripts] to use it.

''pid$target::function:entry'' probes are very slow since DTrace may have to search thousands of functions. All that search time skews elapsed time reported by ''timestamp''. USDT (static) probes are just a few NOP instructions in the code that get fixed up by DTrace as needed so they work much faster.

=== August 24, 2009 ===

Startup time is measured from the entry to XRE_main to the return of the BrowserStartup JS function. It takes a good bit of time but nothing compared to the time elapsed from the start of Firefox to the call to XRE_main.

According to my [http://github.com/wagerlabs/firefox-startup/blob/8260ff9534f25eecf8e9e9feb590168815902a67/static-init.d static-init.d] script, the static initialization time can be ignored. I'm recording the library name and then timing the following call to ''ImageLoader::runInitializers'' in dyld. The cumulative time is too small to be of essence, though.

<pre>
0.000053548s for /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.000053732s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libsoftokn3.dylib
0.000069234s for /System/Library/PrivateFrameworks/Shortcut.framework/Versions/A/Shortcut
0.000070455s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/libnssckbi.dylib
0.000072754s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowsercomps.dylib
0.000073443s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/ATSHI.dylib
0.000074363s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCSync.A.dylib
0.000075845s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGATS.A.dylib
0.000076892s for /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libRIP.A.dylib
0.000089767s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libnssdbm3.dylib
0.000094390s for /Users/joelr/Work/mozilla/startup/./Minefield.app/Contents/MacOS/libfreebl3.dylib
0.000100462s for /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0.000115375s for /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0.001161267s for /Users/joelr/Work/mozilla/startup/MinefieldRelease.app/Contents/MacOS/components/libbrowserdirprovider.dylib
---------------
= 0.002181527s

Initialization: 42.986413480s
Startup : 7.015292701s
---------------
= 50.001706181s
</pre>

Digging deeper...

=== August 21, 2009 ===

[http://wagerlabs.com/post/168237170/faster-mac-firefox Blogged].

My DTrace scripts live [http://github.com/wagerlabs/firefox-startup/tree/master here]. Use like this

<pre>
sudo ./cold.sh static-init.d
</pre>

= DTrace tips and tricks =

=== DTrace overhead and proper measuring technique ===

[http://www.mail-archive.com/dtrace-discuss@opensolaris.org/msg03639.html I'm still not done!]

=== Function entry/return vs static probes ===

Q: I added a couple of static probes to Firefox to measure actual work done. I could have used a pid$target probe with a function name but work is done within an if statement, which is where I placed the static probes. I'm wondering about my use, though. Is the following significantly more
efficient than pid$target::FunName:entry and return?

A: No, it's not significantly more efficient. Both function entry/return probes and USDT probes involve replacing an instruction in the stream with a trap instruction. The expense of the trap into the kernel is going to overwhelm any subtleties in handling between the two different types of probes. --Chad Mynhier

=== Combining scripts ===

I find it convenient to have separate DTrace scripts for various exit probes. This way I can avoid editing the main script whenever I want to stop sampling once 10 seconds go by or ''XRE_main'' is entered or the ''BrowserReturn'' JavaScript function returns.

It can be done like this

<pre>
#!/bin/bash

cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2"
scripts=""

for i in $*; do scripts="$scripts -s $i"; done

sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZ $scripts
</pre>

and then

<pre>
./cold.sh startup.d stop-main.d
Total: 11001.481323ms
</pre>

or

<pre>
./cold.sh startup.d stop-js.d
Total: 7004.799264ms
</pre>

My scripts can be [http://github.com/wagerlabs/firefox-startup/tree/master found on GitHub].

=== No sudo ===

<pre>
sudo chmod u+s `which dtrace`
</pre>

You can now run dtrace as yourself. Enjoy!

=== timestamp vs vtimestamp ===

''vtimestamp'' measures CPU time of the current thread, excluding IO and DTrace overhead. ''timestamp'' can still be used for deltas but the goal is to use as few pid$target probes as possible as they affect ''timestamp'' when dtrace has to switch between kernel and userland. io and syscall providers are fast and run in the kernel.

=== Invalid address ===

Have you seen this kind of error before?

<pre>
dtrace: error on enabled probe ID 27 (ID 22130: pid34547:libSystem.B.dylib:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
dtrace: error on enabled probe ID 2 (ID 22782: pid34547:dyld:dlopen:entry): invalid address (0x2ac204) in action #1 at DIF offset 28
</pre>

More likely than not, you are using copyinstr on memory that hasn't been paged in yet. Try saving the pointer on entry and doing the copying on return or later.
''Italic text''